Loading ...
Sorry, an error occurred while loading the content.

118Re: Bad upload data

Expand Messages
  • wuhu_software
    Jul 31, 2005
    • 0 Attachment
      Mike,

      It appears that the data sent to the weather station and received
      from the weather station does include a simple data integrity check.

      The data frames contain what appears to be a single byte checksum.
      Although this is not a very reliable data integrity check (CRC would
      have been much better), it should catch transmission errors between
      the weather station and the PC. My guess is that the serial data
      between the weather station and the PC is fine and that the problem
      is really with the station and it's data collection from the sensors.

      Reading over the other forums, it appears that the data sent from the
      sensors to the weather station does not include even a simple
      checksum to validate the data.

      When corrupt data is retrieved from the sensors, the weather station
      is not performing data integrity checking, it is probably just
      passing the info to the PC.

      Heavyweather may or not be attempting to peform some checking
      internally (before it sends to it's user interface or history.dat
      file, but all indications are that it passes on the bogus data to
      currdat.lst.

      As consumers, we only have two options to reduce this noise.

      1) Reduce the potential for noise.
      2) Attempt to filter out bad data with range checking.

      In the case of 1 as you mentioned, the sensors should not be mounted
      to a metal mast (I have mine mounted on an aluminum rod). We can also
      attempt to reduce the noise produced by the cheap wiring as mention
      by other posters.

      As for 2, as mentioned, I have done some basic data checking. It is
      far from perfect though. I will continue to pursue improving the
      error checking.

      If any user is detecting bad data in the uploads using version 129 or
      later, please forward the information to me. Please capture the
      entire line of data including temperature, wind speed, ect so that I
      can see if there is a detectable pattern.

      I want to be very careful on filtering the data as I do not want to
      mask out rapidly changing conditions (such as wind gust data during
      severe conditions).

      Thanks for your help.

      --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
      <mfrerkes@m...> wrote:
      > I was running an earlier version (maybe 127 or 126). I have just
      > installed the 129 version, so I'll see if anything changes with
      > respect to this problem.
      >
      > If you are experiencing wind-related errors, consider using PVC
      pipe
      > for the mast. I was using metal pipe up until a few weeks ago. On
      > days above 90, the metal would get hot and cause siginificant
      errors
      > in the wind data, even in wireless mode. After switching to PVC, I
      > was able to run the unit cabled, and now even on 90-degree days my
      > data is error-free.
      >
      > Thanks for you help and suggestions with my problem.
      >
      > Mike
      >
      >
      > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
      > <wuhu_software@y...> wrote:
      > > Mike,
      > >
      > > On my system, I seem to run for weeks without a single bad
      > reading.
      > > Other times, I will have windspeed errors combined with other
      > errors
      > > in rapid succession. I have not been able to find a pattern to it.
      > >
      > > The other possibility for these bad reading being written to
      > > currdat.lst and not to the history file might possibly be a bug
      in
      > > Heavyweather itself. It is difficult to say.
      > >
      > > One thing that is for certain is that Heavyweather is writting
      > bogus
      > > data.
      > >
      > > Here is an example of a bad reading put out by Heavyweather:
      > >
      > > [rain_24h]
      > > mm = "15370.0"
      > > inch = "605.11"
      > >
      > > This was found in the currdat.lst file. As you can see,
      > Heavyweather
      > > is writing bogus data for both mm and inch measurements. The same
      > is
      > > true for bad temps and windspeed data as well.
      > >
      > > These bogus readings will be overwritten on the next update
      > performed
      > > by Heavyweather (which updates the currdat.lst file once per
      > minute).
      > >
      > > As I mentioned before, I attempted to catch some of these errors.
      > For
      > > instance, when reading temperature, if the current reading is not
      > > within 10 degrees of the average temperature readings over the
      > past
      > > ten minutes, then the software will use the last valid
      temperature
      > > reading (that passed the above test) as the current temperature
      > > reading.
      > >
      > > These features were added in one of the later versions and is
      > > definetly in version 129.
      > >
      > > Which version are you running? If you are running version 129 and
      > > still seeing these bogus temperature readings when running
      version
      > > 129 let me know. I can look over the code to see if there is a
      > > mistake.
      > >
      > > Thanks.
      > >
      > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
      > > <mfrerkes@m...> wrote:
      > > > There are no sources of EM interference I'm aware of. I have
      > > > learned that you should never use a metal pole as a mast for
      the
      > > > anemometer, however. A couple of weeks ago, I switched to PVC
      > pipe
      > > > and the wind readings are now flawless, even with a cabled
      > > > connection. Before the switch, I couldn't use the cabled
      > > connection
      > > > because the wind readings would be horribly inaccurate.
      > > >
      > > > I'm puzzled why the bad readings (i.e zero degree temp) DO NOT
      > show
      > > > up on my HeavyWeather program under the min/max tabs for
      > > temperature
      > > > and dewpoint, or on the base unit's display. There is no rhyme
      > or
      > > > reason to their occurence. Sometimes I may go an entire week
      > > > without a false reading. There have been a few instances where
      > > I've
      > > > registered two in the same day. The only common thread is that
      > > they
      > > > occur within the first five minutes of each hour AND they
      always
      > > > take place between 12am and 5am.
      > > >
      > > >
      > > >
      > > > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
      > > > <wuhu_software@y...> wrote:
      > > > >
      > > > > Mike,
      > > > >
      > > > > What is interesting is that the Heavyweather application is
      > > indeed
      > > > > writing bogus data to the currdat.lst file but somehow this
      > does
      > > > not
      > > > > make in to the history.dat file.
      > > > >
      > > > > There are a couple of theories why this is the case. Firstly,
      > the
      > > > > currdat.lst file is written once per minute by Heavyweather,
      > no
      > > > once
      > > > > per update (9 seconds in wired mode). Since it reading 5
      > samples
      > > > per
      > > > > history file save, it might just be random. The other theory
      > is
      > > > that
      > > > > somehow it can detect bad data and rejects it internally.
      > > > >
      > > > > A way to catch the Heavyweather application writing bad data
      > is
      > > to
      > > > use
      > > > > the alarms/events. You can set an alarm for low temperature
      > for
      > > > > instance. When you hear the audio, open currdat.lst with
      > notepad
      > > > within
      > > > > 1 minute. You can see that the bogus data is being generated
      > by
      > > > > Heavyweather.
      > > > >
      > > > > From what I gather from these forums, the best way to
      > eliminate
      > > > bogus
      > > > > data is to build a shielded cable. I have not modified my
      > wiring,
      > > > but
      > > > > it will be interesting to see if this corrects the problem.
      If
      > > > anyone
      > > > > has modified the cable, please post your results.
      > > > >
      > > > > I have attempted to detect and eliminate bogus wind readings
      > and
      > > > > others. I am not sure if I had done the same on version 129
      > that
      > > > is
      > > > > available from the alternative download page. I believe I
      > posted
      > > a
      > > > > message on the rules I used to attempt to detect bad data
      > > > previously.
      > > > >
      > > > > If you have any idea or rules that can be applied to the
      data,
      > > let
      > > > me
      > > > > know. I am willing to modify the application if we can find
      > > simple
      > > > > rules to detect bogus data.
      > > > >
      > > > > P.S. Are there any source of EM radiation, power dips, or
      > > anything
      > > > else
      > > > > that happens at that time around your home? Just wondering.
      > > > >
      > > > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
      > > > > <mfrerkes@m...> wrote:
      > > > > > I have also experienced erroneous data uploads, almost
      > always a
      > > > > > temperature and dew point reading of 0. I've traced the
      > > > defective
      > > > > > data to the WUHU program itself, because those extreme
      > readings
      > > > are
      > > > > > not registered on HeavyWeather or the La Crosse base unit.
      > > > > >
      > > > > > My false readings always seem to appear in the early
      morning
      > > > hours
      > > > > > between 12am and 5am, and always within the first five
      > minutes
      > > > of the
      > > > > > hour (e.g. 1:03am or 4:05am). I have the upload interval
      > set
      > > at
      > > > five
      > > > > > minutes, so it might be related to that in some way. There
      > has
      > > > never
      > > > > > been a bad reading after the first five minutes of each
      hour.
      > > > > >
      > > > > > These incorrect readings are uploaded to both Weather
      > > > Underground and
      > > > > > CWOP, so I've ruled out any data glitches on their end.
      > > > Hopefully
      > > > > > this information may assist you in isolating the cause of
      > the
      > > > problem.
    • Show all 6 messages in this topic