Loading ...
Sorry, an error occurred while loading the content.

Re: Bad upload data

Expand Messages
  • Mike Frerkes
    There are no sources of EM interference I m aware of. I have learned that you should never use a metal pole as a mast for the anemometer, however. A couple
    Message 1 of 6 , Jul 29, 2005
    • 0 Attachment
      There are no sources of EM interference I'm aware of. I have
      learned that you should never use a metal pole as a mast for the
      anemometer, however. A couple of weeks ago, I switched to PVC pipe
      and the wind readings are now flawless, even with a cabled
      connection. Before the switch, I couldn't use the cabled connection
      because the wind readings would be horribly inaccurate.

      I'm puzzled why the bad readings (i.e zero degree temp) DO NOT show
      up on my HeavyWeather program under the min/max tabs for temperature
      and dewpoint, or on the base unit's display. There is no rhyme or
      reason to their occurence. Sometimes I may go an entire week
      without a false reading. There have been a few instances where I've
      registered two in the same day. The only common thread is that they
      occur within the first five minutes of each hour AND they always
      take place between 12am and 5am.



      --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
      <wuhu_software@y...> wrote:
      >
      > Mike,
      >
      > What is interesting is that the Heavyweather application is indeed
      > writing bogus data to the currdat.lst file but somehow this does
      not
      > make in to the history.dat file.
      >
      > There are a couple of theories why this is the case. Firstly, the
      > currdat.lst file is written once per minute by Heavyweather, no
      once
      > per update (9 seconds in wired mode). Since it reading 5 samples
      per
      > history file save, it might just be random. The other theory is
      that
      > somehow it can detect bad data and rejects it internally.
      >
      > A way to catch the Heavyweather application writing bad data is to
      use
      > the alarms/events. You can set an alarm for low temperature for
      > instance. When you hear the audio, open currdat.lst with notepad
      within
      > 1 minute. You can see that the bogus data is being generated by
      > Heavyweather.
      >
      > From what I gather from these forums, the best way to eliminate
      bogus
      > data is to build a shielded cable. I have not modified my wiring,
      but
      > it will be interesting to see if this corrects the problem. If
      anyone
      > has modified the cable, please post your results.
      >
      > I have attempted to detect and eliminate bogus wind readings and
      > others. I am not sure if I had done the same on version 129 that
      is
      > available from the alternative download page. I believe I posted a
      > message on the rules I used to attempt to detect bad data
      previously.
      >
      > If you have any idea or rules that can be applied to the data, let
      me
      > know. I am willing to modify the application if we can find simple
      > rules to detect bogus data.
      >
      > P.S. Are there any source of EM radiation, power dips, or anything
      else
      > that happens at that time around your home? Just wondering.
      >
      > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
      > <mfrerkes@m...> wrote:
      > > I have also experienced erroneous data uploads, almost always a
      > > temperature and dew point reading of 0. I've traced the
      defective
      > > data to the WUHU program itself, because those extreme readings
      are
      > > not registered on HeavyWeather or the La Crosse base unit.
      > >
      > > My false readings always seem to appear in the early morning
      hours
      > > between 12am and 5am, and always within the first five minutes
      of the
      > > hour (e.g. 1:03am or 4:05am). I have the upload interval set at
      five
      > > minutes, so it might be related to that in some way. There has
      never
      > > been a bad reading after the first five minutes of each hour.
      > >
      > > These incorrect readings are uploaded to both Weather
      Underground and
      > > CWOP, so I've ruled out any data glitches on their end.
      Hopefully
      > > this information may assist you in isolating the cause of the
      problem.
    • wuhu_software
      Mike, On my system, I seem to run for weeks without a single bad reading. Other times, I will have windspeed errors combined with other errors in rapid
      Message 2 of 6 , Jul 30, 2005
      • 0 Attachment
        Mike,

        On my system, I seem to run for weeks without a single bad reading.
        Other times, I will have windspeed errors combined with other errors
        in rapid succession. I have not been able to find a pattern to it.

        The other possibility for these bad reading being written to
        currdat.lst and not to the history file might possibly be a bug in
        Heavyweather itself. It is difficult to say.

        One thing that is for certain is that Heavyweather is writting bogus
        data.

        Here is an example of a bad reading put out by Heavyweather:

        [rain_24h]
        mm = "15370.0"
        inch = "605.11"

        This was found in the currdat.lst file. As you can see, Heavyweather
        is writing bogus data for both mm and inch measurements. The same is
        true for bad temps and windspeed data as well.

        These bogus readings will be overwritten on the next update performed
        by Heavyweather (which updates the currdat.lst file once per minute).

        As I mentioned before, I attempted to catch some of these errors. For
        instance, when reading temperature, if the current reading is not
        within 10 degrees of the average temperature readings over the past
        ten minutes, then the software will use the last valid temperature
        reading (that passed the above test) as the current temperature
        reading.

        These features were added in one of the later versions and is
        definetly in version 129.

        Which version are you running? If you are running version 129 and
        still seeing these bogus temperature readings when running version
        129 let me know. I can look over the code to see if there is a
        mistake.

        Thanks.

        --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
        <mfrerkes@m...> wrote:
        > There are no sources of EM interference I'm aware of. I have
        > learned that you should never use a metal pole as a mast for the
        > anemometer, however. A couple of weeks ago, I switched to PVC pipe
        > and the wind readings are now flawless, even with a cabled
        > connection. Before the switch, I couldn't use the cabled
        connection
        > because the wind readings would be horribly inaccurate.
        >
        > I'm puzzled why the bad readings (i.e zero degree temp) DO NOT show
        > up on my HeavyWeather program under the min/max tabs for
        temperature
        > and dewpoint, or on the base unit's display. There is no rhyme or
        > reason to their occurence. Sometimes I may go an entire week
        > without a false reading. There have been a few instances where
        I've
        > registered two in the same day. The only common thread is that
        they
        > occur within the first five minutes of each hour AND they always
        > take place between 12am and 5am.
        >
        >
        >
        > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
        > <wuhu_software@y...> wrote:
        > >
        > > Mike,
        > >
        > > What is interesting is that the Heavyweather application is
        indeed
        > > writing bogus data to the currdat.lst file but somehow this does
        > not
        > > make in to the history.dat file.
        > >
        > > There are a couple of theories why this is the case. Firstly, the
        > > currdat.lst file is written once per minute by Heavyweather, no
        > once
        > > per update (9 seconds in wired mode). Since it reading 5 samples
        > per
        > > history file save, it might just be random. The other theory is
        > that
        > > somehow it can detect bad data and rejects it internally.
        > >
        > > A way to catch the Heavyweather application writing bad data is
        to
        > use
        > > the alarms/events. You can set an alarm for low temperature for
        > > instance. When you hear the audio, open currdat.lst with notepad
        > within
        > > 1 minute. You can see that the bogus data is being generated by
        > > Heavyweather.
        > >
        > > From what I gather from these forums, the best way to eliminate
        > bogus
        > > data is to build a shielded cable. I have not modified my wiring,
        > but
        > > it will be interesting to see if this corrects the problem. If
        > anyone
        > > has modified the cable, please post your results.
        > >
        > > I have attempted to detect and eliminate bogus wind readings and
        > > others. I am not sure if I had done the same on version 129 that
        > is
        > > available from the alternative download page. I believe I posted
        a
        > > message on the rules I used to attempt to detect bad data
        > previously.
        > >
        > > If you have any idea or rules that can be applied to the data,
        let
        > me
        > > know. I am willing to modify the application if we can find
        simple
        > > rules to detect bogus data.
        > >
        > > P.S. Are there any source of EM radiation, power dips, or
        anything
        > else
        > > that happens at that time around your home? Just wondering.
        > >
        > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
        > > <mfrerkes@m...> wrote:
        > > > I have also experienced erroneous data uploads, almost always a
        > > > temperature and dew point reading of 0. I've traced the
        > defective
        > > > data to the WUHU program itself, because those extreme readings
        > are
        > > > not registered on HeavyWeather or the La Crosse base unit.
        > > >
        > > > My false readings always seem to appear in the early morning
        > hours
        > > > between 12am and 5am, and always within the first five minutes
        > of the
        > > > hour (e.g. 1:03am or 4:05am). I have the upload interval set
        at
        > five
        > > > minutes, so it might be related to that in some way. There has
        > never
        > > > been a bad reading after the first five minutes of each hour.
        > > >
        > > > These incorrect readings are uploaded to both Weather
        > Underground and
        > > > CWOP, so I've ruled out any data glitches on their end.
        > Hopefully
        > > > this information may assist you in isolating the cause of the
        > problem.
      • Mike Frerkes
        I was running an earlier version (maybe 127 or 126). I have just installed the 129 version, so I ll see if anything changes with respect to this problem. If
        Message 3 of 6 , Jul 30, 2005
        • 0 Attachment
          I was running an earlier version (maybe 127 or 126). I have just
          installed the 129 version, so I'll see if anything changes with
          respect to this problem.

          If you are experiencing wind-related errors, consider using PVC pipe
          for the mast. I was using metal pipe up until a few weeks ago. On
          days above 90, the metal would get hot and cause siginificant errors
          in the wind data, even in wireless mode. After switching to PVC, I
          was able to run the unit cabled, and now even on 90-degree days my
          data is error-free.

          Thanks for you help and suggestions with my problem.

          Mike


          --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
          <wuhu_software@y...> wrote:
          > Mike,
          >
          > On my system, I seem to run for weeks without a single bad
          reading.
          > Other times, I will have windspeed errors combined with other
          errors
          > in rapid succession. I have not been able to find a pattern to it.
          >
          > The other possibility for these bad reading being written to
          > currdat.lst and not to the history file might possibly be a bug in
          > Heavyweather itself. It is difficult to say.
          >
          > One thing that is for certain is that Heavyweather is writting
          bogus
          > data.
          >
          > Here is an example of a bad reading put out by Heavyweather:
          >
          > [rain_24h]
          > mm = "15370.0"
          > inch = "605.11"
          >
          > This was found in the currdat.lst file. As you can see,
          Heavyweather
          > is writing bogus data for both mm and inch measurements. The same
          is
          > true for bad temps and windspeed data as well.
          >
          > These bogus readings will be overwritten on the next update
          performed
          > by Heavyweather (which updates the currdat.lst file once per
          minute).
          >
          > As I mentioned before, I attempted to catch some of these errors.
          For
          > instance, when reading temperature, if the current reading is not
          > within 10 degrees of the average temperature readings over the
          past
          > ten minutes, then the software will use the last valid temperature
          > reading (that passed the above test) as the current temperature
          > reading.
          >
          > These features were added in one of the later versions and is
          > definetly in version 129.
          >
          > Which version are you running? If you are running version 129 and
          > still seeing these bogus temperature readings when running version
          > 129 let me know. I can look over the code to see if there is a
          > mistake.
          >
          > Thanks.
          >
          > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
          > <mfrerkes@m...> wrote:
          > > There are no sources of EM interference I'm aware of. I have
          > > learned that you should never use a metal pole as a mast for the
          > > anemometer, however. A couple of weeks ago, I switched to PVC
          pipe
          > > and the wind readings are now flawless, even with a cabled
          > > connection. Before the switch, I couldn't use the cabled
          > connection
          > > because the wind readings would be horribly inaccurate.
          > >
          > > I'm puzzled why the bad readings (i.e zero degree temp) DO NOT
          show
          > > up on my HeavyWeather program under the min/max tabs for
          > temperature
          > > and dewpoint, or on the base unit's display. There is no rhyme
          or
          > > reason to their occurence. Sometimes I may go an entire week
          > > without a false reading. There have been a few instances where
          > I've
          > > registered two in the same day. The only common thread is that
          > they
          > > occur within the first five minutes of each hour AND they always
          > > take place between 12am and 5am.
          > >
          > >
          > >
          > > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
          > > <wuhu_software@y...> wrote:
          > > >
          > > > Mike,
          > > >
          > > > What is interesting is that the Heavyweather application is
          > indeed
          > > > writing bogus data to the currdat.lst file but somehow this
          does
          > > not
          > > > make in to the history.dat file.
          > > >
          > > > There are a couple of theories why this is the case. Firstly,
          the
          > > > currdat.lst file is written once per minute by Heavyweather,
          no
          > > once
          > > > per update (9 seconds in wired mode). Since it reading 5
          samples
          > > per
          > > > history file save, it might just be random. The other theory
          is
          > > that
          > > > somehow it can detect bad data and rejects it internally.
          > > >
          > > > A way to catch the Heavyweather application writing bad data
          is
          > to
          > > use
          > > > the alarms/events. You can set an alarm for low temperature
          for
          > > > instance. When you hear the audio, open currdat.lst with
          notepad
          > > within
          > > > 1 minute. You can see that the bogus data is being generated
          by
          > > > Heavyweather.
          > > >
          > > > From what I gather from these forums, the best way to
          eliminate
          > > bogus
          > > > data is to build a shielded cable. I have not modified my
          wiring,
          > > but
          > > > it will be interesting to see if this corrects the problem. If
          > > anyone
          > > > has modified the cable, please post your results.
          > > >
          > > > I have attempted to detect and eliminate bogus wind readings
          and
          > > > others. I am not sure if I had done the same on version 129
          that
          > > is
          > > > available from the alternative download page. I believe I
          posted
          > a
          > > > message on the rules I used to attempt to detect bad data
          > > previously.
          > > >
          > > > If you have any idea or rules that can be applied to the data,
          > let
          > > me
          > > > know. I am willing to modify the application if we can find
          > simple
          > > > rules to detect bogus data.
          > > >
          > > > P.S. Are there any source of EM radiation, power dips, or
          > anything
          > > else
          > > > that happens at that time around your home? Just wondering.
          > > >
          > > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
          > > > <mfrerkes@m...> wrote:
          > > > > I have also experienced erroneous data uploads, almost
          always a
          > > > > temperature and dew point reading of 0. I've traced the
          > > defective
          > > > > data to the WUHU program itself, because those extreme
          readings
          > > are
          > > > > not registered on HeavyWeather or the La Crosse base unit.
          > > > >
          > > > > My false readings always seem to appear in the early morning
          > > hours
          > > > > between 12am and 5am, and always within the first five
          minutes
          > > of the
          > > > > hour (e.g. 1:03am or 4:05am). I have the upload interval
          set
          > at
          > > five
          > > > > minutes, so it might be related to that in some way. There
          has
          > > never
          > > > > been a bad reading after the first five minutes of each hour.
          > > > >
          > > > > These incorrect readings are uploaded to both Weather
          > > Underground and
          > > > > CWOP, so I've ruled out any data glitches on their end.
          > > Hopefully
          > > > > this information may assist you in isolating the cause of
          the
          > > problem.
        • wuhu_software
          Mike, It appears that the data sent to the weather station and received from the weather station does include a simple data integrity check. The data frames
          Message 4 of 6 , Jul 31, 2005
          • 0 Attachment
            Mike,

            It appears that the data sent to the weather station and received
            from the weather station does include a simple data integrity check.

            The data frames contain what appears to be a single byte checksum.
            Although this is not a very reliable data integrity check (CRC would
            have been much better), it should catch transmission errors between
            the weather station and the PC. My guess is that the serial data
            between the weather station and the PC is fine and that the problem
            is really with the station and it's data collection from the sensors.

            Reading over the other forums, it appears that the data sent from the
            sensors to the weather station does not include even a simple
            checksum to validate the data.

            When corrupt data is retrieved from the sensors, the weather station
            is not performing data integrity checking, it is probably just
            passing the info to the PC.

            Heavyweather may or not be attempting to peform some checking
            internally (before it sends to it's user interface or history.dat
            file, but all indications are that it passes on the bogus data to
            currdat.lst.

            As consumers, we only have two options to reduce this noise.

            1) Reduce the potential for noise.
            2) Attempt to filter out bad data with range checking.

            In the case of 1 as you mentioned, the sensors should not be mounted
            to a metal mast (I have mine mounted on an aluminum rod). We can also
            attempt to reduce the noise produced by the cheap wiring as mention
            by other posters.

            As for 2, as mentioned, I have done some basic data checking. It is
            far from perfect though. I will continue to pursue improving the
            error checking.

            If any user is detecting bad data in the uploads using version 129 or
            later, please forward the information to me. Please capture the
            entire line of data including temperature, wind speed, ect so that I
            can see if there is a detectable pattern.

            I want to be very careful on filtering the data as I do not want to
            mask out rapidly changing conditions (such as wind gust data during
            severe conditions).

            Thanks for your help.

            --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
            <mfrerkes@m...> wrote:
            > I was running an earlier version (maybe 127 or 126). I have just
            > installed the 129 version, so I'll see if anything changes with
            > respect to this problem.
            >
            > If you are experiencing wind-related errors, consider using PVC
            pipe
            > for the mast. I was using metal pipe up until a few weeks ago. On
            > days above 90, the metal would get hot and cause siginificant
            errors
            > in the wind data, even in wireless mode. After switching to PVC, I
            > was able to run the unit cabled, and now even on 90-degree days my
            > data is error-free.
            >
            > Thanks for you help and suggestions with my problem.
            >
            > Mike
            >
            >
            > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
            > <wuhu_software@y...> wrote:
            > > Mike,
            > >
            > > On my system, I seem to run for weeks without a single bad
            > reading.
            > > Other times, I will have windspeed errors combined with other
            > errors
            > > in rapid succession. I have not been able to find a pattern to it.
            > >
            > > The other possibility for these bad reading being written to
            > > currdat.lst and not to the history file might possibly be a bug
            in
            > > Heavyweather itself. It is difficult to say.
            > >
            > > One thing that is for certain is that Heavyweather is writting
            > bogus
            > > data.
            > >
            > > Here is an example of a bad reading put out by Heavyweather:
            > >
            > > [rain_24h]
            > > mm = "15370.0"
            > > inch = "605.11"
            > >
            > > This was found in the currdat.lst file. As you can see,
            > Heavyweather
            > > is writing bogus data for both mm and inch measurements. The same
            > is
            > > true for bad temps and windspeed data as well.
            > >
            > > These bogus readings will be overwritten on the next update
            > performed
            > > by Heavyweather (which updates the currdat.lst file once per
            > minute).
            > >
            > > As I mentioned before, I attempted to catch some of these errors.
            > For
            > > instance, when reading temperature, if the current reading is not
            > > within 10 degrees of the average temperature readings over the
            > past
            > > ten minutes, then the software will use the last valid
            temperature
            > > reading (that passed the above test) as the current temperature
            > > reading.
            > >
            > > These features were added in one of the later versions and is
            > > definetly in version 129.
            > >
            > > Which version are you running? If you are running version 129 and
            > > still seeing these bogus temperature readings when running
            version
            > > 129 let me know. I can look over the code to see if there is a
            > > mistake.
            > >
            > > Thanks.
            > >
            > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
            > > <mfrerkes@m...> wrote:
            > > > There are no sources of EM interference I'm aware of. I have
            > > > learned that you should never use a metal pole as a mast for
            the
            > > > anemometer, however. A couple of weeks ago, I switched to PVC
            > pipe
            > > > and the wind readings are now flawless, even with a cabled
            > > > connection. Before the switch, I couldn't use the cabled
            > > connection
            > > > because the wind readings would be horribly inaccurate.
            > > >
            > > > I'm puzzled why the bad readings (i.e zero degree temp) DO NOT
            > show
            > > > up on my HeavyWeather program under the min/max tabs for
            > > temperature
            > > > and dewpoint, or on the base unit's display. There is no rhyme
            > or
            > > > reason to their occurence. Sometimes I may go an entire week
            > > > without a false reading. There have been a few instances where
            > > I've
            > > > registered two in the same day. The only common thread is that
            > > they
            > > > occur within the first five minutes of each hour AND they
            always
            > > > take place between 12am and 5am.
            > > >
            > > >
            > > >
            > > > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
            > > > <wuhu_software@y...> wrote:
            > > > >
            > > > > Mike,
            > > > >
            > > > > What is interesting is that the Heavyweather application is
            > > indeed
            > > > > writing bogus data to the currdat.lst file but somehow this
            > does
            > > > not
            > > > > make in to the history.dat file.
            > > > >
            > > > > There are a couple of theories why this is the case. Firstly,
            > the
            > > > > currdat.lst file is written once per minute by Heavyweather,
            > no
            > > > once
            > > > > per update (9 seconds in wired mode). Since it reading 5
            > samples
            > > > per
            > > > > history file save, it might just be random. The other theory
            > is
            > > > that
            > > > > somehow it can detect bad data and rejects it internally.
            > > > >
            > > > > A way to catch the Heavyweather application writing bad data
            > is
            > > to
            > > > use
            > > > > the alarms/events. You can set an alarm for low temperature
            > for
            > > > > instance. When you hear the audio, open currdat.lst with
            > notepad
            > > > within
            > > > > 1 minute. You can see that the bogus data is being generated
            > by
            > > > > Heavyweather.
            > > > >
            > > > > From what I gather from these forums, the best way to
            > eliminate
            > > > bogus
            > > > > data is to build a shielded cable. I have not modified my
            > wiring,
            > > > but
            > > > > it will be interesting to see if this corrects the problem.
            If
            > > > anyone
            > > > > has modified the cable, please post your results.
            > > > >
            > > > > I have attempted to detect and eliminate bogus wind readings
            > and
            > > > > others. I am not sure if I had done the same on version 129
            > that
            > > > is
            > > > > available from the alternative download page. I believe I
            > posted
            > > a
            > > > > message on the rules I used to attempt to detect bad data
            > > > previously.
            > > > >
            > > > > If you have any idea or rules that can be applied to the
            data,
            > > let
            > > > me
            > > > > know. I am willing to modify the application if we can find
            > > simple
            > > > > rules to detect bogus data.
            > > > >
            > > > > P.S. Are there any source of EM radiation, power dips, or
            > > anything
            > > > else
            > > > > that happens at that time around your home? Just wondering.
            > > > >
            > > > > --- In wuhu_software_group@yahoogroups.com, "Mike Frerkes"
            > > > > <mfrerkes@m...> wrote:
            > > > > > I have also experienced erroneous data uploads, almost
            > always a
            > > > > > temperature and dew point reading of 0. I've traced the
            > > > defective
            > > > > > data to the WUHU program itself, because those extreme
            > readings
            > > > are
            > > > > > not registered on HeavyWeather or the La Crosse base unit.
            > > > > >
            > > > > > My false readings always seem to appear in the early
            morning
            > > > hours
            > > > > > between 12am and 5am, and always within the first five
            > minutes
            > > > of the
            > > > > > hour (e.g. 1:03am or 4:05am). I have the upload interval
            > set
            > > at
            > > > five
            > > > > > minutes, so it might be related to that in some way. There
            > has
            > > > never
            > > > > > been a bad reading after the first five minutes of each
            hour.
            > > > > >
            > > > > > These incorrect readings are uploaded to both Weather
            > > > Underground and
            > > > > > CWOP, so I've ruled out any data glitches on their end.
            > > > Hopefully
            > > > > > this information may assist you in isolating the cause of
            > the
            > > > problem.
          Your message has been successfully submitted and would be delivered to recipients shortly.