Loading ...
Sorry, an error occurred while loading the content.

Re: VB code for history parser

Expand Messages
  • sbillie1
    Has anyone tried using Perl to do the parsing ? Also, I was just curious if anyone out there has written an RS232 driver using Perl ? Thanks, Bill
    Message 1 of 15 , Nov 1, 2005
    • 0 Attachment
      Has anyone tried using Perl to do the parsing ?

      Also, I was just curious if anyone out there has written an RS232 driver using Perl ?

      Thanks,



      Bill






      --- In wuhu_software_group@yahoogroups.com, "stevech" <stevech@s...> wrote:
      >
      > If someone would send me some WS23xx history.dat files I'll try to make my
      > program support it and the WS36xx and perhaps US/Metric.
      >
      > -----Original Message-----
      > From: wuhu_software_group@yahoogroups.com
      > [mailto:wuhu_software_group@yahoogroups.com] On Behalf Of wuhu_software
      > Sent: Saturday, October 22, 2005 12:24 PM
      > To: wuhu_software_group@yahoogroups.com
      > Subject: [wuhu_software_group] Re: VB code for history parser
      >
      >
      > Steve,
      >
      > Defintely upload your source if you want to. Feel free to add
      > whatever you want to the files section. If you need more space, I
      > have a bit at the alternative download site (comcast gives you 30megs
      > of space there).
      >
      > There are some cool activex objects out there for graphing, TChart is
      > one that comes to mind. I have not used it in 4-5 years, but it had
      > just about any feature you can think of. It was $150 for unlimited
      > license back then, not sure what it is up to now. The nice thing is
      > you can distribute it with your app.
      >
      > As far as the format used by the WS23xx, it is defintely a bit
      > different. It seems the relative pressure is not stored in each
      > record (absolute pressure is). I suspect that the last entry in the
      > file may contain an offset from the absolute pressure so that you can
      > determine relative pressure. I do not understand why HW would do
      > this, but that is apparently the case.
      >
      > If I can figure that piece of the puzzle out, I could write the CVS
      > and history.dat uploader for the WS23xx stations as well as other
      > apps.
      >
      > ----
      >
      > Here is what I found on the net:
      >
      > *******************
      >
      > HeavyWeather V2.0 beta
      > La Crosse WS-2310 and WS-2315 weather stations
      >
      > Each row of data is stored in 36 byte chunks starting from the
      > beginning of
      > the file (no header).
      >
      > Row
      > Offset Type Name Unit
      > ------ --------- ---------------- -----
      > 00 ULong [4] unknown - (Value is always 1)
      > 04 ULong [4] Timestamp seconds from 1/1/1900 00:00:00
      > (GMT)
      > 08 Float [4] Abs Pressure hectopascals (millibars)
      > 12 Float [4] Wind Speed meters/second
      > 16 ULong [4] Wind Direction see below
      > 20 Float [4] Total Rainfall millimeters
      > 24 Float [4] Indoor Temp Celsius
      > 28 Float [4] Outdoor Temp Celsius
      > 32 UWord [2] Indoor Humidity %
      > 34 UWord [2] Outdoor Humidity %
      >
      > At the end of the file is an additional 28 bytes providing
      > information about the dataset as a whole.
      >
      > Offset Type Description
      > ------ --------- ----------------
      > 00 ULong [4] unknown
      > 04 Ulong [4] unknown (0)
      > 08 Float [4] unknown (5.698)
      > 12 ULong [4] unknown (0)
      > 16 ULong [4] Number of rows
      > 20 ULong [4] Timestamp of the first row of data
      > 24 ULong [4] Timestamp of the last row of data
      >
      >
      > --- In wuhu_software_group@yahoogroups.com, "stevech" stevech@s...
      > wrote:
      > >
      > > I too have written MS Visual Basic 6 (VB6) code to parse the WS36xx
      > history
      > > file. At the moment, the program displays all the data in a big
      > scrolled
      > > table and also writes it out to a file as plain text (i.e., to go
      > in to
      > > Excel or whatever). I'm working on having the program generate
      > graphs and
      > > save them as JPGs which can be referenced in a web page.
      > >
      > > Making this program read WS23xx instead of WS36xx is trivial.
      > >
      > > I'll send this and the source code to anyone who wants it, or I can
      > upload
      > > it. The graphing part isn't done, as I'm trying to get all the
      > options and
      > > flexibility I want. The reason I'm not using one of the weather
      > graphing
      > > programs out there is that I want my graphs in forms that these
      > programs
      > > don't do- I tried them. Primarily - formatted with large fonts
      > suitable for
      > > display on my home LAN web server which has a lot of info on it (TV
      > > listings, webcams around the world, etc) and this must use large
      > fonts for
      > > viewing on the TV. The web server feeds VGA->video to the TVs.
      > >
      > > Anyway, that's my work in progress.
      > >
      > > Parsing the history data file turns out to be simple- the floating
      > point
      > > format is exactly that used in MS's VB and C languages so I
      > simply "cast"
      > > the byte array to a "single" within a user-defined structure.
      > >
      > > Steve
      > >
      >
      >
      >
      >
      >
      >
      >
      >
      > Yahoo! Groups Links
      >
    • jjantti2
      ... I just ran the program and it looks like WU went completely bananas when I dumped my history.dat file to it. There were over 42 thousand records on that
      Message 2 of 15 , Nov 7, 2005
      • 0 Attachment
        --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
        <wuhu_software@y...> wrote:

        > Since I have never attempted uploading older weather data, I am not
        > sure what happens at WU in the event of duplication. If you abort an
        > upload of the file and later attempt to upload it, I am not sure
        > what the effect is. Hopefully WU will recognize the duplicates and
        > pitch the old ones.
        >
        > Let me know how it goes should you decide to attempt this.

        I just ran the program and it looks like WU went completely bananas
        when I dumped my history.dat file to it. There were over 42 thousand
        records on that file and it took about 36 hours to finish.

        Have a look at the data for October 8th, 2005:

        http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IUUSIMAA2&month=10&day=8&year=2005

        You can clearly see spikes on some of the graphs. This is likely
        because my HeavyWeather software is gathering up weather data every
        minute. The fun part begins when you scroll down the page to see the
        tabular data. There are multiple duplicates with the same time and
        some of the data is plain wrong like those with dew point temperature
        being -17.8°C and the wind gust being 193.1 km/h.

        The other strange thing is that some of the data is actually missing.
        There are jumps of two minutes (usually after the duplicates). My best
        guess is that WU will only accept data that is sent minimum every two
        minutes and dumping a minute-interval database will cause this behavior.

        Best Regards,
        Juha
      • wuhu_software
        Juha, If you look at the tabular data, at the end, you will see either the label 3600HistoryToWU_V1 or WUHU132HW3600DAT. This indicates which program was used
        Message 3 of 15 , Nov 8, 2005
        • 0 Attachment
          Juha,

          If you look at the tabular data, at the end, you will see either the
          label 3600HistoryToWU_V1 or WUHU132HW3600DAT. This indicates which
          program was used to upload the data.

          As far as I can see, the entries with a dewpoint of -17.8°C and the
          wind gust of 193.1 km/h are entries that were created with one of the
          first attempts to use the history.dat file using WUHU (v132), not the
          history uploader. So these incorrect values should not be a problem
          with your current uploads.

          What I do see is some time of time shift. For instance, based upon
          that link you sent, I see the following temperature readings:

          00:16 54.1 °F (this is uploaded by WUHU)
          03:13 54.1 °F (this is uploaded by history uploader)

          So it appears there is a 3 hour difference here. Why that is, I am
          not sure.

          As I mentioned before, I was not sure what meaning of the timestamps
          are stored in history.dat. Is this really local time? If so, does the
          C runtime correctly compute UTC time from local time?

          I am assuming that the C runtime knows the local time zone, etc, by
          examining the operating systems settings (Control panel\Date and
          Time). This appears to be the case as I have not heard anyone
          complain that their WUHU uploads do not have the correct time stamp.

          Here is the C function I am using to convert the time stamps stored
          in history.dat (double floating point value) to a time_t vale. I am
          not sure if this is valid or not. It looks ok to me.

          time_t TDateTime2time_t( double DateTime )
          {
          const int BASEDELTA = 25569; // Days between TDateTime basis
          (12/30/1899) and time_t basis (1/1/1970)
          const int SECS_PER_DAY = 24 * 60 * 60; // Seconds per day

          // the ( time_t )-cast separates the integer part; fractions of
          seconds are not returned
          return ( time_t ) ( ( DateTime - BASEDELTA ) * SECS_PER_DAY );
          }

          The only other thing that I can think of is that I am not calling a
          function _tzset(). I did not believe it was necessary, but it may be.
          This function will attempt to read the time zone information from the
          operating system (I assumed this was done automatically).

          I suppose what we could do is have you upload a few records from that
          same history.dat file you use and see if the _tzset() call makes a
          difference. You can hit cancel at any time during uploads. Do we know
          the date that your history.dat file was created?

          By the way, is your time zone 3 hours from UTC time (coordinated
          universal time)?

          Thanks.

          --- In wuhu_software_group@yahoogroups.com, "jjantti2"
          <jjantti2@y...> wrote:
          >
          > --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
          > <wuhu_software@y...> wrote:
          >
          > > Since I have never attempted uploading older weather data, I am
          not
          > > sure what happens at WU in the event of duplication. If you abort
          an
          > > upload of the file and later attempt to upload it, I am not sure
          > > what the effect is. Hopefully WU will recognize the duplicates
          and
          > > pitch the old ones.
          > >
          > > Let me know how it goes should you decide to attempt this.
          >
          > I just ran the program and it looks like WU went completely bananas
          > when I dumped my history.dat file to it. There were over 42 thousand
          > records on that file and it took about 36 hours to finish.
          >
          > Have a look at the data for October 8th, 2005:
          >
          > http://www.wunderground.com/weatherstation/WXDailyHistory.asp?
          ID=IUUSIMAA2&month=10&day=8&year=2005
          >
          > You can clearly see spikes on some of the graphs. This is likely
          > because my HeavyWeather software is gathering up weather data every
          > minute. The fun part begins when you scroll down the page to see the
          > tabular data. There are multiple duplicates with the same time and
          > some of the data is plain wrong like those with dew point
          temperature
          > being -17.8°C and the wind gust being 193.1 km/h.
          >
          > The other strange thing is that some of the data is actually
          missing.
          > There are jumps of two minutes (usually after the duplicates). My
          best
          > guess is that WU will only accept data that is sent minimum every
          two
          > minutes and dumping a minute-interval database will cause this
          behavior.
          >
          > Best Regards,
          > Juha
          >
        • wuhu_software
          Juha, Another thought I had. What if the timestamp in history.dat is already a UTC time stamp. If we treat this is local time (which is currently the case),
          Message 4 of 15 , Nov 8, 2005
          • 0 Attachment
            Juha,

            Another thought I had.

            What if the timestamp in history.dat is already a UTC time stamp.

            If we treat this is local time (which is currently the case), and then
            call a function that computes UTC time, then the computed time would be
            off by offset from local time to UTC time.

            That might explain the problem.

            Just for kicks, I will create another version of the history upload to
            test that theory.

            Thanks.
          • wuhu_software
            Scratch that... The current assumption is that the timestamps in history.dat are UTC timestamps, not local timestamps. If these timestamps are local, then a
            Message 5 of 15 , Nov 8, 2005
            • 0 Attachment
              Scratch that...

              The current assumption is that the timestamps in history.dat are UTC
              timestamps, not local timestamps.

              If these timestamps are local, then a bias must be added to the
              timestamps to compensate.

              --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
              <wuhu_software@y...> wrote:
              >
              > Juha,
              >
              > Another thought I had.
              >
              > What if the timestamp in history.dat is already a UTC time stamp.
              >
              > If we treat this is local time (which is currently the case), and
              then
              > call a function that computes UTC time, then the computed time
              would be
              > off by offset from local time to UTC time.
              >
              > That might explain the problem.
              >
              > Just for kicks, I will create another version of the history upload
              to
              > test that theory.
              >
              > Thanks.
              >
            • wuhu_software
              Juha, I have created a new version of the history file uploader (version 2). This will subtract your local time offset from the timestamps (in history.dat) to
              Message 6 of 15 , Nov 8, 2005
              • 0 Attachment
                Juha,

                I have created a new version of the history file uploader (version 2).

                This will subtract your local time offset from the timestamps (in
                history.dat) to create a UTC time stamp.

                Hopefully this will address the problem.

                If you could peform a few uploads (hit cancel after it uploads a few
                records) and let me know the starting time and date of your history.dat
                records (probably by reviewing the history in Heavyweather), we can
                check out the results at WU.

                Thanks.
              • jjantti2
                ... Oops, my bad. Sure. I do know the exact date and time of the history.dat file s creation. The part I forgot to mention was that I had a thought, that WU
                Message 7 of 15 , Nov 9, 2005
                • 0 Attachment
                  --- In wuhu_software_group@yahoogroups.com, "wuhu_software" > same


                  > Do we know the date that your history.dat file was created?

                  Oops, my bad. Sure. I do know the exact date and time of the
                  history.dat file's creation.

                  The part I forgot to mention was that I had a thought, that WU would
                  replace the old weather data with the one being uploaded by
                  HistoryToWU.exe, but apparently I was mistaking.

                  > By the way, is your time zone 3 hours from UTC time (coordinated
                  > universal time)?

                  No. Officially it's +2 hours, but my time zone differs according to
                  Daylight Time change. In Daylight time, we drift +3 hours from UTC and
                  in Standard Time, +2 hours.

                  I have to manually adjust the time because my weather station is
                  unable to synchronize itself with DCF77-signal. This is because the
                  Frankfurt (Germany) transmitter is about 1,500 km away from my
                  location and all the surrounding exterior walls are so thick, they
                  block the signal completely. If I really want to get a signal
                  reception, it's only during the night if the weather station is
                  relocated to my bedroom window.

                  Best Regards,
                  Juha
                • jjantti2
                  ... Thanks! I ll give it try as soon as I locate any missing weather data from my site. I ll report the date and time here. Best Regards, Juha
                  Message 8 of 15 , Nov 9, 2005
                  • 0 Attachment
                    --- In wuhu_software_group@yahoogroups.com, "wuhu_software"
                    <wuhu_software@y...> wrote:
                    >
                    >
                    > Juha,
                    >
                    > I have created a new version of the history file uploader
                    > (version 2).
                    >
                    > This will subtract your local time offset from the timestamps (in
                    > history.dat) to create a UTC time stamp.
                    >
                    > Hopefully this will address the problem.
                    >
                    > If you could peform a few uploads (hit cancel after it uploads a few
                    > records) and let me know the starting time and date of your
                    > history.dat records (probably by reviewing the history in
                    > Heavyweather), we can check out the results at WU.

                    Thanks! I'll give it try as soon as I locate any missing weather data
                    from my site. I'll report the date and time here.

                    Best Regards,
                    Juha
                  Your message has been successfully submitted and would be delivered to recipients shortly.