Loading ...
Sorry, an error occurred while loading the content.
 

RE: [libertybasic] Reading binary files

Expand Messages
  • Tim Hewitt
    Dave, Many thanks for your quick reply and helpful information. I had some spare time today so I tried some ideas based on what you suggest. However, whilst I
    Message 1 of 11 , Jun 18, 2012
      Dave,



      Many thanks for your quick reply and helpful information.



      I had some spare time today so I tried some ideas based on what you suggest.
      However, whilst I can now get more ordered results I still cannot make sense
      of them ! I am sure this is me not really understanding my problem, mainly
      because I do not have a detailed sample of what the file contents should
      look like. I will keep trying.



      Regards and thanks again,



      Tim



      From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
      Behalf Of David Speck
      Sent: 18 June 2012 13:11
      To: libertybasic@yahoogroups.com
      Subject: Re: [libertybasic] Reading binary files





      Tim,

      If these are numeric values, you have to read them into a numeric variable.

      You also have to adjust the byte values for their position.

      I will leave it to you to get the values for each high and low byte.

      If you read the file as binary, then you will be retrieving numeric
      variables, so the pseudocode below will work directly.

      You can also access the data file in memory as one long ASCII string,
      and use the MID$() function to pull the bytes out one at a time.
      However, this way, LB will see the bytes as ASCII characters which you
      will have to convert back to numeric values with the LB builtin ASC()
      function.
      e.g.

      Highbyte = ASC(MID$(DataFile$,Index,1))
      Lowbyte = ASC(MID$(DataFile$,Index+1,1))

      If they were 16 bit unsigned values, the pseudocode would be:

      WordValue = 256*Highbyte + Lowbyte

      Usually, in signed 16 bit numbers, if the most significant bit of the
      most significant byte is a "1", then it is a negative number. I
      apologize that I've never quite gotten the hang of the two's complement
      math that would turn the words with the most significant bit set into
      negative numbers, but that should get you started.

      Note that LB has an intrinsic limit of about 70 megabytes total for data
      and program in memory at one time. With an input array of your size,
      plus your programming and actually doing something with your input, you
      can easily bump into this limit.

      Dave

      On 6/18/2012 6:18 AM, Tim Hewitt wrote:
      > Can anyone help please ? I have some large data files which, whatever I
      try,
      > I cannot read into LB4.0.4.
      >
      >
      >
      > The files (~57MB) are described as .binary, 16-bit signed integers, with
      > the bytes in "big-endian" (i.e. MSB first) order.
      >
      >
      >
      > I am aware the byte order needs reversing for PCs etc., but it should
      still
      > be possible to read the bytes in individually. I have tried to read them
      one
      > byte at a time by opening the files .for binary. (and alternatively .for
      > input. ) and using e.g. a$=input$(#h, 1), but to no avail. The files seem
      > to open OK, but I do not get meaningful data. I have tried several other
      > input possibilities without success.
      >
      >
      >
      > If I can get the individual bytes in then it should be straightforward to
      > reverse the byte order etc., but I cannot get past the first hurdle.
      >
      >
      >
      > The public domain data is from a US Govt. web site so I am confident it is
      > correct.
      >
      >
      >
      > If anyone can help I would be very grateful.



      No virus found in this message.
      Checked by AVG - www.avg.com
      Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12



      [Non-text portions of this message have been removed]
    • Tim Hewitt
      Rod, Fortunately the header problem does not arise as the header information is in separate header files (several different header formats). The 30arc second
      Message 2 of 11 , Jun 18, 2012
        Rod,



        Fortunately the header problem does not arise as the header information is
        in separate header files (several different header formats). The 30arc
        second grid global terrain height data is clearly intended for GIS use, but
        I do not have access to ArcGIS or similar in order to convert to ASCII.



        The data is well specified online by the GTOPO30 community who put the data
        together, but for the actual data files it only really gives the info I
        provided in my original e-mail. All can be seen at
        http://www1.gsi.go.jp/geowww/globalmap-gsi/gtopo30/README.html.



        The data itself is available via Google by entering GTOPO30.



        Regards



        Tim





        From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
        Behalf Of Rod
        Sent: 18 June 2012 16:41
        To: libertybasic@yahoogroups.com
        Subject: [libertybasic] Re: Reading binary files





        There may also be a file header to skip past, is there a file spec on line?

        --- In libertybasic@yahoogroups.com <mailto:libertybasic%40yahoogroups.com>
        , "Tim Hewitt" <tim.hewitt@...> wrote:
        >
        > Can anyone help please ? I have some large data files which, whatever I
        try,
        > I cannot read into LB4.0.4.
        >
        >
        >
        > The files (~57MB) are described as .binary, 16-bit signed integers, with
        > the bytes in "big-endian" (i.e. MSB first) order.
        >
        >
        >
        > I am aware the byte order needs reversing for PCs etc., but it should
        still
        > be possible to read the bytes in individually. I have tried to read them
        one
        > byte at a time by opening the files .for binary. (and alternatively .for
        > input. ) and using e.g. a$=input$(#h, 1), but to no avail. The files seem
        > to open OK, but I do not get meaningful data. I have tried several other
        > input possibilities without success.
        >
        >
        >
        > If I can get the individual bytes in then it should be straightforward to
        > reverse the byte order etc., but I cannot get past the first hurdle.
        >
        >
        >
        > The public domain data is from a US Govt. web site so I am confident it is
        > correct.
        >
        >
        >
        > If anyone can help I would be very grateful.
        >
        >
        >
        >
        >
        >
        >
        > [Non-text portions of this message have been removed]
        >



        No virus found in this message.
        Checked by AVG - www.avg.com
        Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12



        [Non-text portions of this message have been removed]
      • Rod
        I am finding conflicting descriptions. Some speak of 16bit signed data others of 1024 byte blocks ABC. The more info you can give us about the origin of the
        Message 3 of 11 , Jun 18, 2012
          I am finding conflicting descriptions. Some speak of 16bit signed data others of 1024 byte blocks ABC. The more info you can give us about the origin of the actual file you are reading the better.



          --- In libertybasic@yahoogroups.com, "Tim Hewitt" <tim.hewitt@...> wrote:
          >
          > Rod,
          >
          >
          >
          > Fortunately the header problem does not arise as the header information is
          > in separate header files (several different header formats). The 30arc
          > second grid global terrain height data is clearly intended for GIS use, but
          > I do not have access to ArcGIS or similar in order to convert to ASCII.
          >
          >
          >
          > The data is well specified online by the GTOPO30 community who put the data
          > together, but for the actual data files it only really gives the info I
          > provided in my original e-mail. All can be seen at
          > http://www1.gsi.go.jp/geowww/globalmap-gsi/gtopo30/README.html.
          >
          >
          >
          > The data itself is available via Google by entering GTOPO30.
          >
          >
          >
          > Regards
          >
          >
          >
          > Tim
          >
          >
          >
          >
          >
          > From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
          > Behalf Of Rod
          > Sent: 18 June 2012 16:41
          > To: libertybasic@yahoogroups.com
          > Subject: [libertybasic] Re: Reading binary files
          >
          >
          >
          >
          >
          > There may also be a file header to skip past, is there a file spec on line?
          >
          > --- In libertybasic@yahoogroups.com <mailto:libertybasic%40yahoogroups.com>
          > , "Tim Hewitt" <tim.hewitt@> wrote:
          > >
          > > Can anyone help please ? I have some large data files which, whatever I
          > try,
          > > I cannot read into LB4.0.4.
          > >
          > >
          > >
          > > The files (~57MB) are described as .binary, 16-bit signed integers, with
          > > the bytes in "big-endian" (i.e. MSB first) order.
          > >
          > >
          > >
          > > I am aware the byte order needs reversing for PCs etc., but it should
          > still
          > > be possible to read the bytes in individually. I have tried to read them
          > one
          > > byte at a time by opening the files .for binary. (and alternatively .for
          > > input. ) and using e.g. a$=input$(#h, 1), but to no avail. The files seem
          > > to open OK, but I do not get meaningful data. I have tried several other
          > > input possibilities without success.
          > >
          > >
          > >
          > > If I can get the individual bytes in then it should be straightforward to
          > > reverse the byte order etc., but I cannot get past the first hurdle.
          > >
          > >
          > >
          > > The public domain data is from a US Govt. web site so I am confident it is
          > > correct.
          > >
          > >
          > >
          > > If anyone can help I would be very grateful.
          > >
          > >
          > >
          > >
          > >
          > >
          > >
          > > [Non-text portions of this message have been removed]
          > >
          >
          >
          >
          > No virus found in this message.
          > Checked by AVG - www.avg.com
          > Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12
          >
          >
          >
          > [Non-text portions of this message have been removed]
          >
        • David Speck
          Tim, You can cut and past your program into one of these eMails and several folks on the list can take a look at it and make suggestions. Dave
          Message 4 of 11 , Jun 18, 2012
            Tim,

            You can cut and past your program into one of these eMails and several
            folks on the list can take a look at it and make suggestions.

            Dave
            On 6/18/2012 2:26 PM, Tim Hewitt wrote:
            > Dave,
            >
            > Many thanks for your quick reply and helpful information.
            >
            > I had some spare time today so I tried some ideas based on what you suggest.
            > However, whilst I can now get more ordered results I still cannot make sense
            > of them ! I am sure this is me not really understanding my problem, mainly
            > because I do not have a detailed sample of what the file contents should
            > look like. I will keep trying.
            >
            > Regards and thanks again,
            >
            > Tim
          • mr.john.f@gmail.com
            Thanks for introducing me to this rich data set of global elevations. I ve been away from coding for two months & am concentrating on my RaspberryPi & Python
            Message 5 of 11 , Jun 20, 2012
              Thanks for introducing me to this rich data set of global elevations.
              I've been away from coding for two months & am concentrating on my RaspberryPi & Python at present!

              You could look at http://diga.me.uk/elevation.html to see how I read the data in LB for the UK & represented it on a colour scale from blue to red representing 0 to around 1000m.

              I'd strongly recommend using a hex editor to examine data like this- you quickly see the format. But I don't see why they state that -9999 is sea-level/no data but actually use hex d8f1...

              I'll have another look at better colour coding- the 'fake 'spectrum' range often used for met. data would look a lot better. Or a 3D representation.
            Your message has been successfully submitted and would be delivered to recipients shortly.