## RE: [libertybasic] Reading binary files

Expand Messages
• Dave, Many thanks for your quick reply and helpful information. I had some spare time today so I tried some ideas based on what you suggest. However, whilst I
Message 1 of 11 , Jun 18, 2012
Dave,

I had some spare time today so I tried some ideas based on what you suggest.
However, whilst I can now get more ordered results I still cannot make sense
of them ! I am sure this is me not really understanding my problem, mainly
because I do not have a detailed sample of what the file contents should
look like. I will keep trying.

Regards and thanks again,

Tim

From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
Behalf Of David Speck
Sent: 18 June 2012 13:11
To: libertybasic@yahoogroups.com
Subject: Re: [libertybasic] Reading binary files

Tim,

If these are numeric values, you have to read them into a numeric variable.

You also have to adjust the byte values for their position.

I will leave it to you to get the values for each high and low byte.

If you read the file as binary, then you will be retrieving numeric
variables, so the pseudocode below will work directly.

You can also access the data file in memory as one long ASCII string,
and use the MID\$() function to pull the bytes out one at a time.
However, this way, LB will see the bytes as ASCII characters which you
will have to convert back to numeric values with the LB builtin ASC()
function.
e.g.

Highbyte = ASC(MID\$(DataFile\$,Index,1))
Lowbyte = ASC(MID\$(DataFile\$,Index+1,1))

If they were 16 bit unsigned values, the pseudocode would be:

WordValue = 256*Highbyte + Lowbyte

Usually, in signed 16 bit numbers, if the most significant bit of the
most significant byte is a "1", then it is a negative number. I
apologize that I've never quite gotten the hang of the two's complement
math that would turn the words with the most significant bit set into
negative numbers, but that should get you started.

Note that LB has an intrinsic limit of about 70 megabytes total for data
and program in memory at one time. With an input array of your size,
can easily bump into this limit.

Dave

On 6/18/2012 6:18 AM, Tim Hewitt wrote:
> Can anyone help please ? I have some large data files which, whatever I
try,
> I cannot read into LB4.0.4.
>
>
>
> The files (~57MB) are described as .binary, 16-bit signed integers, with
> the bytes in "big-endian" (i.e. MSB first) order.
>
>
>
> I am aware the byte order needs reversing for PCs etc., but it should
still
> be possible to read the bytes in individually. I have tried to read them
one
> byte at a time by opening the files .for binary. (and alternatively .for
> input. ) and using e.g. a\$=input\$(#h, 1), but to no avail. The files seem
> to open OK, but I do not get meaningful data. I have tried several other
> input possibilities without success.
>
>
>
> If I can get the individual bytes in then it should be straightforward to
> reverse the byte order etc., but I cannot get past the first hurdle.
>
>
>
> The public domain data is from a US Govt. web site so I am confident it is
> correct.
>
>
>
> If anyone can help I would be very grateful.

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12

[Non-text portions of this message have been removed]
• Rod, Fortunately the header problem does not arise as the header information is in separate header files (several different header formats). The 30arc second
Message 2 of 11 , Jun 18, 2012
Rod,

Fortunately the header problem does not arise as the header information is
second grid global terrain height data is clearly intended for GIS use, but
I do not have access to ArcGIS or similar in order to convert to ASCII.

The data is well specified online by the GTOPO30 community who put the data
together, but for the actual data files it only really gives the info I
provided in my original e-mail. All can be seen at

The data itself is available via Google by entering GTOPO30.

Regards

Tim

From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
Behalf Of Rod
Sent: 18 June 2012 16:41
To: libertybasic@yahoogroups.com
Subject: [libertybasic] Re: Reading binary files

There may also be a file header to skip past, is there a file spec on line?

--- In libertybasic@yahoogroups.com <mailto:libertybasic%40yahoogroups.com>
, "Tim Hewitt" <tim.hewitt@...> wrote:
>
> Can anyone help please ? I have some large data files which, whatever I
try,
> I cannot read into LB4.0.4.
>
>
>
> The files (~57MB) are described as .binary, 16-bit signed integers, with
> the bytes in "big-endian" (i.e. MSB first) order.
>
>
>
> I am aware the byte order needs reversing for PCs etc., but it should
still
> be possible to read the bytes in individually. I have tried to read them
one
> byte at a time by opening the files .for binary. (and alternatively .for
> input. ) and using e.g. a\$=input\$(#h, 1), but to no avail. The files seem
> to open OK, but I do not get meaningful data. I have tried several other
> input possibilities without success.
>
>
>
> If I can get the individual bytes in then it should be straightforward to
> reverse the byte order etc., but I cannot get past the first hurdle.
>
>
>
> The public domain data is from a US Govt. web site so I am confident it is
> correct.
>
>
>
> If anyone can help I would be very grateful.
>
>
>
>
>
>
>
> [Non-text portions of this message have been removed]
>

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12

[Non-text portions of this message have been removed]
• I am finding conflicting descriptions. Some speak of 16bit signed data others of 1024 byte blocks ABC. The more info you can give us about the origin of the
Message 3 of 11 , Jun 18, 2012
I am finding conflicting descriptions. Some speak of 16bit signed data others of 1024 byte blocks ABC. The more info you can give us about the origin of the actual file you are reading the better.

--- In libertybasic@yahoogroups.com, "Tim Hewitt" <tim.hewitt@...> wrote:
>
> Rod,
>
>
>
> Fortunately the header problem does not arise as the header information is
> second grid global terrain height data is clearly intended for GIS use, but
> I do not have access to ArcGIS or similar in order to convert to ASCII.
>
>
>
> The data is well specified online by the GTOPO30 community who put the data
> together, but for the actual data files it only really gives the info I
> provided in my original e-mail. All can be seen at
>
>
>
> The data itself is available via Google by entering GTOPO30.
>
>
>
> Regards
>
>
>
> Tim
>
>
>
>
>
> From: libertybasic@yahoogroups.com [mailto:libertybasic@yahoogroups.com] On
> Behalf Of Rod
> Sent: 18 June 2012 16:41
> To: libertybasic@yahoogroups.com
> Subject: [libertybasic] Re: Reading binary files
>
>
>
>
>
> There may also be a file header to skip past, is there a file spec on line?
>
> --- In libertybasic@yahoogroups.com <mailto:libertybasic%40yahoogroups.com>
> , "Tim Hewitt" <tim.hewitt@> wrote:
> >
> > Can anyone help please ? I have some large data files which, whatever I
> try,
> > I cannot read into LB4.0.4.
> >
> >
> >
> > The files (~57MB) are described as .binary, 16-bit signed integers, with
> > the bytes in "big-endian" (i.e. MSB first) order.
> >
> >
> >
> > I am aware the byte order needs reversing for PCs etc., but it should
> still
> > be possible to read the bytes in individually. I have tried to read them
> one
> > byte at a time by opening the files .for binary. (and alternatively .for
> > input. ) and using e.g. a\$=input\$(#h, 1), but to no avail. The files seem
> > to open OK, but I do not get meaningful data. I have tried several other
> > input possibilities without success.
> >
> >
> >
> > If I can get the individual bytes in then it should be straightforward to
> > reverse the byte order etc., but I cannot get past the first hurdle.
> >
> >
> >
> > The public domain data is from a US Govt. web site so I am confident it is
> > correct.
> >
> >
> >
> > If anyone can help I would be very grateful.
> >
> >
> >
> >
> >
> >
> >
> > [Non-text portions of this message have been removed]
> >
>
>
>
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.2178 / Virus Database: 2433/5076 - Release Date: 06/17/12
>
>
>
> [Non-text portions of this message have been removed]
>
• Tim, You can cut and past your program into one of these eMails and several folks on the list can take a look at it and make suggestions. Dave
Message 4 of 11 , Jun 18, 2012
Tim,

You can cut and past your program into one of these eMails and several
folks on the list can take a look at it and make suggestions.

Dave
On 6/18/2012 2:26 PM, Tim Hewitt wrote:
> Dave,
>
>
> I had some spare time today so I tried some ideas based on what you suggest.
> However, whilst I can now get more ordered results I still cannot make sense
> of them ! I am sure this is me not really understanding my problem, mainly
> because I do not have a detailed sample of what the file contents should
> look like. I will keep trying.
>
> Regards and thanks again,
>
> Tim
• Thanks for introducing me to this rich data set of global elevations. I ve been away from coding for two months & am concentrating on my RaspberryPi & Python
Message 5 of 11 , Jun 20, 2012
Thanks for introducing me to this rich data set of global elevations.
I've been away from coding for two months & am concentrating on my RaspberryPi & Python at present!

You could look at http://diga.me.uk/elevation.html to see how I read the data in LB for the UK & represented it on a colour scale from blue to red representing 0 to around 1000m.

I'd strongly recommend using a hex editor to examine data like this- you quickly see the format. But I don't see why they state that -9999 is sea-level/no data but actually use hex d8f1...

I'll have another look at better colour coding- the 'fake 'spectrum' range often used for met. data would look a lot better. Or a 3D representation.
Your message has been successfully submitted and would be delivered to recipients shortly.