Loading ...
Sorry, an error occurred while loading the content.

rounding issue

Expand Messages
  • dlsmith67
    can anybody explain this one, i ve found another work around for now... code: x$ = 574.42 print val( x$ ) print val( x$ ) * 100 print int( val( x$ ) * 100 )
    Message 1 of 6 , Jun 30, 2008
    • 0 Attachment
      can anybody explain this one, i've found another work around for now...

      code:

      x$ = "574.42"
      print val( x$ )
      print val( x$ ) * 100
      print int( val( x$ ) * 100 )

      results:

      574.42
      57442.0
      57441
    • gjbilik
      It gets even more interesting.. code: y=574.42 print y y=y*100 print y print int(y) print y print print y=y*10 print y print int(y) result: 574.42
      Message 2 of 6 , Jun 30, 2008
      • 0 Attachment
        It gets even more interesting..

        code:

        y=574.42
        print y
        y=y*100
        print y
        print int(y)
        print y
        print ""
        print ""
        y=y*10
        print y
        print int(y)


        result:

        574.42
        57442.0
        57441
        57442.0


        574420.0
        574419

        May be a small bug in the int function

        --- In libertybasic@yahoogroups.com, "dlsmith67" <david.smith@...>
        wrote:
        >
        > can anybody explain this one, i've found another work around for
        now...
        >
        > code:
        >
        > x$ = "574.42"
        > print val( x$ )
        > print val( x$ ) * 100
        > print int( val( x$ ) * 100 )
        >
        > results:
        >
        > 574.42
        > 57442.0
        > 57441
        >
      • Richard Russell
        ... This is a Frequently Asked Question! It is not a bug, or in any way strange, it just results from the fact that Liberty Basic (like the vast majority of
        Message 3 of 6 , Jun 30, 2008
        • 0 Attachment
          --- In libertybasic@yahoogroups.com, "gjbilik" <gjbilik@...> wrote:
          > May be a small bug in the int function

          This is a Frequently Asked Question!

          It is not a bug, or in any way strange, it just results from the
          fact that Liberty Basic (like the vast majority of programming
          languages) works internally in BINARY, not DECIMAL.

          Here is what happens internally. You enter the number 574.42 which,
          in order to be meaningful to LB, has to be converted to binary.
          Like the majority of non-integer decimal numbers, 574.42 *cannot be
          represented exactly in binary* so LB converts it to the *closest*
          binary equivalent, which happens to be this:

          1000111110.0110101110000101000111101011100001010001111

          Now you multiply this number by 100 (binary 1100100) giving this:

          1110000001100001.1111111111111111111111111111111111111

          Finally you take the integer part (INT) of this number:

          1110000001100001

          which is 57441 (the result LB prints).

          So this behavior is to be expected, and all languages which work
          internally in binary will produce the same result. A few
          programming languages work internally in BCD (binary-coded decimal)
          and don't suffer from this effect, but they tend to run slower
          because it's harder for the processor to perform BCD calculations
          than binary ones.

          Richard.
        • gjbilik
          Thanks, Richard ... which,
          Message 4 of 6 , Jun 30, 2008
          • 0 Attachment
            Thanks, Richard

            --- In libertybasic@yahoogroups.com, "Richard Russell" <yahoo@...>
            wrote:
            >
            > --- In libertybasic@yahoogroups.com, "gjbilik" <gjbilik@> wrote:
            > > May be a small bug in the int function
            >
            > This is a Frequently Asked Question!
            >
            > It is not a bug, or in any way strange, it just results from the
            > fact that Liberty Basic (like the vast majority of programming
            > languages) works internally in BINARY, not DECIMAL.
            >
            > Here is what happens internally. You enter the number 574.42
            which,
            > in order to be meaningful to LB, has to be converted to binary.
            > Like the majority of non-integer decimal numbers, 574.42 *cannot be
            > represented exactly in binary* so LB converts it to the *closest*
            > binary equivalent, which happens to be this:
            >
            > 1000111110.0110101110000101000111101011100001010001111
            >
            > Now you multiply this number by 100 (binary 1100100) giving this:
            >
            > 1110000001100001.1111111111111111111111111111111111111
            >
            > Finally you take the integer part (INT) of this number:
            >
            > 1110000001100001
            >
            > which is 57441 (the result LB prints).
            >
            > So this behavior is to be expected, and all languages which work
            > internally in binary will produce the same result. A few
            > programming languages work internally in BCD (binary-coded decimal)
            > and don't suffer from this effect, but they tend to run slower
            > because it's harder for the processor to perform BCD calculations
            > than binary ones.
            >
            > Richard.
            >
          • Carl Gundel
            Richard has it right. In addition most programming systems don t actually do floating point math themselves, but they use the computer s built in math
            Message 5 of 6 , Jun 30, 2008
            • 0 Attachment
              Richard has it right. In addition most programming systems don't
              actually do floating point math themselves, but they use the
              computer's built in math coprocessor. If you did find a floating
              point bug it would usually be in the coprocessor.

              -Carl Gundel
              Liberty BASIC for Windows - http://www.libertybasic.com
              Run BASIC, easy web programming - http://www.runbasic.com

              On Jun 30, 2008, at 12:30 PM, "Richard Russell"
              <yahoo@...> wrote:

              > --- In libertybasic@yahoogroups.com, "gjbilik" <gjbilik@...> wrote:
              >> May be a small bug in the int function
              >
              > This is a Frequently Asked Question!
              >
              > It is not a bug, or in any way strange, it just results from the
              > fact that Liberty Basic (like the vast majority of programming
              > languages) works internally in BINARY, not DECIMAL.
              >
              > Here is what happens internally. You enter the number 574.42 which,
              > in order to be meaningful to LB, has to be converted to binary.
              > Like the majority of non-integer decimal numbers, 574.42 *cannot be
              > represented exactly in binary* so LB converts it to the *closest*
              > binary equivalent, which happens to be this:
              >
              > 1000111110.0110101110000101000111101011100001010001111
              >
              > Now you multiply this number by 100 (binary 1100100) giving this:
              >
              > 1110000001100001.1111111111111111111111111111111111111
              >
              > Finally you take the integer part (INT) of this number:
              >
              > 1110000001100001
              >
              > which is 57441 (the result LB prints).
              >
              > So this behavior is to be expected, and all languages which work
              > internally in binary will produce the same result. A few
              > programming languages work internally in BCD (binary-coded decimal)
              > and don't suffer from this effect, but they tend to run slower
              > because it's harder for the processor to perform BCD calculations
              > than binary ones.
              >
              > Richard.
              >
              >
              >
              > ------------------------------------
              >
              > Yahoo! Groups Links
              >
              >
              >
            • itmentor_consultant
              I agree. We had a rounding problem with a Basic program when we installed it on a new machine that had an AMD processor. We sent the machine back to the
              Message 6 of 6 , Jul 1, 2008
              • 0 Attachment
                I agree. We had a rounding problem with a Basic program when we
                installed it on a new machine that had an AMD processor. We sent the
                machine back to the manufacturer with the program and tried to explain
                the problem to them.
                They called and said the machine was ready to be picked up and when I
                went I asked if they had tested the program. They said no, they were
                hardware people. We set the machine up and tested the program and
                found the same problem.
                I got them to change to an Intel processor and the program worked
                perfectly.
                By the way I am using an Intel P4 and when I ran your code I got the
                same answer.

                --- In libertybasic@yahoogroups.com, Carl Gundel <carlg@...> wrote:
                >
                > Richard has it right. In addition most programming systems don't
                > actually do floating point math themselves, but they use the
                > computer's built in math coprocessor. If you did find a floating
                > point bug it would usually be in the coprocessor.
                >
                > -Carl Gundel
                > Liberty BASIC for Windows - http://www.libertybasic.com
                > Run BASIC, easy web programming - http://www.runbasic.com
                >
                > On Jun 30, 2008, at 12:30 PM, "Richard Russell"
                > <yahoo@...> wrote:
                >
                > > --- In libertybasic@yahoogroups.com, "gjbilik" <gjbilik@> wrote:
                > >> May be a small bug in the int function
                > >
                > > This is a Frequently Asked Question!
                > >
                > > It is not a bug, or in any way strange, it just results from the
                > > fact that Liberty Basic (like the vast majority of programming
                > > languages) works internally in BINARY, not DECIMAL.
                > >
                > > Here is what happens internally. You enter the number 574.42 which,
                > > in order to be meaningful to LB, has to be converted to binary.
                > > Like the majority of non-integer decimal numbers, 574.42 *cannot be
                > > represented exactly in binary* so LB converts it to the *closest*
                > > binary equivalent, which happens to be this:
                > >
                > > 1000111110.0110101110000101000111101011100001010001111
                > >
                > > Now you multiply this number by 100 (binary 1100100) giving this:
                > >
                > > 1110000001100001.1111111111111111111111111111111111111
                > >
                > > Finally you take the integer part (INT) of this number:
                > >
                > > 1110000001100001
                > >
                > > which is 57441 (the result LB prints).
                > >
                > > So this behavior is to be expected, and all languages which work
                > > internally in binary will produce the same result. A few
                > > programming languages work internally in BCD (binary-coded decimal)
                > > and don't suffer from this effect, but they tend to run slower
                > > because it's harder for the processor to perform BCD calculations
                > > than binary ones.
                > >
                > > Richard.
                > >
                > >
                > >
                > > ------------------------------------
                > >
                > > Yahoo! Groups Links
                > >
                > >
                > >
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.