Loading ...
Sorry, an error occurred while loading the content.

Re: [json] Re: Universal Binary JSON Specification

Expand Messages
  • Don Owens
    I ve seen very large numbers used in JSON. In Perl, that can be represented as a Math::BigInt object. And that is the way I have implemented it in my JSON
    Message 1 of 76 , Sep 21, 2011
    View Source
    • 0 Attachment
      I've seen very large numbers used in JSON. In Perl, that can be represented
      as a Math::BigInt object. And that is the way I have implemented it in my
      JSON module for Perl (JSON::DWIW). Python has arbitrary length integers
      built-in. For my own language that I'm working on, I'm using libgmp in C to
      handle arbitrary length integers.

      JSON is used as a data exchange format. I want to be able to do a
      roundtrip, e.g., Python -> encoded -> Python with native integers (with
      arbitrary length in this case). In JSON, this just works, as far as the
      encoding is concerned. I see the need for this in any binary JSON format as
      well. If a large number is represented as a string, then on the decoding
      side, you don't know if that was a number or a string (just because it looks
      like a number doesn't mean that the sender means it's a number). If, when
      decoding JSON, the library can't handle large numbers, it has to throw an
      error anyway. The same should go for binary JSON.

      ./don


      On Wed, Sep 21, 2011 at 11:58 AM, rkalla123 <rkalla@...> wrote:

      > **
      >
      >
      > Don,
      > Interesting point. Stephan and I had a discussion this morning about the
      > portability of the numeric types across most (all) platforms breaking
      > down when it comes to 64-bit integers and JavaScript/C89. Going beyond
      > that to add an arbitrary length number seems to me like the same
      > concerns apply, only more so.
      > I think one of the beauties of JSON was that every construct presented
      > in the spec could be modeled in every language easily and immediately. I
      > have very little familiarity with arbitrarily long integer and decimal
      > numbers, but can the same be said for them and all the languages making
      > heavy use of JSON?
      > I think as a workaround to this, people utilizing the binary spec and
      > working with > 64-bit numbers would be able to store and transfer the
      > values as Strings. Maybe not optimal, but me expanding the spec without
      > a clearer understanding of the goals and compatibility implications
      > would be worse.
      > Thank you for surfacing this issue.
      >
      > --- In json@yahoogroups.com, Don Owens <don@...> wrote:
      > >
      > > What happens if you need to encode an integer larger than 64-bits?
      > > Shouldn't there be a way to encode a large integer as a buffer with a
      > byte
      > > length? I don't think the JSON spec puts a limit on the size of an
      > integer.
      > > If there is no way to encode large integers in your spec, people will
      > > invent their own way of doing it -- it's better to have a standard way
      > of
      > > doing it. I think you should just have another type, e.g., bigint,
      > that is
      > > something like a byte count (the way string has) and a set of bytes in
      > > big-endian order representing the integer.
      > >
      > > ./don
      > >
      > > On Wed, Sep 21, 2011 at 6:58 AM, rkalla123 rkalla@... wrote:
      > >
      > > > **
      > > >
      > > >
      > > >
      > > >
      > > > Stephan,
      > > >
      > > > Great feedback, these are exactly the kind of nuances I wanted to
      > uncover
      > > > and discuss.
      > > >
      > > > When I dug through the language specs looking for the most
      > well-supported
      > > > numeric types before settling on the given 4, it seemed of all the
      > ones I
      > > > checked JavaScript was the only one that didn't have native int64
      > support as
      > > > you mentioned.
      > > >
      > > > The Chrome team suggests treating it like two int32s:
      > > > http://code.google.com/p/v8/issues/detail?id=1339
      > > >
      > > > I found no way to deserialize JavaScript objects directly from a
      > > > server-provided byte stream which I anticipate meaning that this
      > binary
      > > > format doesn't benefit users in the server-to-Browser communication
      > workflow
      > > > (which is already highly optimized for text-based JSON) but as a
      > general
      > > > application binary interchange format. Like values stored as files
      > on the
      > > > server or processes communicating with one another.
      > > >
      > > > It was my thinking that this leaves the int64 inclusion in the
      > binary spec
      > > > in a relatively safe position as the primary use case will be
      > between
      > > > languages that support 64-bit ints.
      > > >
      > > > Said another way, I certainly see your point, but I am trying to
      > avoid a
      > > > "cut off nose despite your face" situation with forward-thinking
      > > > enhancements to JavaScript engines in the next few years that I
      > assume will
      > > > eventually give us 64-bit integers as the JS spec advances.
      > > >
      > > > I would hate then to be prompted to add 64-bit integers back to the
      > binary
      > > > spec after it has been out and in circulation for a few years.
      > > >
      > > > ASIDE: If I am missing a native way to reconstitute JavaScript
      > objects from
      > > > a server-provided byte stream and using this binary format for
      > > > server-browser communication is an optimized reality, please correct
      > me. I
      > > > was unable to dig up methods to do this.
      > > >
      > > > --- In json@yahoogroups.com, Stephan Beal sgbeal@ wrote:
      > > >
      > > > >
      > > > > On Wed, Sep 21, 2011 at 3:15 PM, rkalla123 rkalla@ wrote:
      > > > >
      > > > > > **
      > > > > >
      > > > > > The only difference from JSON being that "Number" is broken out
      > into:
      > > > > > int32, int64 and double types for the purposes of making parsing
      > of the
      > > > > >
      > > > >
      > > > > Keep in mind that JSON does not specify any required numeric
      > precision,
      > > > and
      > > > > some platforms cannot use int64. (JavaScript specifies 53 bits of
      > > > precision,
      > > > > in case it matters.) e.g. in C89 there is no _portable_ int64
      > construct
      > > > > (that was introduced with C99, but lots of projects still
      > use/require C89
      > > > > because of the very different levels of C99 support in various
      > > > compilers). i
      > > > > know that Java is everyone's special baby, but some of us actually
      > write
      > > > > JSON-consuming/producing C89 code. In the world of C++, Google's
      > v8
      > > > > JavaScript engine doesn't support 64-bit integers: numbers >32
      > bits need
      > > > to
      > > > > be doubles on that platform.
      > > > >
      > > > > In any case, i currently have a use case which will eventually
      > require
      > > > some
      > > > > type of binary support, and i will be reading through what you've
      > posted.
      > > > > Thanks for sharing :).
      > > > >
      > > > > Happy Hacking!
      > > > >
      > > > >
      > > > > --
      > > > > ----- stephan beal
      > > > > http://wanderinghorse.net/home/stephan/
      > > > >
      > > > >
      > > > > [Non-text portions of this message have been removed]
      > > > >
      > > >
      > > >
      > > >
      > >
      > >
      > >
      > > --
      > > Don Owens
      > > don@...
      >
      > >
      > >
      > > [Non-text portions of this message have been removed]
      > >
      >
      > [Non-text portions of this message have been removed]
      >
      >
      >



      --
      Don Owens
      don@...


      [Non-text portions of this message have been removed]
    • Tatu Saloranta
      ... For what it is worth, I also consider support for only signed values a good thing. -+ Tatu +-
      Message 76 of 76 , Feb 20, 2012
      View Source
      • 0 Attachment
        On Mon, Feb 20, 2012 at 9:42 AM, rkalla123 <rkalla@...> wrote:
        > Stephan,
        >
        > No problem; your feedback are still very applicable and much appreciated.
        >
        > The additional view-point on the signed/unsigned issue was exactly what I was hoping for. My primary goal has always been simplicity and I know at least from the Java world, going with unsigned values would have made the impl distinctly *not* simple (and an annoying API).
        >
        > So I am glad to get some validation there that I am not alienating every other language at the cost of Java.

        For what it is worth, I also consider support for only signed values a
        good thing.

        -+ Tatu +-
      Your message has been successfully submitted and would be delivered to recipients shortly.