Loading ...
Sorry, an error occurred while loading the content.

Re: JSON.hpack - JavaScript, PHP, C# (Python Soon)

Expand Messages
  • an_red@ymail.com
    Kris, hope I got your question. Basically, as you said gzip is able to reduce drastically redundant words (in JSON case, every collection repeated key) This
    Message 1 of 8 , May 27, 2009
    • 0 Attachment
      Kris, hope I got your question.
      Basically, as you said gzip is able to reduce drastically redundant words (in JSON case, every collection repeated key)

      This simply means that while JSON vs JSON.hpack difference could be up to 80% less characters for the hpack best option, gzipped JSON.hpack against gzipped JSON will be not that consistent at all (0.85~0.15:1.0) so in this case we have different benefits:

      1. JSON.hpack could make runtime gzip usage a plus, rather than a must

      2. JSON.hpack make faster, and in some case "just possible" Client TO Server interaction over homogeneous collection. While gzip is a common practice on the server, in the client side we have almost nothing, except the in-core request inflate/gunzip. In previous posted example, my host does not support more than 50Kb of posted data. This means that is not possible to send back in JSON format the retrieved, and eventually elaborated, big result set, while as soon as the test uses JSON.hpack, the client can quickly send back everything in a shot ( rather than a call for each updated row/field, can you see my point? )

      3. except for Internet Explorer 8, where JSON.parse/stringify are native, JSON.hpack make the entire "create JSON, send, receive and convert to native" faster in every other browser. The reason is simply, Big D did a great job with json2.js but more big is the object or more long is the string, more time it will take to convert, or convert back. You can simply tests results with FireFox2, 3, Opera, Safari beta, or Chrome.

      Accordingly, JSON.hpack best environment is in the client side, where it could significally make the difference but since its JSON.hunpack method is "that simple", there is no reason to do not have at least that one on the whatever server side language, in order to be able to retrieve JSON.hpack data. As summary, potentially faster in the server (with cached results and no gzip runtime compression), always preferable in the client side (still performances over thousands of objects plus "one/shot" client to server possibility reducing delay up to 80%).

      Did I reply to your concerns? Regards.

      --- In json@yahoogroups.com, Kris Zyp <kriszyp@...> wrote:
      >
      > What level of difference is there between raw database result set name
      > value paired that has been gzipped and hunpack'ed data that has been
      > gzipped? I know that gzipping also finds redundancies and reduces them,
      > so I am curious how well this works if you are compressing all your data
      > with gzip already. Anyway, good work.
      > Thanks,
      > Kris
      >
      > Andrea Giammarchi wrote:
      > >
      > >
      > > (it was a direct message, I have copied and pasted here as well)
      > > Good afternoon Mr Douglas Crockford,
      > > I wonder if you have seen already my latest JSON related project, called
      > > JSON.hpack,
      > > an Homogeneous Collection Packer.
      > >
      > > As summary, hpack transform a JSON list of objects, usually a database
      > > resultSet
      > > [{name:value},{name:value},{name:value}]
      > > into an array only collection with columns info in the special index 0
      > >
      > > [[name],[value],[value],[value]]
      > >
      > > With 4 different compression levels, it is able to cut down number of
      > > characters from 75Kb to 9Kb ( no gzip ) and to make big client side
      > > elaborated resultSets send via Ajax possible without having gzip/deflate
      > > compression on client side. The JSON.hunpack method is extremely fast and
      > > compatible with every chosed compression level.
      > >
      > > I hope it will find a place into json.org and I'd love to read a comment
      > > about it as soon as you have some free minute (... even if I know it is
      > > something hard to find in IT).
      > >
      > > Blog Entry:
      > >
      > http://webreflection.blogspot.com/2009/05/re-introduction-to-jsonhpack.html
      > >
      > > Github Project:
      > > http://github.com/WebReflection/json.hpack/tree/master
      > >
      > > Here is a live demo page:
      > > http://www.3site.eu/examples/json.hpack/test/
      > >
      > > Best Regards,
      > > Andrea Giammarchi
      > >
      > > [Non-text portions of this message have been removed]
      > >
      > >
      >
    • Tatu Saloranta
      On Wed, May 27, 2009 at 6:27 AM, Andrea Giammarchi ... This seems like a simple and possibly sensible convention. But in addition to having an implementation,
      Message 2 of 8 , May 27, 2009
      • 0 Attachment
        On Wed, May 27, 2009 at 6:27 AM, Andrea Giammarchi
        <andrea.giammarchi@...> wrote:
        > (it was a direct message, I have copied and pasted here as well)
        > Good afternoon Mr Douglas Crockford,
        > I wonder if you have seen already my latest JSON related project, called
        > JSON.hpack,
        > an Homogeneous Collection Packer.
        >
        > As summary, hpack transform a JSON list of objects, usually a database
        > resultSet
        > [{name:value},{name:value},{name:value}]
        > into an array only collection with columns info in the special index 0

        This seems like a simple and possibly sensible convention.
        But in addition to having an implementation, it would seem most
        interesting to specify the convention. This would allow
        implementations for other languages: verbosity of textual data formats
        matters for many use cases, and json is being used a lot on
        server-to-server communication too.

        Wrt compression: I am sure gzip would yield more additional
        compression, since it would not only eliminate redundancy wrt. names
        (which compress very well obviously) but also for values. It can
        definitely be added as a separate layer.
        But given how CPU hungry gzip is (see
        [http://www.cowtowncoder.com/blog/archives/2009/05/entry_263.html%5d for
        example; for java json, overhead is 3x - 5x basic processing),
        conventions can help a lot, so the two can be complementary; even if
        plain gzip would probably compress things more if size reduction was
        the only goal.

        -+ Tatu +-
      • an_red@ymail.com
        Tatu, gzip works faster in any case over JSON.hpack(ed) stuff for the same reason JSON.stringify / parse is faster over JSON.hpacked stuff, less characters to
        Message 3 of 8 , May 28, 2009
        • 0 Attachment
          Tatu, gzip works faster in any case over JSON.hpack(ed) stuff for the same reason JSON.stringify / parse is faster over JSON.hpacked stuff, less characters to work over (you gzip a string smaller up to 80%)

          It could be a separated layer in any case, but even over, will be faster. The C# implementation of both hpack and hunpack are extremely fast, I need time to create a Python version and I was planning to create a PHP extension in C as well but only one Java guy told me he will try to create a parser ... I mean, C# version is not that different, I guess a porting will take a couple of hours and nothing else.

          Anyway, I miss your point about specify the convention ... do you mean proper specs? I could write something more than I already wrote on github homepage, let me know.

          Regards

          --- In json@yahoogroups.com, Tatu Saloranta <tsaloranta@...> wrote:
          >
          > On Wed, May 27, 2009 at 6:27 AM, Andrea Giammarchi
          > <andrea.giammarchi@...> wrote:
          > > (it was a direct message, I have copied and pasted here as well)
          > > Good afternoon Mr Douglas Crockford,
          > > I wonder if you have seen already my latest JSON related project, called
          > > JSON.hpack,
          > > an Homogeneous Collection Packer.
          > >
          > > As summary, hpack transform a JSON list of objects, usually a database
          > > resultSet
          > > [{name:value},{name:value},{name:value}]
          > > into an array only collection with columns info in the special index 0
          >
          > This seems like a simple and possibly sensible convention.
          > But in addition to having an implementation, it would seem most
          > interesting to specify the convention. This would allow
          > implementations for other languages: verbosity of textual data formats
          > matters for many use cases, and json is being used a lot on
          > server-to-server communication too.
          >
          > Wrt compression: I am sure gzip would yield more additional
          > compression, since it would not only eliminate redundancy wrt. names
          > (which compress very well obviously) but also for values. It can
          > definitely be added as a separate layer.
          > But given how CPU hungry gzip is (see
          > [http://www.cowtowncoder.com/blog/archives/2009/05/entry_263.html%5d for
          > example; for java json, overhead is 3x - 5x basic processing),
          > conventions can help a lot, so the two can be complementary; even if
          > plain gzip would probably compress things more if size reduction was
          > the only goal.
          >
          > -+ Tatu +-
          >
        • Tatu Saloranta
          On Thu, May 28, 2009 at 1:25 AM, an_red@ymail.com ... Yes, certainly, not arguing with that, wrt. speed. ... No doubt, it should be a fairly simple
          Message 4 of 8 , May 28, 2009
          • 0 Attachment
            On Thu, May 28, 2009 at 1:25 AM, an_red@...
            <andrea.giammarchi@...> wrote:
            > Tatu, gzip works faster in any case over JSON.hpack(ed) stuff for the same reason JSON.stringify / parse is faster over JSON.hpacked stuff, less characters to work over (you gzip a string smaller up to 80%)

            Yes, certainly, not arguing with that, wrt. speed.

            > It could be a separated layer in any case, but even over, will be faster. The C# implementation of both hpack and hunpack are extremely fast, I need time to create a Python version and I was planning to create a PHP

            No doubt, it should be a fairly simple transformation on other
            languages as well.

            > extension in C as well but only one Java guy told me he will try to create a parser ... I mean, C# version is not that different, I guess a porting will take a

            There are many good fast Java parsers, so I would recommend using one
            of those, and add transformation layer on top. No need to write a
            parser (IMO -- but I have written mine so I may be biased).

            > Anyway, I miss your point about specify the convention ... do you mean
            > proper specs? I could write something more than I already wrote on github homepage, let me know.

            Yeah, specification of this convention (or transformation),
            documenting how it should behave. This is necessary (or at least
            useful) to ensure interoperability. Given how many people want to
            write their own json parsers (... just a random example...), because
            they don't like what is available, I'm sure there'll be many who'd
            want to reimplement this mapping as well.
            But at least implementations would be interoperable.

            So yes, need not be anything too heavy-weight, but somewhat formal
            notation helps in my opinion.

            Thanks!

            -+ Tatu +-
          • an_red@ymail.com
            Is something like this enough? http://wiki.github.com/WebReflection/json.hpack/specs-details Best Regards. Andrea Giammarchi
            Message 5 of 8 , May 29, 2009
            • 0 Attachment
              Is something like this enough?
              http://wiki.github.com/WebReflection/json.hpack/specs-details

              Best Regards.

              Andrea Giammarchi

              --- In json@yahoogroups.com, Tatu Saloranta <tsaloranta@...> wrote:
              >
              > On Thu, May 28, 2009 at 1:25 AM, an_red@...
              > <andrea.giammarchi@...> wrote:
              > > Tatu, gzip works faster in any case over JSON.hpack(ed) stuff for the same reason JSON.stringify / parse is faster over JSON.hpacked stuff, less characters to work over (you gzip a string smaller up to 80%)
              >
              > Yes, certainly, not arguing with that, wrt. speed.
              >
              > > It could be a separated layer in any case, but even over, will be faster. The C# implementation of both hpack and hunpack are extremely fast, I need time to create a Python version and I was planning to create a PHP
              >
              > No doubt, it should be a fairly simple transformation on other
              > languages as well.
              >
              > > extension in C as well but only one Java guy told me he will try to create a parser ... I mean, C# version is not that different, I guess a porting will take a
              >
              > There are many good fast Java parsers, so I would recommend using one
              > of those, and add transformation layer on top. No need to write a
              > parser (IMO -- but I have written mine so I may be biased).
              >
              > > Anyway, I miss your point about specify the convention ... do you mean
              > > proper specs? I could write something more than I already wrote on github homepage, let me know.
              >
              > Yeah, specification of this convention (or transformation),
              > documenting how it should behave. This is necessary (or at least
              > useful) to ensure interoperability. Given how many people want to
              > write their own json parsers (... just a random example...), because
              > they don't like what is available, I'm sure there'll be many who'd
              > want to reimplement this mapping as well.
              > But at least implementations would be interoperable.
              >
              > So yes, need not be anything too heavy-weight, but somewhat formal
              > notation helps in my opinion.
              >
              > Thanks!
              >
              > -+ Tatu +-
              >
            • Tatu Saloranta
              On Fri, May 29, 2009 at 3:29 AM, an_red@ymail.com ... I think so (didn t read in detail) -- and if there are questions, it s easy to contact you and suggest
              Message 6 of 8 , May 29, 2009
              • 0 Attachment
                On Fri, May 29, 2009 at 3:29 AM, an_red@...
                <andrea.giammarchi@...> wrote:
                > Is something like this enough?
                > http://wiki.github.com/WebReflection/json.hpack/specs-details

                I think so (didn't read in detail) -- and if there are questions, it's
                easy to contact you and suggest clarifications.

                Thanks!

                -+ Tatu +-
              Your message has been successfully submitted and would be delivered to recipients shortly.