Loading ...
Sorry, an error occurred while loading the content.

standard content-type for "streaming JSON" (newline delimited JSON objects)?

Expand Messages
  • Ken Woodruff
    There seem to be a growing number of tools & packages around that implement some form of JSON streaming where multiple standard JSON objects are delimited by
    Message 1 of 3 , May 23, 2012
    • 0 Attachment
      There seem to be a growing number of tools & packages around that
      implement some form of JSON streaming where multiple standard JSON
      objects are delimited by extra newlines, e.g.

      { "id": 1, "foo": "bar" }
      { "id": 2, "foo": "baz" }
      ...

      This format seems both pragmatic and useful, but is not JSON compliant
      (i.e. doesn't parse with a standard JSON parser), so it seems
      inappropriate to serve up as "application/json".   The NPM package
      Request-JSONStream uses "application/jsonstream", Google searching
      shows at least one use of "application/x-json-stream", and there are a
      number of services that use "application/json" and expect clients to
      just deal with it (cf.
      https://github.com/senchalabs/connect/issues/538).

      Since I'm about to make heavy use of this technique in a way that will
      be a little difficult to unwind later I'd like if at all possible get
      on board with whatever will become the standard (defacto or official).
      Anyone aware of any efforts underway to standardize this, or
      packages/services that have enough momentum to drive a standard in the
      future?

      --Ken
    • Stephan Beal
      ... Wouldn t a standards-compliant (or sane?) parser stop reading at the end of the first object (implying that the first above is, strictly speaking, standard
      Message 2 of 3 , May 23, 2012
      • 0 Attachment
        On Wed, May 23, 2012 at 11:10 PM, Ken Woodruff <ken.woodruff@...>wrote:

        > **
        >
        > { "id": 1, "foo": "bar" }
        > { "id": 2, "foo": "baz" }
        > ...
        >
        > This format seems both pragmatic and useful, but is not JSON compliant
        > (i.e. doesn't parse with a standard JSON parser)
        >

        Wouldn't a standards-compliant (or sane?) parser stop reading at the end of
        the first object (implying that the first above is, strictly speaking,
        standard JSON with extra crud at the end)?

        If so... adding the convention "if the byte following the root node is a
        newline or \r\n combination, attempt to parse remaining bytes as JSON..."
        might be more or less all you need. Such an extension would need to address
        what to do (or not do) with the consumed 1-2 bytes if they are _not_ \n or
        \r\n, but those are details we'll leave to the spec writers...

        Happy Hacking!

        --
        ----- stephan beal
        http://wanderinghorse.net/home/stephan/
        http://gplus.to/sgbeal


        [Non-text portions of this message have been removed]
      • Matthew Morley
        ... This is came up a while back, we were discussing streams of json over sockets/tcp, in relation to json-rpc streams here:
        Message 3 of 3 , May 23, 2012
        • 0 Attachment
          On Wed, May 23, 2012 at 5:10 PM, Ken Woodruff <ken.woodruff@...>wrote:

          > **
          >
          >
          > There seem to be a growing number of tools & packages around that
          > implement some form of JSON streaming where multiple standard JSON
          > objects are delimited by extra newlines, e.g.
          >
          > { "id": 1, "foo": "bar" }
          > { "id": 2, "foo": "baz" }
          > ...
          >
          > This format seems both pragmatic and useful, but is not JSON compliant
          > (i.e. doesn't parse with a standard JSON parser), so it seems
          > inappropriate to serve up as "application/json". The NPM package
          > Request-JSONStream uses "application/jsonstream", Google searching
          > shows at least one use of "application/x-json-stream", and there are a
          > number of services that use "application/json" and expect clients to
          > just deal with it (cf.
          > https://github.com/senchalabs/connect/issues/538).
          >
          > Since I'm about to make heavy use of this technique in a way that will
          > be a little difficult to unwind later I'd like if at all possible get
          > on board with whatever will become the standard (defacto or official).
          > Anyone aware of any efforts underway to standardize this, or
          > packages/services that have enough momentum to drive a standard in the
          > future?
          >
          > --Ken
          >
          >

          This is came up a while back, we were discussing streams of json over
          sockets/tcp, in relation to json-rpc streams here:

          https://groups.google.com/forum/?fromgroups#!msg/json-rpc/fd1_anYEnDs/5k_w5Vu9XtMJ

          Use of the ascii character 0x04 EOT was also an option others have
          suggested in these discussions.

          Using newlines seem like the same topic, just a different end of json
          marker. There are a lot of ways to achieve this of course, EOL markers seem
          as straight forward of a solution as any. The ability for a single bad json
          object in the stream to corrupt parsing down the line always seemed like a
          concern, perhaps unavoidable and unlikely at the same time?

          I still feel like sending a json array with a parser capable of spinning
          out parsed array values as soon as it can makes the most sense, like
          SAX<http://en.wikipedia.org/wiki/Simple_API_for_XML> based
          parsing. But I'll be glad to see a standard/best practice come of out this
          thread.

          --
          Matt (MPCM)


          [Non-text portions of this message have been removed]
        Your message has been successfully submitted and would be delivered to recipients shortly.