Loading ...
Sorry, an error occurred while loading the content.

Re: How to handle big soap attachments ?

Expand Messages
  • allierogers@yahoo.com
    Paul, You make many good points, and I see the issues. Maybe we need to decompose the problem into smaller spaces. As I see it, there are 2 basic issues
    Message 1 of 13 , Mar 22, 2001
    • 0 Attachment
      Paul,

      You make many good points, and I see the issues. Maybe we need to
      decompose the problem into smaller spaces. As I see it, there are 2
      basic issues related to large SOAP RPC calls:

      1. SOAP request where arguments are large

      2. SOAP response where method result is large

      I thought the proper way to handle number 1 was to use the HTTP file
      upload capability where the SOAP argument references the data in the
      HTTP upload or via URL somehow. Maybe I am wrong about this because
      my particular implementations never make use of this feature. I
      realize this is an HTTP-centric approach, but the scalability issues
      really revolve around HTTP implementations and not so much in SMTP
      and FTP where disk rather than RAM storage is the default and
      dispatch and execution threads are not as memory sensitive as HTTP
      (for instance, an SMTP server is only handling one message at a time
      while an HTTP server may be handling 100,000 requests at once).

      For me, number 2 is more important. In this case, the method call is
      simple and its data is relatively small, so how you parse and handle
      the request on the server side is fine as it is. However, the
      response should be streamed in all cases. There is no issue of XML
      parsing on the server side in this case. But, the server should
      assume that the method return could be arbitrarily large, so it
      should not attempt to receive the entire method return before it
      starts passing it back to the client. The server should start, right
      away, streaming back the SOAP envelope, unbuffered, or at least allow
      this as a setting. For SOAP::Lite, you have the problem that you are
      dynamically trying to figure out what types are in the method
      response, rather than through static definition (e.g., a
      configuration file on the server to map method name, namespace, uri,
      arugments, return types, etc.). Without a static configuration,
      there is no way to stream as I would like. Maybe it could be a
      performance option?

      This is how we have solved the problem, here, both for SOAP and XML-
      RPC. However, we use SOAP::Lite servers only for some prototyping.
      In other cases, we have COM-based SOAP/XML-RPC servers (e.g., 4s4c
      from pocketsoap.com) and IIS configured for unbuffered response.

      Regards,

      Allie
    • allierogers@yahoo.com
      Petr, I know Don is an important figure, but I disagree with him. SOAP can be used for large data, just as all HTTP servers today handle large data. We use
      Message 2 of 13 , Mar 22, 2001
      • 0 Attachment
        Petr,

        I know Don is an important figure, but I disagree with him. SOAP can
        be used for large data, just as all HTTP servers today handle large
        data. We use it for this purpose here in all of our products and it
        works well. Maybe he did not intend this use of SOAP, but it does
        work. However, SOAP::Lite, currently, can not be used in this way.
        Perhaps that may change.

        Allie

        --- In soaplite@y..., "Petr Janata" <petr.janata@i...> wrote:
        > Hello,
        >
        > I would just like to remark that we had Don Box last week in Prague
        giving a
        > SOAP talk and he said that SOAP is meant to work exactly like
        this, i.e.
        > not to handle large amounts of data. He also said that if you need
        to do
        > that you can e.g. pass just the URL in a SOAP message and use simple
        > download (or SAX parser ) to handle the transfer.
        >
        > Petr Janata
      • sebaklu@yahoo.com
        Hi Paul, That s right, but the current version of SOAP::Lite should never expect large requests. The server will give no response and fill out the complete
        Message 3 of 13 , Mar 22, 2001
        • 0 Attachment
          Hi Paul,

          That's right, but the current version of SOAP::Lite should never
          expect large requests. The server will give no response and fill out
          the complete memory on the machine. Since it expect SOAP messages
          with attachements it should be able to handle large amount of data.

          However, it works fine with simple requests. But general for
          handling SOAP messages with attachments (the 7 MB attachment was an
          example, i had also problems to handle smaller attachments) should
          use stream mechanism. If not you should not read it into memory but
          reject the request. Maybe i'm wrong but it is a weak point and DOS
          attacks may use it.


          Sebastian

          --- In soaplite@y..., Paul Kulchenko <paulclinger@y...> wrote:
          > Hi, Sebastian!
          >
          > That's true, but at the same time it's easy to imagine situation
          when
          > you send something not directly, but thru the several different
          > intermediaries and each of them will need to handle this huge
          > request. If this piece is encoded as external reference then handler
          > could be smart enough to get it only if it's required (yet I don't
          > know about such smart handlers :)). Ideas, ideas...
          >
          > Ideally implementation should be flexible enough to handle both (and
          > maybe man others) approaches, maybe with manual hints.
          >
          > Best wishes, Paul.
        • Paul Kulchenko
          Hi, Sebastian! Absolutely agree. That s the reason why I want to introduce some additional transport options, like ACCEPTABLE_CONTENT_TYPE (if you want to
          Message 4 of 13 , Mar 22, 2001
          • 0 Attachment
            Hi, Sebastian!

            Absolutely agree. That's the reason why I want to introduce some
            additional transport options, like ACCEPTABLE_CONTENT_TYPE (if you
            want to accept ONLY text/xml or multipart/related) and
            MAX_CONTENT_SIZE that should take care about it and request will be
            rejected. As for DOS attack it could be introduced even with small
            request which has complex XML structure. Anyway, these options should
            make server side more robust.

            Best wishes, Paul.

            --- sebaklu@... wrote:
            > Hi Paul,
            >
            > That's right, but the current version of SOAP::Lite should never
            > expect large requests. The server will give no response and fill
            > out
            > the complete memory on the machine. Since it expect SOAP messages
            > with attachements it should be able to handle large amount of data.
            >
            > However, it works fine with simple requests. But general for
            > handling SOAP messages with attachments (the 7 MB attachment was
            > an
            > example, i had also problems to handle smaller attachments) should
            > use stream mechanism. If not you should not read it into memory but
            >
            > reject the request. Maybe i'm wrong but it is a weak point and DOS
            > attacks may use it.
            >
            >
            > Sebastian
            >
            > --- In soaplite@y..., Paul Kulchenko <paulclinger@y...> wrote:
            > > Hi, Sebastian!
            > >
            > > That's true, but at the same time it's easy to imagine situation
            > when
            > > you send something not directly, but thru the several different
            > > intermediaries and each of them will need to handle this huge
            > > request. If this piece is encoded as external reference then
            > handler
            > > could be smart enough to get it only if it's required (yet I
            > don't
            > > know about such smart handlers :)). Ideas, ideas...
            > >
            > > Ideally implementation should be flexible enough to handle both
            > (and
            > > maybe man others) approaches, maybe with manual hints.
            > >
            > > Best wishes, Paul.
            >
            >
            >
            > ------------------------ Yahoo! Groups Sponsor
            >
            > To unsubscribe from this group, send an email to:
            > soaplite-unsubscribe@yahoogroups.com
            >
            >
            >
            > Your use of Yahoo! Groups is subject to
            > http://docs.yahoo.com/info/terms/
            >
            >


            __________________________________________________
            Do You Yahoo!?
            Get email at your own domain with Yahoo! Mail.
            http://personal.mail.yahoo.com/
          Your message has been successfully submitted and would be delivered to recipients shortly.