Loading ...
Sorry, an error occurred while loading the content.

5603Re: [soaplite] [Q] How to process the client-side SOAP response AS it arrives, not AFTER arrives?

Expand Messages
  • Mike B
    Sep 13, 2006
    • 0 Attachment
      Jesse / all,

      Actually, yes.  My content is delivered as XML but I immediately convert it to SQL statements for a local database operation. So, I don't need the XML after I've converted it. Thus, I can read from the results stream, convert to SQL, dump the processed XML fragments and not take the hit of both XML DOM and a collection of SQL statements in memory at the same time. 

      Again, I only wanted to do this for very large XML documents ( 2M or larger ) that were not going to live on the client side at all.  I wasn't sweating the small stuff.

      Having said that, the results thus far are no.  I attempted to override parts of the SOAP::Deserializer unsuccessfully today.  I managed to get the object installed into SOAP::Lite but I had some package problems I didn't know how to get around.  In the end, I was out of time to work on it further. 

      If I feel that this is a pattern of use, I may hobble together a SOAP:Lite-ish client front end and an XML::Parser/Expat client backend to do this kind of stream processing.

      Since all this work was for performance gains, here are the three biggest gains I was able to achieve.  If you are always dealing with small data sets, these changes probably won't help you noticeably:
      1. Don't use XML::XPath for XML DOM iteration.  XPath was really very slow to process the data and caused a very large memory signature, around 110M for my service.
      2. Use XML::Records for XML DOM iteration.  XML::Records was fast with a light weight memory signature of about 4M for my service.  The memory savings alone probably saved me some time.
      3. Added compression avialable to the client so the server had the opportunity to compress my data (2.7M of XML to about 246K of data) and improve transfer times.
        • options => {compress_threshold => 10000}
      Note: all my Perl packages come from ActiveState so performance may vary between versions, platforms and venders.

      Summary: these small changes took my process from 172 seconds+ and over 210M of memory down to 32 seconds+ and 75M of memory, a 5.3 times improvement!

      Anyway, on to other things.

      Mike Bigg at (no spam) ya H oo dot com

      Jesse Brown <jbrown@...> wrote:
      Depending on what you need.

      You could use that to get ahold of the XML data during parsing, but you
      will then need to store that information somewhere, and pass references
      back, etc. All this would happen before it was dispatched to your
      method, so it would require a lot of logic built in.

      Depending on what you are trying to do, it would probably be easier to
      just implement yourself without wading through the guts of SOAP::Lite,
      but it might be a win.

      Mike B wrote:
      > Jesse / all,
      >
      > Good afternoon. Thanks for the response - I was afraid that would be
      > the case.
      >
      > I was looked into the SOAP::Deserializer to see if sub classing that
      > would provide the necessary access to the XML::Parser as data arrives.
      > Do you think that would work?
      >
      > Mike
      >
      >
      > */Jesse Brown <jbrown@postini. com>/* wrote:
      >
      > To my knowledge, the abstraction model of SOAP::Lite makes this
      > impossible. (your code is not even called until after the request has
      > been completely deserialized and converted).
      >
      > That said, look at the code - it is not overly hard to write your own
      > SOAP handler (completely bypassing SOAP::Lite) - we have done that
      > anyways, because we need better interoperability. That said, it does
      > require a proctological level of knowledge of the SOAP standards.
      >
      > Mike B wrote:
      > > All,
      > >
      > > Using SOAP::Lite, how can I process the client-side incoming SOAP
      > > response AS it arrives, instead of AFTER arrives?
      > >
      > > I have a large XML data transfer (more than 2.6M of XML, and
      > getting
      > > larger.) If I wait for all of it to arrive on the client, the
      > XML DOM
      > > memory size, required DOM parsing/iteration, and processing time
      > are
      > > simply too large and too costly.
      > >
      > > I want to parse the client side SOAP result so that, as each XML
      > > fragment arrives, the SOAP::Lite client can process the XML
      > fragment
      > > and get ready for the next XML fragment.
      > >
      > > I think what I'm looking for are client-side hooks into
      > SOAP::Lite so
      > > that I can perform SAX-style processing, similar to XML::Parser
      > > Example below.
      > >
      > > Mike Bigg at (no spam) ya H oo dot com
      > >
      > > >> original msg: #5594 - "How to "setHandlers" on Soap::Lite (and
      > > related questions) ///
      > >
      > <http://proxify. com/p/011010A000 0110/http: =2f=2ftech. groups.yahoo. com=2fgroup= 2fsoaplite= 2fmessage/ 5594
      > <http://proxify. com/p/011010A000 0110/http: =2f=2ftech. groups.yahoo. com=2fgroup= 2fsoaplite= 2fmessage/ 5594>>/
      >
      > > ")
      > >
      > > */Mike B <mikebigg@yahoo. com <mailto:mikebigg% 40yahoo.com> >/* wrote:
      > >
      > > Greetings. I have been using SOAP::Lite as a client in Perl for
      > > most of a year now and found it pretty nice. I do have some
      > > questions that, regretfully, I have not been able to answer for
      > > myself.
      > >
      > > My Questions
      > >
      > > 1. As a web service client, how can I set low-level "Handlers"
      > > to process in-bound data? (See XML::Parser eexample below
      > > for a sample of what I'd like to accomplish with SOAP::Lite)
      > > 2. If I can set Handlers on SOAP::Lite, what impact does the
      > > use of compression have on performance?
      > > 3. If I can set Handlers on SOAP::Lite, what impact would the
      > > use of encryption have on performance?
      > >
      > >
      > > The Context
      > >
      > > 1. Using SOAP::Lite for a Perl client
      > > 2. Transferring XML in the body of the SOAP result
      > > 3. Currently using compression
      > >
      > >
      > > My Basic Usage
      > >
      > > This is my basic usage for SOAP::Lite usage as a Perl Client (see
      > > pseudo-code: )
      > > ...
      > > my $ws = SOAP::Lite-> proxy( $urlLocation, timeout => $SOAPTIMEOUT );
      > > my $header = SOAP::Header- >name("") ->value ...
      > > my $body = SOAP::Data-> name("")- >value( $content )->type('xml' );
      > > $result = $ws->$action( $header, $body );
      > > ...
      > >
      > >
      > > XML::Parser Example
      > >
      > > I know how do what I want at the XML::Parser level but not sure
      > > how do to this with SOAP::Lite (see pseudo-code: )
      > > ...
      > > my $handlers = { Start => \&handle_start,
      > > End => \&handle_end
      > > Char => \&handle_char # not used
      > > };
      > > my $parser = new XML::Parser(
      > > Handlers => { %{$handlers} }
      > > );
      > > ...
      > > $parser->parse( *TEMPFILE );
      > > ...
      > >
      > >
      > > Thanks in advance for any tips, url links, class names,
      > > subroutines names, or mental projections towards the answers to my
      > > questions.
      > >
      > > Mike Bigg at (no spam) ya H oo dot com
      > >
      > > <http://us.rd. yahoo.com/ evt=42973/ *http://www. yahoo.com/ preview
      > <http://us.rd. yahoo.com/ evt=42973/ *http://www. yahoo.com/ preview>>
      > >
      > >
      > > ____________ _________ _________ _________ _________ __
      > > Do You Yahoo!?
      > > Tired of spam? Yahoo! Mail has the best spam protection around
      > > http://mail. yahoo.com <http://mail. yahoo.com>
      >
      > ------------ --------- --------- --------- --------- --------- -
      > This message may contain confidential and/or privileged
      > information. This information is intended to be read only
      > by the individual or entity to whom it is addressed. If
      > you are not the intended recipient, you are on notice that
      > any review, disclosure, copying, distribution or use of
      > the contents of this message is strictly prohibited. If
      > you have received this message in error, please notify the
      > sender immediately and delete or destroy any copy of this
      > message.
      >
      >
      >
      >
      > ------------ --------- --------- --------- --------- --------- -
      > Do you Yahoo!?
      > Everyone is raving about the all-new Yahoo! Mail.
      > <http://us.rd. yahoo.com/ evt=42297/ *http://advision .webevents. yahoo.com/ mailbeta>

      ------------ --------- --------- --------- --------- --------- -
      This message may contain confidential and/or privileged
      information. This information is intended to be read only
      by the individual or entity to whom it is addressed. If
      you are not the intended recipient, you are on notice that
      any review, disclosure, copying, distribution or use of
      the contents of this message is strictly prohibited. If
      you have received this message in error, please notify the
      sender immediately and delete or destroy any copy of this
      message.



      Do you Yahoo!?
      Get on board. You're invited to try the new Yahoo! Mail.

    • Show all 5 messages in this topic