Loading ...
Sorry, an error occurred while loading the content.

Re: [soaplite] returning a huge amount of data from SOAP::Lite server

Expand Messages
  • Paul Kulchenko
    Hi, Chris! ... You concern is valid, SOAP::Lite won t be able to handle this unless you have enough virtual memory, and even then it ll take some time to
    Message 1 of 2 , Aug 14 1:53 PM
    • 0 Attachment
      Hi, Chris!

      > the server's hard drive). I kinda doubt SOAP::Lite supports this,
      > but is
      > there a way to return that much data (in one HTTP response) without
      > eating
      > up 3+ gigs of (virtual) memory on both the client and server side?
      You concern is valid, SOAP::Lite won't be able to handle this unless
      you have enough virtual memory, and even then it'll take some time to
      process it.

      I don't think algorithm you describe can help much unless you split
      your datastream on separate SOAP calls. This problem reminds me
      discussing that happened several years ago on Delphi maillist about
      returning tens of thousands records on client side, so user will be
      able to navigate there. Though you *can* do it, your best bet is
      limit request, so it'll return only limited amount of information.
      Same thing is very true here also. I don't think there is any toolkit
      on market that can handle such requests/responses. Even if you use
      SAX or pull parser, you still have copy of parsed XML in memory and
      copy of your data you're building from it, which in most cases
      doubles the amount of required memory. And it's the best case.
      Consider two possible cases: (1) you have many small elements and
      (2)your have one huge element.

      If you have one huge element, want it or not, it will be copied
      somewhere at least a couple of time, which triple or quadruple memory
      requirements. If you have many small elements, you'll spend quite a
      bit of your time, managing memory for those, esp. if it has a complex
      structure.

      I would advise you to review application logic rather than try to
      solve in on protocol level. Let me know if you find way to handle it.
      I'm interested.

      Best wishes, Paul.

      --- Chris Sidi <sidi@...> wrote:
      > Hi,
      >
      > I'm about to write something that will take a few parameters (a
      > date range
      > and some other filters) and may return 3+ gigabytes of data (from
      > files on
      > the server's hard drive). I kinda doubt SOAP::Lite supports this,
      > but is
      > there a way to return that much data (in one HTTP response) without
      > eating
      > up 3+ gigs of (virtual) memory on both the client and server side?
      >
      > Like instead of just one return call on the server, I imagine print
      > or
      > some SOAP::Lite method getting called thousands of times, and that
      > data
      > getting sent to socket rather than queued till the end of data.
      > And then
      > the client would have some way of reading back portions of the
      > response.
      > Perhaps something like this:
      >
      > Server:
      >
      > import SOAP::Data 'name';
      > sub filter_data {
      > my ($class, $regex) = @_;
      >
      > open(FH, "/tmp/huge_file") || die $!;
      >
      > while (<FH>) {
      > if (/$regex/o) {
      > SOAP::Lite->response_part( name('matching_line' => $_) );
      > }
      > }
      >
      > return SOAP::Success;
      > }
      >
      > Client:
      >
      > my $response = SOAP::Lite->uri(...)->proxy(...)->filter_data(
      > "foo.*bar" );
      >
      > # calls to next_part() cause more data to be read from socket
      > # so that memory usage is not overly high.
      > while ( defined($part = $response->next_part()) ) {
      > process_matching_data( $part );
      > }
      >
      > Thanks,
      > Chris Sidi
      >
      >
      > To unsubscribe from this group, send an email to:
      > soaplite-unsubscribe@yahoogroups.com
      >
      >
      >
      > Your use of Yahoo! Groups is subject to
      > http://docs.yahoo.com/info/terms/
      >
      >


      __________________________________________________
      Do You Yahoo!?
      Make international calls for as low as $.04/minute with Yahoo! Messenger
      http://phonecard.yahoo.com/
    Your message has been successfully submitted and would be delivered to recipients shortly.