Loading ...
Sorry, an error occurred while loading the content.

returning a huge amount of data from SOAP::Lite server

Expand Messages
  • Chris Sidi
    Hi, I m about to write something that will take a few parameters (a date range and some other filters) and may return 3+ gigabytes of data (from files on the
    Message 1 of 2 , Aug 14, 2001
    • 0 Attachment
      Hi,

      I'm about to write something that will take a few parameters (a date range
      and some other filters) and may return 3+ gigabytes of data (from files on
      the server's hard drive). I kinda doubt SOAP::Lite supports this, but is
      there a way to return that much data (in one HTTP response) without eating
      up 3+ gigs of (virtual) memory on both the client and server side?

      Like instead of just one return call on the server, I imagine print or
      some SOAP::Lite method getting called thousands of times, and that data
      getting sent to socket rather than queued till the end of data. And then
      the client would have some way of reading back portions of the response.
      Perhaps something like this:

      Server:

      import SOAP::Data 'name';
      sub filter_data {
      my ($class, $regex) = @_;

      open(FH, "/tmp/huge_file") || die $!;

      while (<FH>) {
      if (/$regex/o) {
      SOAP::Lite->response_part( name('matching_line' => $_) );
      }
      }

      return SOAP::Success;
      }

      Client:

      my $response = SOAP::Lite->uri(...)->proxy(...)->filter_data( "foo.*bar" );

      # calls to next_part() cause more data to be read from socket
      # so that memory usage is not overly high.
      while ( defined($part = $response->next_part()) ) {
      process_matching_data( $part );
      }

      Thanks,
      Chris Sidi
    • Paul Kulchenko
      Hi, Chris! ... You concern is valid, SOAP::Lite won t be able to handle this unless you have enough virtual memory, and even then it ll take some time to
      Message 2 of 2 , Aug 14, 2001
      • 0 Attachment
        Hi, Chris!

        > the server's hard drive). I kinda doubt SOAP::Lite supports this,
        > but is
        > there a way to return that much data (in one HTTP response) without
        > eating
        > up 3+ gigs of (virtual) memory on both the client and server side?
        You concern is valid, SOAP::Lite won't be able to handle this unless
        you have enough virtual memory, and even then it'll take some time to
        process it.

        I don't think algorithm you describe can help much unless you split
        your datastream on separate SOAP calls. This problem reminds me
        discussing that happened several years ago on Delphi maillist about
        returning tens of thousands records on client side, so user will be
        able to navigate there. Though you *can* do it, your best bet is
        limit request, so it'll return only limited amount of information.
        Same thing is very true here also. I don't think there is any toolkit
        on market that can handle such requests/responses. Even if you use
        SAX or pull parser, you still have copy of parsed XML in memory and
        copy of your data you're building from it, which in most cases
        doubles the amount of required memory. And it's the best case.
        Consider two possible cases: (1) you have many small elements and
        (2)your have one huge element.

        If you have one huge element, want it or not, it will be copied
        somewhere at least a couple of time, which triple or quadruple memory
        requirements. If you have many small elements, you'll spend quite a
        bit of your time, managing memory for those, esp. if it has a complex
        structure.

        I would advise you to review application logic rather than try to
        solve in on protocol level. Let me know if you find way to handle it.
        I'm interested.

        Best wishes, Paul.

        --- Chris Sidi <sidi@...> wrote:
        > Hi,
        >
        > I'm about to write something that will take a few parameters (a
        > date range
        > and some other filters) and may return 3+ gigabytes of data (from
        > files on
        > the server's hard drive). I kinda doubt SOAP::Lite supports this,
        > but is
        > there a way to return that much data (in one HTTP response) without
        > eating
        > up 3+ gigs of (virtual) memory on both the client and server side?
        >
        > Like instead of just one return call on the server, I imagine print
        > or
        > some SOAP::Lite method getting called thousands of times, and that
        > data
        > getting sent to socket rather than queued till the end of data.
        > And then
        > the client would have some way of reading back portions of the
        > response.
        > Perhaps something like this:
        >
        > Server:
        >
        > import SOAP::Data 'name';
        > sub filter_data {
        > my ($class, $regex) = @_;
        >
        > open(FH, "/tmp/huge_file") || die $!;
        >
        > while (<FH>) {
        > if (/$regex/o) {
        > SOAP::Lite->response_part( name('matching_line' => $_) );
        > }
        > }
        >
        > return SOAP::Success;
        > }
        >
        > Client:
        >
        > my $response = SOAP::Lite->uri(...)->proxy(...)->filter_data(
        > "foo.*bar" );
        >
        > # calls to next_part() cause more data to be read from socket
        > # so that memory usage is not overly high.
        > while ( defined($part = $response->next_part()) ) {
        > process_matching_data( $part );
        > }
        >
        > Thanks,
        > Chris Sidi
        >
        >
        > To unsubscribe from this group, send an email to:
        > soaplite-unsubscribe@yahoogroups.com
        >
        >
        >
        > Your use of Yahoo! Groups is subject to
        > http://docs.yahoo.com/info/terms/
        >
        >


        __________________________________________________
        Do You Yahoo!?
        Make international calls for as low as $.04/minute with Yahoo! Messenger
        http://phonecard.yahoo.com/
      Your message has been successfully submitted and would be delivered to recipients shortly.