Loading ...
Sorry, an error occurred while loading the content.

RE: [soaplite] returning a huge amount of data from SOAP::Lite se rver

Expand Messages
  • Michael Percy
    I have done something similar to this, but it involves quite a bit of programming overhead. It looks similar to FTP, with one TCP channel and one SOAP control
    Message 1 of 1 , Aug 14, 2001
    • 0 Attachment
      I have done something similar to this, but it involves quite a bit of
      programming overhead.

      It looks similar to FTP, with one TCP channel and one SOAP control channel.

      Client initiates two connections, one for data streaming back to client and
      one for SOAP request. On SOAP server side, STDOUT is opened to the TCP
      socket and then the data is dumped to STDOUT. Client collects this data at
      its leisure. Benefits are much improved efficiency over parsing the data out
      of XML as well as streaming output (which is not possible with SOAP). There
      are issues with synchronization and sessions, and the solutions I have come
      up with to those problems are less than ideal, but perl starts to show some
      weakness when doing those types of things. Maybe perl 6 + stable threads
      will simplify life (someday :)...

      HTH,
      Mike

      > -----Original Message-----
      > From: Chris Sidi [mailto:sidi@...]
      > Sent: Tuesday, August 14, 2001 12:21 PM
      > To: soaplite@yahoogroups.com
      > Subject: [soaplite] returning a huge amount of data from SOAP::Lite
      > server
      >
      >
      > Hi,
      >
      > I'm about to write something that will take a few parameters
      > (a date range
      > and some other filters) and may return 3+ gigabytes of data
      > (from files on
      > the server's hard drive). I kinda doubt SOAP::Lite supports
      > this, but is
      > there a way to return that much data (in one HTTP response)
      > without eating
      > up 3+ gigs of (virtual) memory on both the client and server side?
      >
      > Like instead of just one return call on the server, I imagine print or
      > some SOAP::Lite method getting called thousands of times, and
      > that data
      > getting sent to socket rather than queued till the end of
      > data. And then
      > the client would have some way of reading back portions of
      > the response.
      > Perhaps something like this:
      >
      > Server:
      >
      > import SOAP::Data 'name';
      > sub filter_data {
      > my ($class, $regex) = @_;
      >
      > open(FH, "/tmp/huge_file") || die $!;
      >
      > while (<FH>) {
      > if (/$regex/o) {
      > SOAP::Lite->response_part( name('matching_line'
      > => $_) );
      > }
      > }
      >
      > return SOAP::Success;
      > }
      >
      > Client:
      >
      > my $response =
      > SOAP::Lite->uri(...)->proxy(...)->filter_data( "foo.*bar" );
      >
      > # calls to next_part() cause more data to be read from socket
      > # so that memory usage is not overly high.
      > while ( defined($part = $response->next_part()) ) {
      > process_matching_data( $part );
      > }
      >
      > Thanks,
      > Chris Sidi
      >
      >
      > To unsubscribe from this group, send an email to:
      > soaplite-unsubscribe@yahoogroups.com
      >
      >
      >
      > Your use of Yahoo! Groups is subject to
      > http://docs.yahoo.com/info/terms/
      >
      >
    Your message has been successfully submitted and would be delivered to recipients shortly.