Loading ...
Sorry, an error occurred while loading the content.

Returning "::" causes out of memory error on Windows XP under ASPerl 5.6.1 B633

Expand Messages
  • chesal
    I have a script that gathers a bunch of data as an array from an Apache server running a SOAP::Lite interface, serializes it and returns it. On Linux and under
    Message 1 of 4 , Nov 7, 2005
    • 0 Attachment
      I have a script that gathers a bunch of data as an array from an
      Apache server running a SOAP::Lite interface, serializes it and
      returns it.

      On Linux and under Cygwin the approximatly 300kB of data is returned,
      parsed and printed by the script with barely any memory use at all.

      But the script, run ActiveState Perl 5.6.1 B633, ends up running out
      of memory. Memory use climbs to 2GB and soon after the script enters
      the sub:

      SOAP::Lite::deserialize()

      Specifically the line:

      my $parsed = $self->decode($_[0]);

      The memory usage quickly climbs to 2GB and then the script dies as it
      hits the upper bound on user memory in Windows XP. On Linux or under
      Cygwin this script runs fine. Barely a blip in memory usage.

      The data is an array of strings where each string matches the form:

      (\d+)::(\d+)::(\S+)::(.*)\n

      The "::" is used as a field separator for some data. The data is
      returned as a scalar from the SOAP server by saying:

      return join("", @data);

      The data is then deserialized on the script side with:

      my @data = split(/\n/, $result->result());

      I went with the scalar translation in an attempt to fix the memory
      problem but whether I return @data or join("", @data) I get the same
      result -- out of memory on Windows XP.

      BUT if I change the strings in @data to:

      (\d+)\s+(\d+)\s+(\S+)\s+(.*)\n

      Then the problem goes away. The SOAP::Lite::deserialization() sub is
      fast and accounts for barely a blip on the memory monitor.

      Does anyone know what it is with "::" that causes the deserializer
      such problem under Windows?

      - Ian
    • Peter Sinnott
      ... Are you using perl 5.6.1 on all 3 platforms? ... Can you trace the execution any further? ... Its a bit of a guess but XML::Parser::Lite is used to parse
      Message 2 of 4 , Nov 8, 2005
      • 0 Attachment
        On Mon, Nov 07, 2005 at 09:07:24PM -0000, chesal wrote:
        >
        > On Linux and under Cygwin the approximatly 300kB of data is returned,
        > parsed and printed by the script with barely any memory use at all.
        >
        > But the script, run ActiveState Perl 5.6.1 B633, ends up running out
        > of memory.

        Are you using perl 5.6.1 on all 3 platforms?

        > my $parsed = $self->decode($_[0]);
        >

        Can you trace the execution any further?

        > Does anyone know what it is with "::" that causes the deserializer
        > such problem under Windows?
        >

        Its a bit of a guess but XML::Parser::Lite is used to parse xml if
        XML::Parser is not available. The pod says :

        This Perl module gives you access to XML parser with interface
        similar to XML::Parser interface. Though only basic calls are
        supported (init, final, start, char, and end) you should be
        able to use it in the same way you use XML::Parser. Due to
        using experimantal regexp features it'll work only on Perl 5.6
        and may behave differently on different platforms.


        Things probably aren't as scary as they sound but it might be worth
        checking if you have XML::Parser installed for the 3 perls in question.

        --
        Our goal is to assertively pursue interdependent deliverables as well as
        to globally fashion low-risk high-yield paradigms in order to solve business
        problems
      • chesal
        Hi Peter. Thanks for the quick response. Some comments below... ... Yes. Although Windows is the only platform using the ActiveState distribution. Cygwin and
        Message 3 of 4 , Nov 23, 2005
        • 0 Attachment
          Hi Peter. Thanks for the quick response. Some comments below...

          --- In soaplite@yahoogroups.com, Peter Sinnott <psinnottie@a...> wrote:
          >
          > On Mon, Nov 07, 2005 at 09:07:24PM -0000, chesal wrote:
          > >
          > > On Linux and under Cygwin the approximatly 300kB of data is returned,
          > > parsed and printed by the script with barely any memory use at all.
          > >
          > > But the script, run ActiveState Perl 5.6.1 B633, ends up running out
          > > of memory.
          >
          > Are you using perl 5.6.1 on all 3 platforms?

          Yes. Although Windows is the only platform using the ActiveState
          distribution. Cygwin and Linux both use the Cygwin and RH9 binary
          packages respectivily.

          > > my $parsed = $self->decode($_[0]);
          > >
          >
          > Can you trace the execution any further?

          Yes. I worked down to the XML::Parser::parse() routine -- that's where
          memory starts to climb.

          The call to $self->decode_object() that occurs in the SOAP::Lite
          module right after the expat parsing does not increase memory. Memory
          use peaks (and never declines by much) in the XML::Parser::parse() call.

          It looks like it might be an issue with the expat parse on Windows.
          One thing that comes to mind is character encoding. While the data is
          pure ASCII text, perhaps I'm incurring a bunch of unecessary
          serialization/deserialization overhead in the XML payload of the SOAP
          call because there are character encoding mismatches. Is that a
          possibility? I couldn't see an easy way to indicate the encoding was
          ASCII text instead of UTF-8 when making the SOAP lite call. Is there?

          > > Does anyone know what it is with "::" that causes the deserializer
          > > such problem under Windows?
          > >
          >
          > Its a bit of a guess but XML::Parser::Lite is used to parse xml if
          > XML::Parser is not available.

          All three platforms have XML::Parser available. The ActiveState 5.6.1
          distribution shipped with it. Linux and Cygwin had the package added
          via cpan install call a long time ago.

          What I'm noticing is that in general memory consumption is orders of
          magnitude higher for ASPerl 5.6.1 SOAP::Lite calls than on any other
          platform. Even after dropping the '::' separators memory use is still
          climbing to ~50MB for approximately 1MB of data returned. It's all
          plain text data. Nothing particular interesting about it. There are no
          special character encodings. The data is a document with lines in the
          form:

          ^(\d+)\s(\d+)\s(\S+)\s(.*)

          The (.*) porition at the end can be up to 1024 characters long.
          Because it's all one document it's being returned to the result()
          storage point in the SOAP::Lite calling object. But I did try making
          it an array and returning the array, where each line was an element,
          instead of a flat document and then using result() and paramsout() and
          it made no difference. The memory is still being consumed by the
          deserialization.

          Any tips on reducing memory usage on Windows are greatly appreciated.
          Thanks.

          - Ian
        • Peter Sinnott
          ... If you can reduce the test case to some xml and calls to XML::Parser then maybe some perl xml mailing list can be of help. ... What version of XML::Parser
          Message 4 of 4 , Nov 23, 2005
          • 0 Attachment
            On Wed, Nov 23, 2005 at 03:53:29PM -0000, chesal wrote:
            >
            > Yes. I worked down to the XML::Parser::parse() routine -- that's where
            > memory starts to climb.
            >

            If you can reduce the test case to some xml and calls
            to XML::Parser then maybe some perl xml mailing list can be of help.

            >
            > All three platforms have XML::Parser available. The ActiveState 5.6.1
            > distribution shipped with it. Linux and Cygwin had the package added
            > via cpan install call a long time ago.
            >

            What version of XML::Parser do they each have?

            perl -e'use XML::Parser 10;'


            http://aspn.activestate.com/ASPN/Perl/Modules/

            http://aspn.activestate.com/ASPN/Modules?module_name=X&order=name&query_start=36

            lists XML::Parser as v2.30. The changelog on CPAN lists a memory leak
            fixed in 2.32. May not be the problem but no harm checking it out.


            --
            We exist to collaboratively network error-free paradigms while continuing
            to assertively foster virtual services while promoting personal employee
            growth
          Your message has been successfully submitted and would be delivered to recipients shortly.