Loading ...
Sorry, an error occurred while loading the content.

Using SOAP::Lite and C++, memory consumption problem

Expand Messages
  • Andersson, Kimmo
    Hi ! I have already posted this message to a Perl newsgroup but as SOAP::Lite is implemented with Perl I thought why not also to this mailing list. This is
    Message 1 of 2 , Oct 28, 2002
    • 0 Attachment
      Hi !

      I have already posted this message to a Perl newsgroup but as SOAP::Lite is
      implemented with Perl I thought why not also to this mailing list.
      This is more a Perl problem, not about SOAP::Lite as such. But I' hoping
      that some one might have met a similar problem and even solved it.

      I have written a library in C++ which acts as an wrapper for sending SOAP
      messages. It is done in HP-UX environment and the SOAP::Lite toolkit is
      used

      As SOAP::Lite is implemented with Perl I have written my own Perl module
      which acts as an intermediary between my C++ code and SOAP::Lite. My own
      Perl module calls SOAP::Lite and returns the SOAP response to my C++ class
      which has called it.

      Unfortunately it has been discovered that the processes that use my library
      seem consume more and more memory while running and several SOAP messages
      are sent via my library. This has been verified using several processes and
      I have also verified it using my own test client, we are absolutely sure
      that SOAP::Lite and Perl are to be blamed.

      A clear pattern hasn't however emerged, it is not that say 25 K per each
      SOAP message memory is used and not released, it varies.
      Some time memory decreases and several SOAP messages are sent before memory
      again starts growing. I have used Purify but Purify does not report memory
      leaks. Actually there leaks from some system libraries but their amount
      remains constant whether I send one or hundred SOAP messages.

      Because the Perl we use is not compiled with usemultiplicity-option, the
      persistent processes which use my library must use just one instance of a
      class in my library and via it just one instance of the Perl interpreter
      during their lifetime.

      I have got the impression that Perl does release memory but it is released
      to Perl's own memory pool, not to the operating system.

      So I'm asking is there anything to be done ? I have just one subroutine in
      my Perl module and all the variables used are lexical variables (my)
      declared in that subroutine. They should be released when gone out of scope.
      In my C++ code I use perl_call_argv to call my Perl module and it and
      reading the Perl stack are surrounded by ENTER/SAVETMPS and FREETMPS/LEAVE.
      According to the Perlcall document that should get rid of any values
      returned by the Perl module.

      So possible solutions ?

      1. Find some mysterious Perl trick which forces Perl to release memory from
      it's memory pool or makes Perl reuse memory in it's pool more efficiently.
      - I haven't found even if I have searched intensively

      2. Re-compile Perl with usemultiplicity-option and create an instance of the
      Perl interpreter for each time a SOAP message is sent.
      - I don't know the possible consequences and I'm not the only one to use
      Perl. Others might not agree

      3. Shutdown and restart the persistent processes which use my library. For
      example a script is run every Saturday night.
      - Sounds quite quick and dirty

      4. Find the bug in my code.
      - I have tried to follow the instructions in Perlcall and Perlembed
      documents but You never know


      Kimmo Andersson

      Yomi Solutions Ltd E-mail: Kimmo.Andersson@....O
      Box 354, Piippukatu 11 Mobile: +358 50 384 3734
      FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709FINLAND
    • Paul Kulchenko
      Hi Kimmo, ... I believe that s the correct assumption: The way Perl handles memory is very simple. When it needs to allocate memory it will allocate
      Message 2 of 2 , Oct 29, 2002
      • 0 Attachment
        Hi Kimmo,

        > I have got the impression that Perl does release memory but it is
        > released to Perl's own memory pool, not to the operating system.

        I believe that's the correct assumption:

        "The way Perl handles memory is very simple. When it needs to
        allocate memory it will allocate everything it needs to perform it's
        tasks. For example, if a variable's content is 1MB, then Perl will
        allocate 1MB for the variable. When that variable goes out of scope
        or is catched by the garbage fairy, Perl will not release that 1MB to
        the kernel right away, it will reuse it in case it needs it later on
        during execution."

        Unless there is a memory leak somewhere in my or Perl's code, this
        memory pool should stop growing after some number of requests. I've
        done extensive tests on my windows and linux boxes chasing down
        memory leaks and tests showed that pool stops growing.

        > 1. Find some mysterious Perl trick which forces Perl to release
        > memory from
        > it's memory pool or makes Perl reuse memory in it's pool more
        > efficiently.
        > - I haven't found even if I have searched intensively
        It may exist, but I'm unaware of any such option.

        > 2. Re-compile Perl with usemultiplicity-option and create an
        > instance of the
        > Perl interpreter for each time a SOAP message is sent.
        > - I don't know the possible consequences and I'm not the only one
        > to use Perl. Others might not agree
        Will probably work, but may have performance impact. Can't say ho
        big.

        > 3. Shutdown and restart the persistent processes which use my
        > library. For example a script is run every Saturday night.
        > - Sounds quite quick and dirty
        It may be better than #2. You may think about adding file semaphore
        and graceful shutdown to your code.

        > 4. Find the bug in my code.
        > - I have tried to follow the instructions in Perlcall and Perlembed
        > documents but You never know
        From what you describe I don't there is any in memory handling piece
        (In other words, I would expect similar behavior from properly
        written code).

        5. it *may* be possible to restrict the amount of memory available
        for your process in your OS. Although the amount of required memory
        depends on the size of the message you want to process/generate, so
        you may not be able to handle all of the messages.

        Unfortunately I can't be more helpful on that. Some other options
        are: (1) reduce the SOAP::Lite's memory footprint, (2) provide
        streaming parsing as an option, (3) do something else. #1 and 2 are
        on my list, but they both only experiments for now and it's not clear
        when they will make into the code.

        Best wishes, Paul.

        --- "Andersson, Kimmo" <Kimmo.Andersson@...> wrote:
        > Hi !
        >
        > I have already posted this message to a Perl newsgroup but as
        > SOAP::Lite is
        > implemented with Perl I thought why not also to this mailing list.
        > This is more a Perl problem, not about SOAP::Lite as such. But I'
        > hoping
        > that some one might have met a similar problem and even solved it.
        >
        > I have written a library in C++ which acts as an wrapper for
        > sending SOAP
        > messages. It is done in HP-UX environment and the SOAP::Lite
        > toolkit is
        > used
        >
        > As SOAP::Lite is implemented with Perl I have written my own Perl
        > module
        > which acts as an intermediary between my C++ code and SOAP::Lite.
        > My own
        > Perl module calls SOAP::Lite and returns the SOAP response to my
        > C++ class
        > which has called it.
        >
        > Unfortunately it has been discovered that the processes that use my
        > library
        > seem consume more and more memory while running and several SOAP
        > messages
        > are sent via my library. This has been verified using several
        > processes and
        > I have also verified it using my own test client, we are absolutely
        > sure
        > that SOAP::Lite and Perl are to be blamed.
        >
        > A clear pattern hasn't however emerged, it is not that say 25 K per
        > each
        > SOAP message memory is used and not released, it varies.
        > Some time memory decreases and several SOAP messages are sent
        > before memory
        > again starts growing. I have used Purify but Purify does not
        > report memory
        > leaks. Actually there leaks from some system libraries but their
        > amount
        > remains constant whether I send one or hundred SOAP messages.
        >
        > Because the Perl we use is not compiled with
        > usemultiplicity-option, the
        > persistent processes which use my library must use just one
        > instance of a
        > class in my library and via it just one instance of the Perl
        > interpreter
        > during their lifetime.
        >
        > I have got the impression that Perl does release memory but it is
        > released
        > to Perl's own memory pool, not to the operating system.
        >
        > So I'm asking is there anything to be done ? I have just one
        > subroutine in
        > my Perl module and all the variables used are lexical variables
        > (my)
        > declared in that subroutine. They should be released when gone out
        > of scope.
        > In my C++ code I use perl_call_argv to call my Perl module and it
        > and
        > reading the Perl stack are surrounded by ENTER/SAVETMPS and
        > FREETMPS/LEAVE.
        > According to the Perlcall document that should get rid of any
        > values
        > returned by the Perl module.
        >
        > So possible solutions ?
        >
        > 1. Find some mysterious Perl trick which forces Perl to release
        > memory from
        > it's memory pool or makes Perl reuse memory in it's pool more
        > efficiently.
        > - I haven't found even if I have searched intensively
        >
        > 2. Re-compile Perl with usemultiplicity-option and create an
        > instance of the
        > Perl interpreter for each time a SOAP message is sent.
        > - I don't know the possible consequences and I'm not the only one
        > to use
        > Perl. Others might not agree
        >
        > 3. Shutdown and restart the persistent processes which use my
        > library. For
        > example a script is run every Saturday night.
        > - Sounds quite quick and dirty
        >
        > 4. Find the bug in my code.
        > - I have tried to follow the instructions in Perlcall and Perlembed
        > documents but You never know
        >
        >
        > Kimmo Andersson
        >
        > Yomi Solutions Ltd E-mail: Kimmo.Andersson@....O
        > Box 354, Piippukatu 11 Mobile: +358 50 384 3734
        > FIN-40101 JYV�SKYL� Fax : +358 14 240 4709FINLAND
        >
        >
        > ------------------------ Yahoo! Groups Sponsor
        >
        > To unsubscribe from this group, send an email to:
        > soaplite-unsubscribe@yahoogroups.com
        >
        >
        >
        > Your use of Yahoo! Groups is subject to
        > http://docs.yahoo.com/info/terms/
        >
        >


        __________________________________________________
        Do you Yahoo!?
        HotJobs - Search new jobs daily now
        http://hotjobs.yahoo.com/
      Your message has been successfully submitted and would be delivered to recipients shortly.