Loading ...
Sorry, an error occurred while loading the content.

RE: [soaplite] Using SOAP::Lite and C++, memory consumption probl em

Expand Messages
  • Sean.Meisner@VerizonWireless.com
    Hi Kimmo, You are correct in that the Perl interpreter doesn t like to give memory back to the operating system. It s a bit of ugliness that I ve been forced
    Message 1 of 2 , Oct 29, 2002
    • 0 Attachment
      Hi Kimmo,

      You are correct in that the Perl interpreter doesn't like
      to give memory back to the operating system. It's a bit of
      ugliness that I've been forced to code around myself on
      occasion.

      My suggestion is, if you are willing to give up some execution
      speed, create a standalone Perl script that wraps the SOAP::Lite
      calls. Then in your C++ program, instead of embedding the Perl
      interpreter and calling your functions with perl_call_argv, use
      a system call to shell out and call the Perl script. This way
      the perl interpreter will be created and destroyed in it's own
      process space and the memory it consumes will be released when
      that forked process exits.

      Again, this approach will make your SOAP calls slower but maybe
      it will be worth it in your case.

      Cheers,

      Sean


      > -----Original Message-----
      > From: Andersson, Kimmo [mailto:Kimmo.Andersson@...]
      > Sent: Tuesday, October 29, 2002 2:21 AM
      > To: 'soaplite@yahoogroups.com'
      > Subject: [soaplite] Using SOAP::Lite and C++, memory
      > consumption problem
      >
      >
      > Hi !
      >
      > I have already posted this message to a Perl newsgroup but as
      > SOAP::Lite is
      > implemented with Perl I thought why not also to this mailing list.
      > This is more a Perl problem, not about SOAP::Lite as such.
      > But I' hoping
      > that some one might have met a similar problem and even solved it.
      >
      > I have written a library in C++ which acts as an wrapper for
      > sending SOAP
      > messages. It is done in HP-UX environment and the SOAP::Lite
      > toolkit is
      > used
      >
      > As SOAP::Lite is implemented with Perl I have written my own
      > Perl module
      > which acts as an intermediary between my C++ code and
      > SOAP::Lite. My own
      > Perl module calls SOAP::Lite and returns the SOAP response to
      > my C++ class
      > which has called it.
      >
      > Unfortunately it has been discovered that the processes that
      > use my library
      > seem consume more and more memory while running and several
      > SOAP messages
      > are sent via my library. This has been verified using
      > several processes and
      > I have also verified it using my own test client, we are
      > absolutely sure
      > that SOAP::Lite and Perl are to be blamed.
      >
      > A clear pattern hasn't however emerged, it is not that say 25
      > K per each
      > SOAP message memory is used and not released, it varies.
      > Some time memory decreases and several SOAP messages are sent
      > before memory
      > again starts growing. I have used Purify but Purify does not
      > report memory
      > leaks. Actually there leaks from some system libraries but
      > their amount
      > remains constant whether I send one or hundred SOAP messages.
      >
      > Because the Perl we use is not compiled with
      > usemultiplicity-option, the
      > persistent processes which use my library must use just one
      > instance of a
      > class in my library and via it just one instance of the Perl
      > interpreter
      > during their lifetime.
      >
      > I have got the impression that Perl does release memory but
      > it is released
      > to Perl's own memory pool, not to the operating system.
      >
      > So I'm asking is there anything to be done ? I have just one
      > subroutine in
      > my Perl module and all the variables used are lexical variables (my)
      > declared in that subroutine. They should be released when
      > gone out of scope.
      > In my C++ code I use perl_call_argv to call my Perl module and it and
      > reading the Perl stack are surrounded by ENTER/SAVETMPS and
      > FREETMPS/LEAVE.
      > According to the Perlcall document that should get rid of any values
      > returned by the Perl module.
      >
      > So possible solutions ?
      >
      > 1. Find some mysterious Perl trick which forces Perl to
      > release memory from
      > it's memory pool or makes Perl reuse memory in it's pool more
      > efficiently.
      > - I haven't found even if I have searched intensively
      >
      > 2. Re-compile Perl with usemultiplicity-option and create an
      > instance of the
      > Perl interpreter for each time a SOAP message is sent.
      > - I don't know the possible consequences and I'm not the only
      > one to use
      > Perl. Others might not agree
      >
      > 3. Shutdown and restart the persistent processes which use my
      > library. For
      > example a script is run every Saturday night.
      > - Sounds quite quick and dirty
      >
      > 4. Find the bug in my code.
      > - I have tried to follow the instructions in Perlcall and Perlembed
      > documents but You never know
      >
      >
      > Kimmo Andersson
      >
      > Yomi Solutions Ltd E-mail:
      > Kimmo.Andersson@....O
      > Box 354, Piippukatu 11 Mobile: +358 50 384 3734
      > FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709FINLAND
      >
      >
      > ------------------------ Yahoo! Groups Sponsor
      > ---------------------~-->
      > Get 128 Bit SSL Encryption!
      > http://us.click.yahoo.com/JjlUgA/vN2EAA/kG8FAA/W6uqlB/TM
      > --------------------------------------------------------------
      > -------~->
      >
      > To unsubscribe from this group, send an email to:
      > soaplite-unsubscribe@yahoogroups.com
      >
      >
      >
      > Your use of Yahoo! Groups is subject to
      http://docs.yahoo.com/info/terms/
    • Andersson, Kimmo
      Hi ! Thanks Paul for Your answer. I got confirmation that I was on the right track. So the Perl garbage collection works as I expected. Still I don t
      Message 2 of 2 , Oct 30, 2002
      • 0 Attachment
        Hi !

        Thanks Paul for Your answer. I got confirmation that I was on the right
        track. So the Perl garbage collection works as I expected.
        Still I don't understand why memory consumption seems to be growing if Perl
        should reuse the memory in it's pool. But I continue testing further and
        hopefully the situation is as Paul said, ie the pool stops growing.
        As I mentioned Purify reported some memory leak from a system library but
        the amount remains constant irrespective of the number of sent SOAP
        messages.

        Any way I will keep the options open and not jump to hasty conclusions.

        Many thanks also to Sean Meisner for suggesting launching the Perl module
        with the system command instead of embedding the Perl interpreter.
        I originally considered this but discarded it because I thought it was a
        little bit awkward to return the SOAP response. But of course that can be
        done via a file. The Perl module could write the SOAP response and in case
        of an error, the error message to a file. And I could read the file in my
        C++ code. There could be some performance overhead in this option but it
        should be quite a minor one. Any way this is a serious alternative.

        Best regards

        Kimmo Andersson

        Yomi Solutions Ltd E-mail: Kimmo.Andersson@...
        P.O Box 354, Piippukatu 11 Mobile: +358 50 384 3734

        FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709
        FINLAND

        -----Original Message-----
        From: Paul Kulchenko [mailto:paulclinger@...]
        Sent: 29. lokakuuta 2002 17:52
        To: Andersson, Kimmo; 'soaplite@yahoogroups.com'
        Subject: Re: [soaplite] Using SOAP::Lite and C++, memory consumption
        problem

        Hi Kimmo,

        > I have got the impression that Perl does release memory but it is
        > released to Perl's own memory pool, not to the operating system.

        I believe that's the correct assumption:

        "The way Perl handles memory is very simple. When it needs to
        allocate memory it will allocate everything it needs to perform it's
        tasks. For example, if a variable's content is 1MB, then Perl will
        allocate 1MB for the variable. When that variable goes out of scope
        or is catched by the garbage fairy, Perl will not release that 1MB to
        the kernel right away, it will reuse it in case it needs it later on
        during execution."

        Unless there is a memory leak somewhere in my or Perl's code, this
        memory pool should stop growing after some number of requests. I've
        done extensive tests on my windows and linux boxes chasing down
        memory leaks and tests showed that pool stops growing.

        > 1. Find some mysterious Perl trick which forces Perl to release
        > memory from
        > it's memory pool or makes Perl reuse memory in it's pool more
        > efficiently.
        > - I haven't found even if I have searched intensively
        It may exist, but I'm unaware of any such option.

        > 2. Re-compile Perl with usemultiplicity-option and create an
        > instance of the
        > Perl interpreter for each time a SOAP message is sent.
        > - I don't know the possible consequences and I'm not the only one
        > to use Perl. Others might not agree
        Will probably work, but may have performance impact. Can't say ho
        big.

        > 3. Shutdown and restart the persistent processes which use my
        > library. For example a script is run every Saturday night.
        > - Sounds quite quick and dirty
        It may be better than #2. You may think about adding file semaphore
        and graceful shutdown to your code.

        > 4. Find the bug in my code.
        > - I have tried to follow the instructions in Perlcall and Perlembed
        > documents but You never know
        From what you describe I don't there is any in memory handling piece
        (In other words, I would expect similar behavior from properly
        written code).

        5. it *may* be possible to restrict the amount of memory available
        for your process in your OS. Although the amount of required memory
        depends on the size of the message you want to process/generate, so
        you may not be able to handle all of the messages.

        Unfortunately I can't be more helpful on that. Some other options
        are: (1) reduce the SOAP::Lite's memory footprint, (2) provide
        streaming parsing as an option, (3) do something else. #1 and 2 are
        on my list, but they both only experiments for now and it's not clear
        when they will make into the code.

        Best wishes, Paul.

        --- "Andersson, Kimmo" <Kimmo.Andersson@...> wrote:
        > Hi !
        >
        > I have already posted this message to a Perl newsgroup but as
        > SOAP::Lite is
        > implemented with Perl I thought why not also to this mailing list.
        > This is more a Perl problem, not about SOAP::Lite as such. But I'
        > hoping
        > that some one might have met a similar problem and even solved it.
        >
        > I have written a library in C++ which acts as an wrapper for
        > sending SOAP
        > messages. It is done in HP-UX environment and the SOAP::Lite
        > toolkit is
        > used
        >
        > As SOAP::Lite is implemented with Perl I have written my own Perl
        > module
        > which acts as an intermediary between my C++ code and SOAP::Lite.
        > My own
        > Perl module calls SOAP::Lite and returns the SOAP response to my
        > C++ class
        > which has called it.
        >
        > Unfortunately it has been discovered that the processes that use my
        > library
        > seem consume more and more memory while running and several SOAP
        > messages
        > are sent via my library. This has been verified using several
        > processes and
        > I have also verified it using my own test client, we are absolutely
        > sure
        > that SOAP::Lite and Perl are to be blamed.
        >
        > A clear pattern hasn't however emerged, it is not that say 25 K per
        > each
        > SOAP message memory is used and not released, it varies.
        > Some time memory decreases and several SOAP messages are sent
        > before memory
        > again starts growing. I have used Purify but Purify does not
        > report memory
        > leaks. Actually there leaks from some system libraries but their
        > amount
        > remains constant whether I send one or hundred SOAP messages.
        >
        > Because the Perl we use is not compiled with
        > usemultiplicity-option, the
        > persistent processes which use my library must use just one
        > instance of a
        > class in my library and via it just one instance of the Perl
        > interpreter
        > during their lifetime.
        >
        > I have got the impression that Perl does release memory but it is
        > released
        > to Perl's own memory pool, not to the operating system.
        >
        > So I'm asking is there anything to be done ? I have just one
        > subroutine in
        > my Perl module and all the variables used are lexical variables
        > (my)
        > declared in that subroutine. They should be released when gone out
        > of scope.
        > In my C++ code I use perl_call_argv to call my Perl module and it
        > and
        > reading the Perl stack are surrounded by ENTER/SAVETMPS and
        > FREETMPS/LEAVE.
        > According to the Perlcall document that should get rid of any
        > values
        > returned by the Perl module.
        >
        > So possible solutions ?
        >
        > 1. Find some mysterious Perl trick which forces Perl to release
        > memory from
        > it's memory pool or makes Perl reuse memory in it's pool more
        > efficiently.
        > - I haven't found even if I have searched intensively
        >
        > 2. Re-compile Perl with usemultiplicity-option and create an
        > instance of the
        > Perl interpreter for each time a SOAP message is sent.
        > - I don't know the possible consequences and I'm not the only one
        > to use
        > Perl. Others might not agree
        >
        > 3. Shutdown and restart the persistent processes which use my
        > library. For
        > example a script is run every Saturday night.
        > - Sounds quite quick and dirty
        >
        > 4. Find the bug in my code.
        > - I have tried to follow the instructions in Perlcall and Perlembed
        > documents but You never know
        >
        >
        > Kimmo Andersson
        >
        > Yomi Solutions Ltd E-mail: Kimmo.Andersson@....O
        > Box 354, Piippukatu 11 Mobile: +358 50 384 3734
        > FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709FINLAND
        >
        >
        > ------------------------ Yahoo! Groups Sponsor
        >
        > To unsubscribe from this group, send an email to:
        > soaplite-unsubscribe@yahoogroups.com
        >
        >
        >
        > Your use of Yahoo! Groups is subject to
        > http://docs.yahoo.com/info/terms/
        >
        >


        __________________________________________________
        Do you Yahoo!?
        HotJobs - Search new jobs daily now
        http://hotjobs.yahoo.com/
      Your message has been successfully submitted and would be delivered to recipients shortly.