Loading ...
Sorry, an error occurred while loading the content.

RE: [soaplite] Using SOAP::Lite and C++, memory consumption probl em

Expand Messages
  • Andersson, Kimmo
    Hi ! Thanks Paul for Your answer. I got confirmation that I was on the right track. So the Perl garbage collection works as I expected. Still I don t
    Message 1 of 2 , Oct 30, 2002
    • 0 Attachment
      Hi !

      Thanks Paul for Your answer. I got confirmation that I was on the right
      track. So the Perl garbage collection works as I expected.
      Still I don't understand why memory consumption seems to be growing if Perl
      should reuse the memory in it's pool. But I continue testing further and
      hopefully the situation is as Paul said, ie the pool stops growing.
      As I mentioned Purify reported some memory leak from a system library but
      the amount remains constant irrespective of the number of sent SOAP
      messages.

      Any way I will keep the options open and not jump to hasty conclusions.

      Many thanks also to Sean Meisner for suggesting launching the Perl module
      with the system command instead of embedding the Perl interpreter.
      I originally considered this but discarded it because I thought it was a
      little bit awkward to return the SOAP response. But of course that can be
      done via a file. The Perl module could write the SOAP response and in case
      of an error, the error message to a file. And I could read the file in my
      C++ code. There could be some performance overhead in this option but it
      should be quite a minor one. Any way this is a serious alternative.

      Best regards

      Kimmo Andersson

      Yomi Solutions Ltd E-mail: Kimmo.Andersson@...
      P.O Box 354, Piippukatu 11 Mobile: +358 50 384 3734

      FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709
      FINLAND

      -----Original Message-----
      From: Paul Kulchenko [mailto:paulclinger@...]
      Sent: 29. lokakuuta 2002 17:52
      To: Andersson, Kimmo; 'soaplite@yahoogroups.com'
      Subject: Re: [soaplite] Using SOAP::Lite and C++, memory consumption
      problem

      Hi Kimmo,

      > I have got the impression that Perl does release memory but it is
      > released to Perl's own memory pool, not to the operating system.

      I believe that's the correct assumption:

      "The way Perl handles memory is very simple. When it needs to
      allocate memory it will allocate everything it needs to perform it's
      tasks. For example, if a variable's content is 1MB, then Perl will
      allocate 1MB for the variable. When that variable goes out of scope
      or is catched by the garbage fairy, Perl will not release that 1MB to
      the kernel right away, it will reuse it in case it needs it later on
      during execution."

      Unless there is a memory leak somewhere in my or Perl's code, this
      memory pool should stop growing after some number of requests. I've
      done extensive tests on my windows and linux boxes chasing down
      memory leaks and tests showed that pool stops growing.

      > 1. Find some mysterious Perl trick which forces Perl to release
      > memory from
      > it's memory pool or makes Perl reuse memory in it's pool more
      > efficiently.
      > - I haven't found even if I have searched intensively
      It may exist, but I'm unaware of any such option.

      > 2. Re-compile Perl with usemultiplicity-option and create an
      > instance of the
      > Perl interpreter for each time a SOAP message is sent.
      > - I don't know the possible consequences and I'm not the only one
      > to use Perl. Others might not agree
      Will probably work, but may have performance impact. Can't say ho
      big.

      > 3. Shutdown and restart the persistent processes which use my
      > library. For example a script is run every Saturday night.
      > - Sounds quite quick and dirty
      It may be better than #2. You may think about adding file semaphore
      and graceful shutdown to your code.

      > 4. Find the bug in my code.
      > - I have tried to follow the instructions in Perlcall and Perlembed
      > documents but You never know
      From what you describe I don't there is any in memory handling piece
      (In other words, I would expect similar behavior from properly
      written code).

      5. it *may* be possible to restrict the amount of memory available
      for your process in your OS. Although the amount of required memory
      depends on the size of the message you want to process/generate, so
      you may not be able to handle all of the messages.

      Unfortunately I can't be more helpful on that. Some other options
      are: (1) reduce the SOAP::Lite's memory footprint, (2) provide
      streaming parsing as an option, (3) do something else. #1 and 2 are
      on my list, but they both only experiments for now and it's not clear
      when they will make into the code.

      Best wishes, Paul.

      --- "Andersson, Kimmo" <Kimmo.Andersson@...> wrote:
      > Hi !
      >
      > I have already posted this message to a Perl newsgroup but as
      > SOAP::Lite is
      > implemented with Perl I thought why not also to this mailing list.
      > This is more a Perl problem, not about SOAP::Lite as such. But I'
      > hoping
      > that some one might have met a similar problem and even solved it.
      >
      > I have written a library in C++ which acts as an wrapper for
      > sending SOAP
      > messages. It is done in HP-UX environment and the SOAP::Lite
      > toolkit is
      > used
      >
      > As SOAP::Lite is implemented with Perl I have written my own Perl
      > module
      > which acts as an intermediary between my C++ code and SOAP::Lite.
      > My own
      > Perl module calls SOAP::Lite and returns the SOAP response to my
      > C++ class
      > which has called it.
      >
      > Unfortunately it has been discovered that the processes that use my
      > library
      > seem consume more and more memory while running and several SOAP
      > messages
      > are sent via my library. This has been verified using several
      > processes and
      > I have also verified it using my own test client, we are absolutely
      > sure
      > that SOAP::Lite and Perl are to be blamed.
      >
      > A clear pattern hasn't however emerged, it is not that say 25 K per
      > each
      > SOAP message memory is used and not released, it varies.
      > Some time memory decreases and several SOAP messages are sent
      > before memory
      > again starts growing. I have used Purify but Purify does not
      > report memory
      > leaks. Actually there leaks from some system libraries but their
      > amount
      > remains constant whether I send one or hundred SOAP messages.
      >
      > Because the Perl we use is not compiled with
      > usemultiplicity-option, the
      > persistent processes which use my library must use just one
      > instance of a
      > class in my library and via it just one instance of the Perl
      > interpreter
      > during their lifetime.
      >
      > I have got the impression that Perl does release memory but it is
      > released
      > to Perl's own memory pool, not to the operating system.
      >
      > So I'm asking is there anything to be done ? I have just one
      > subroutine in
      > my Perl module and all the variables used are lexical variables
      > (my)
      > declared in that subroutine. They should be released when gone out
      > of scope.
      > In my C++ code I use perl_call_argv to call my Perl module and it
      > and
      > reading the Perl stack are surrounded by ENTER/SAVETMPS and
      > FREETMPS/LEAVE.
      > According to the Perlcall document that should get rid of any
      > values
      > returned by the Perl module.
      >
      > So possible solutions ?
      >
      > 1. Find some mysterious Perl trick which forces Perl to release
      > memory from
      > it's memory pool or makes Perl reuse memory in it's pool more
      > efficiently.
      > - I haven't found even if I have searched intensively
      >
      > 2. Re-compile Perl with usemultiplicity-option and create an
      > instance of the
      > Perl interpreter for each time a SOAP message is sent.
      > - I don't know the possible consequences and I'm not the only one
      > to use
      > Perl. Others might not agree
      >
      > 3. Shutdown and restart the persistent processes which use my
      > library. For
      > example a script is run every Saturday night.
      > - Sounds quite quick and dirty
      >
      > 4. Find the bug in my code.
      > - I have tried to follow the instructions in Perlcall and Perlembed
      > documents but You never know
      >
      >
      > Kimmo Andersson
      >
      > Yomi Solutions Ltd E-mail: Kimmo.Andersson@....O
      > Box 354, Piippukatu 11 Mobile: +358 50 384 3734
      > FIN-40101 JYVÄSKYLÄ Fax : +358 14 240 4709FINLAND
      >
      >
      > ------------------------ Yahoo! Groups Sponsor
      >
      > To unsubscribe from this group, send an email to:
      > soaplite-unsubscribe@yahoogroups.com
      >
      >
      >
      > Your use of Yahoo! Groups is subject to
      > http://docs.yahoo.com/info/terms/
      >
      >


      __________________________________________________
      Do you Yahoo!?
      HotJobs - Search new jobs daily now
      http://hotjobs.yahoo.com/
    Your message has been successfully submitted and would be delivered to recipients shortly.