Loading ...
Sorry, an error occurred while loading the content.

RE: [PBML] Memory release problem?

Expand Messages
  • michael watson (IAH-C)
    Hi Thanks for the response. The data is directed to a file, not to the screen, and there are plenty of new-lines in there so perl should be flushing it s
    Message 1 of 9 , Oct 3, 2007
    • 0 Attachment
      Hi

      Thanks for the response. The data is directed to a file, not to the
      screen, and there are plenty of new-lines in there so perl should be
      flushing it's buffers enough times.

      Mick

      ________________________________

      From: perl-beginner@yahoogroups.com
      [mailto:perl-beginner@yahoogroups.com] On Behalf Of Ramesh Polepalli
      Sent: 03 October 2007 06:37
      To: perl-beginner@yahoogroups.com
      Subject: RE: [PBML] Memory release problem?



      I Guess,

      The problem is due to the limitation of screen buffer.

      I would suggest you to,

      Try redirecting the output to a file instead of screen and

      share your experience.

      ________________________________

      From: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      [mailto:perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Mick Watson
      Sent: Tuesday, October 02, 2007 3:22 PM
      To: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      Subject: [PBML] Memory release problem?

      Hi

      I am not a beginner to perl, but am a beginner to this problem!

      I create a perl object, read about 500,000 rows from a MySQL
      database, populate some structures and print out a table of results.
      I do this using DBI. I create one database handle, which I
      disconnect from, and about 6 statement handles, all of which I
      explicitly finish.

      After the results have printed, the script simply dies. However, the
      results finish printing after about 4 minutes from execution; the
      process then actually finishes some 20 minutes later - what could be
      going on?

      In my object's DESTROY method, I explicitly disconnect from the db
      handle (takes less than one second) and explicitly undef $self (takes
      less than one second). So I am guessing that the rest of the 20
      minutes is perl returning memory to the OS...? Why would it take so
      long? The server has 3Gb of RAM, so this should not be a problem.

      Thanks
      Mick

      I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
      Could this be the problem?

      Ramesh Polepalli

      mFormation Technologies Inc.
      Direct Line: +91 80 6620 5939
      Switchboard: +91 80 6620 5900

      http://www.mformation.com <http://www.mformation.com>


      Visit WWW.MFORMATION.COM and see how the MFORMATION SERVICE MANAGER can
      help you to transform your business.

      [Non-text portions of this message have been removed]






      [Non-text portions of this message have been removed]
    • John Francini
      Ah, but Perl doesn t always flush output out to the terminal as it s generated unless you have the special variable $| (or $AUTOFLUSH if you used English) set
      Message 2 of 9 , Oct 3, 2007
      • 0 Attachment
        Ah, but Perl doesn't always flush output out to the terminal as it's
        generated unless you have the special variable $| (or $AUTOFLUSH if
        you used English) set to 1. Otherwise, it may not flush until the
        output buffer is full or the program exits.

        John


        At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
        >Hi
        >
        >I have "print STDERR..." statements throught my script, with a call to
        >`date` appended so I know when it happens.
        >I also have similar statements before and after disconnecting from MySQL
        >using DBI and undef-ing $self in my perl object.
        >
        >As far as I can tell, the script has completely finished executing, and
        >then there is about a 20 minute wait for the process to end.
        >
        >Mick
        >
        >________________________________
        >
        >From: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
        >[mailto:<mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com]
        >On Behalf Of Jenda Krynicky
        >Sent: 03 October 2007 11:12
        >To: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
        >Subject: Re: [PBML] Memory release problem?
        >
        >On 2 Oct 2007 at 9:51, Mick Watson wrote:
        >> I am not a beginner to perl, but am a beginner to this problem!
        >>
        >> I create a perl object, read about 500,000 rows from a MySQL
        >> database, populate some structures and print out a table of results.
        >> I do this using DBI. I create one database handle, which I disconnect
        >> from, and about 6 statement handles, all of which I explicitly finish.
        >>
        >> After the results have printed, the script simply dies. However, the
        >> results finish printing after about 4 minutes from execution; the
        >> process then actually finishes some 20 minutes later - what could be
        >> going on?
        >>
        >> In my object's DESTROY method, I explicitly disconnect from the db
        >> handle (takes less than one second) and explicitly undef $self (takes
        >> less than one second). So I am guessing that the rest of the 20
        >> minutes is perl returning memory to the OS...? Why would it take so
        >> long? The server has 3Gb of RAM, so this should not be a problem.
        >
        >I don't think you are looking at the right place. Though it's hard to
        >
        >suggest where to look.
        >
        >What does it do if you add something like
        >
        >END {print "Exiting the script: 1"}
        >
        >before the use statements, after the use statements and at the end of
        >
        >the script Are these messages printed immedialtely after the normal
        >output of the script or at the end of those 20 minutes of waiting?
        >
        >Jenda
        >
        >===== <mailto:Jenda%40Krynicky.cz>Jenda@...
        ><mailto:Jenda%40Krynicky.cz> ===
        ><http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz
        ><<http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz> =====
        >When it comes to wine, women and song, wizards are allowed
        >to get drunk and croon as much as they like.
        >-- Terry Pratchett in Sourcery
        >
        >[Non-text portions of this message have been removed]
        >
        >

        --
        John Francini, francini@...

        "The journey is more important than the destination -- that's part of life.
        If you only live for getting to the end, you're almost always disappointed."
        -- Donald Knuth

        [Non-text portions of this message have been removed]
      • michael watson (IAH-C)
        Hmmm, OK, I will try this, thank you, but as there are only 17000 rows of output, is it really possible perl takes 20 mins to write that to a file?!
        Message 3 of 9 , Oct 3, 2007
        • 0 Attachment
          Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
          of output, is it really possible perl takes 20 mins to write that to a
          file?!

          ________________________________

          From: John Francini [mailto:francini@...]
          Sent: 03 October 2007 14:08
          To: perl-beginner@yahoogroups.com
          Cc: michael watson (IAH-C)
          Subject: RE: [PBML] Memory release problem?


          Ah, but Perl doesn't always flush output out to the terminal as it's
          generated unless you have the special variable $| (or $AUTOFLUSH if you
          used English) set to 1. Otherwise, it may not flush until the output
          buffer is full or the program exits.

          John


          At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:

          Hi

          I have "print STDERR..." statements throught my script, with a
          call to
          `date` appended so I know when it happens.
          I also have similar statements before and after disconnecting
          from MySQL
          using DBI and undef-ing $self in my perl object.

          As far as I can tell, the script has completely finished
          executing, and
          then there is about a 20 minute wait for the process to end.

          Mick

          ________________________________

          From: perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com>
          [mailto:perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
          Sent: 03 October 2007 11:12
          To: perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com>
          Subject: Re: [PBML] Memory release problem?

          On 2 Oct 2007 at 9:51, Mick Watson wrote:
          > I am not a beginner to perl, but am a beginner to this
          problem!
          >
          > I create a perl object, read about 500,000 rows from a MySQL
          > database, populate some structures and print out a table of
          results.
          > I do this using DBI. I create one database handle, which I
          disconnect
          > from, and about 6 statement handles, all of which I explicitly
          finish.
          >
          > After the results have printed, the script simply dies.
          However, the
          > results finish printing after about 4 minutes from execution;
          the
          > process then actually finishes some 20 minutes later - what
          could be
          > going on?
          >
          > In my object's DESTROY method, I explicitly disconnect from
          the db
          > handle (takes less than one second) and explicitly undef $self
          (takes
          > less than one second). So I am guessing that the rest of the
          20
          > minutes is perl returning memory to the OS...? Why would it
          take so
          > long? The server has 3Gb of RAM, so this should not be a
          problem.

          I don't think you are looking at the right place. Though it's
          hard to

          suggest where to look.

          What does it do if you add something like

          END {print "Exiting the script: 1"}

          before the use statements, after the use statements and at the
          end of

          the script Are these messages printed immedialtely after the
          normal
          output of the script or at the end of those 20 minutes of
          waiting?

          Jenda

          ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
          <mailto:Jenda%40Krynicky.cz> ===
          http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
          When it comes to wine, women and song, wizards are allowed
          to get drunk and croon as much as they like.
          -- Terry Pratchett in Sourcery

          [Non-text portions of this message have been removed]





          --
          John Francini, francini@...

          "The journey is more important than the destination -- that's part of
          life.
          If you only live for getting to the end, you're almost always
          disappointed."
          -- Donald Knuth


          [Non-text portions of this message have been removed]
        • John Francini
          Not likely; it s just that output to STDERR/STDOUT is buffered and isn t guaranteed to come out at the moment it s generated unless you ve set $| to 1 on
          Message 4 of 9 , Oct 3, 2007
          • 0 Attachment
            Not likely; it's just that output to STDERR/STDOUT is buffered and
            isn't guaranteed to come out at the moment it's generated unless
            you've set $| to 1 on STDERR and/or STDOUT.

            One way you can see if the disk I/O is indeed taking a long time is
            to watch the size of the output file, either through repeated 'ls'
            commands (Unix/Linux) or 'dir' commands (Windows), or equivalent
            means, in another window.

            John



            On 3 Oct 2007, at 10:12, michael watson (IAH-C) wrote:

            > Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
            > of output, is it really possible perl takes 20 mins to write that to a
            > file?!
            >
            > ________________________________
            >
            > From: John Francini [mailto:francini@...]
            > Sent: 03 October 2007 14:08
            > To: perl-beginner@yahoogroups.com
            > Cc: michael watson (IAH-C)
            > Subject: RE: [PBML] Memory release problem?
            >
            > Ah, but Perl doesn't always flush output out to the terminal as it's
            > generated unless you have the special variable $| (or $AUTOFLUSH if
            > you
            > used English) set to 1. Otherwise, it may not flush until the output
            > buffer is full or the program exits.
            >
            > John
            >
            > At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
            >
            > Hi
            >
            > I have "print STDERR..." statements throught my script, with a
            > call to
            > `date` appended so I know when it happens.
            > I also have similar statements before and after disconnecting
            > from MySQL
            > using DBI and undef-ing $self in my perl object.
            >
            > As far as I can tell, the script has completely finished
            > executing, and
            > then there is about a 20 minute wait for the process to end.
            >
            > Mick
            >
            > ________________________________
            >
            > From: perl-beginner@yahoogroups.com
            > <mailto:perl-beginner%40yahoogroups.com>
            > [mailto:perl-beginner@yahoogroups.com
            > <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
            > Sent: 03 October 2007 11:12
            > To: perl-beginner@yahoogroups.com
            > <mailto:perl-beginner%40yahoogroups.com>
            > Subject: Re: [PBML] Memory release problem?
            >
            > On 2 Oct 2007 at 9:51, Mick Watson wrote:
            > > I am not a beginner to perl, but am a beginner to this
            > problem!
            > >
            > > I create a perl object, read about 500,000 rows from a MySQL
            > > database, populate some structures and print out a table of
            > results.
            > > I do this using DBI. I create one database handle, which I
            > disconnect
            > > from, and about 6 statement handles, all of which I explicitly
            > finish.
            > >
            > > After the results have printed, the script simply dies.
            > However, the
            > > results finish printing after about 4 minutes from execution;
            > the
            > > process then actually finishes some 20 minutes later - what
            > could be
            > > going on?
            > >
            > > In my object's DESTROY method, I explicitly disconnect from
            > the db
            > > handle (takes less than one second) and explicitly undef $self
            > (takes
            > > less than one second). So I am guessing that the rest of the
            > 20
            > > minutes is perl returning memory to the OS...? Why would it
            > take so
            > > long? The server has 3Gb of RAM, so this should not be a
            > problem.
            >
            > I don't think you are looking at the right place. Though it's
            > hard to
            >
            > suggest where to look.
            >
            > What does it do if you add something like
            >
            > END {print "Exiting the script: 1"}
            >
            > before the use statements, after the use statements and at the
            > end of
            >
            > the script Are these messages printed immedialtely after the
            > normal
            > output of the script or at the end of those 20 minutes of
            > waiting?
            >
            > Jenda
            >
            > ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
            > <mailto:Jenda%40Krynicky.cz> ===
            > http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
            > When it comes to wine, women and song, wizards are allowed
            > to get drunk and croon as much as they like.
            > -- Terry Pratchett in Sourcery
            >
            > [Non-text portions of this message have been removed]
            >
            >
            >
            >
            > --
            > John Francini, francini@...
            >
            > "The journey is more important than the destination -- that's part of
            > life.
            > If you only live for getting to the end, you're almost always
            > disappointed."
            > -- Donald Knuth
            >
            > [Non-text portions of this message have been removed]
            >
            >
            >



            [Non-text portions of this message have been removed]
          • merlyn@stonehenge.com
            ... michael Hmmm, OK, I will try this, thank you, but as there are only 17000 rows michael of output, is it really possible perl takes 20 mins to write that
            Message 5 of 9 , Oct 4, 2007
            • 0 Attachment
              >>>>> "michael" == michael watson \(IAH-C\) <michael> writes:

              michael> Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
              michael> of output, is it really possible perl takes 20 mins to write that to a
              michael> file?!

              When you profiled your program using Devel::DProf or Devel::SmallProf,
              where did it show that it was spending its time? It would help to know
              that.

              --
              Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
              <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
              Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
              See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
            Your message has been successfully submitted and would be delivered to recipients shortly.