Loading ...
Sorry, an error occurred while loading the content.

RE: [PBML] Memory release problem?

Expand Messages
  • michael watson (IAH-C)
    Hi Thanks for the response. The data is directed to a file, not to the screen, and there are plenty of new-lines in there so perl should be flushing it s
    Message 1 of 9 , Oct 3, 2007
    • 0 Attachment
      Hi

      Thanks for the response. The data is directed to a file, not to the
      screen, and there are plenty of new-lines in there so perl should be
      flushing it's buffers enough times.

      Mick

      ________________________________

      From: perl-beginner@yahoogroups.com
      [mailto:perl-beginner@yahoogroups.com] On Behalf Of Ramesh Polepalli
      Sent: 03 October 2007 06:37
      To: perl-beginner@yahoogroups.com
      Subject: RE: [PBML] Memory release problem?



      I Guess,

      The problem is due to the limitation of screen buffer.

      I would suggest you to,

      Try redirecting the output to a file instead of screen and

      share your experience.

      ________________________________

      From: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      [mailto:perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Mick Watson
      Sent: Tuesday, October 02, 2007 3:22 PM
      To: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      Subject: [PBML] Memory release problem?

      Hi

      I am not a beginner to perl, but am a beginner to this problem!

      I create a perl object, read about 500,000 rows from a MySQL
      database, populate some structures and print out a table of results.
      I do this using DBI. I create one database handle, which I
      disconnect from, and about 6 statement handles, all of which I
      explicitly finish.

      After the results have printed, the script simply dies. However, the
      results finish printing after about 4 minutes from execution; the
      process then actually finishes some 20 minutes later - what could be
      going on?

      In my object's DESTROY method, I explicitly disconnect from the db
      handle (takes less than one second) and explicitly undef $self (takes
      less than one second). So I am guessing that the rest of the 20
      minutes is perl returning memory to the OS...? Why would it take so
      long? The server has 3Gb of RAM, so this should not be a problem.

      Thanks
      Mick

      I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
      Could this be the problem?

      Ramesh Polepalli

      mFormation Technologies Inc.
      Direct Line: +91 80 6620 5939
      Switchboard: +91 80 6620 5900

      http://www.mformation.com <http://www.mformation.com>


      Visit WWW.MFORMATION.COM and see how the MFORMATION SERVICE MANAGER can
      help you to transform your business.

      [Non-text portions of this message have been removed]






      [Non-text portions of this message have been removed]
    • Jenda Krynicky
      ... I don t think you are looking at the right place. Though it s hard to suggest where to look. What does it do if you add something like END {print Exiting
      Message 2 of 9 , Oct 3, 2007
      • 0 Attachment
        On 2 Oct 2007 at 9:51, Mick Watson wrote:
        > I am not a beginner to perl, but am a beginner to this problem!
        >
        > I create a perl object, read about 500,000 rows from a MySQL
        > database, populate some structures and print out a table of results.
        > I do this using DBI. I create one database handle, which I disconnect
        > from, and about 6 statement handles, all of which I explicitly finish.
        >
        > After the results have printed, the script simply dies. However, the
        > results finish printing after about 4 minutes from execution; the
        > process then actually finishes some 20 minutes later - what could be
        > going on?
        >
        > In my object's DESTROY method, I explicitly disconnect from the db
        > handle (takes less than one second) and explicitly undef $self (takes
        > less than one second). So I am guessing that the rest of the 20
        > minutes is perl returning memory to the OS...? Why would it take so
        > long? The server has 3Gb of RAM, so this should not be a problem.

        I don't think you are looking at the right place. Though it's hard to

        suggest where to look.

        What does it do if you add something like

        END {print "Exiting the script: 1"}

        before the use statements, after the use statements and at the end of

        the script Are these messages printed immedialtely after the normal
        output of the script or at the end of those 20 minutes of waiting?

        Jenda

        ===== Jenda@... === http://Jenda.Krynicky.cz =====
        When it comes to wine, women and song, wizards are allowed
        to get drunk and croon as much as they like.
        -- Terry Pratchett in Sourcery
      • michael watson (IAH-C)
        Hi I have print STDERR... statements throught my script, with a call to `date` appended so I know when it happens. I also have similar statements before and
        Message 3 of 9 , Oct 3, 2007
        • 0 Attachment
          Hi

          I have "print STDERR..." statements throught my script, with a call to
          `date` appended so I know when it happens.
          I also have similar statements before and after disconnecting from MySQL
          using DBI and undef-ing $self in my perl object.

          As far as I can tell, the script has completely finished executing, and
          then there is about a 20 minute wait for the process to end.

          Mick

          ________________________________

          From: perl-beginner@yahoogroups.com
          [mailto:perl-beginner@yahoogroups.com] On Behalf Of Jenda Krynicky
          Sent: 03 October 2007 11:12
          To: perl-beginner@yahoogroups.com
          Subject: Re: [PBML] Memory release problem?



          On 2 Oct 2007 at 9:51, Mick Watson wrote:
          > I am not a beginner to perl, but am a beginner to this problem!
          >
          > I create a perl object, read about 500,000 rows from a MySQL
          > database, populate some structures and print out a table of results.
          > I do this using DBI. I create one database handle, which I disconnect
          > from, and about 6 statement handles, all of which I explicitly finish.
          >
          > After the results have printed, the script simply dies. However, the
          > results finish printing after about 4 minutes from execution; the
          > process then actually finishes some 20 minutes later - what could be
          > going on?
          >
          > In my object's DESTROY method, I explicitly disconnect from the db
          > handle (takes less than one second) and explicitly undef $self (takes
          > less than one second). So I am guessing that the rest of the 20
          > minutes is perl returning memory to the OS...? Why would it take so
          > long? The server has 3Gb of RAM, so this should not be a problem.

          I don't think you are looking at the right place. Though it's hard to

          suggest where to look.

          What does it do if you add something like

          END {print "Exiting the script: 1"}

          before the use statements, after the use statements and at the end of

          the script Are these messages printed immedialtely after the normal
          output of the script or at the end of those 20 minutes of waiting?

          Jenda

          ===== Jenda@... <mailto:Jenda%40Krynicky.cz> ===
          http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
          When it comes to wine, women and song, wizards are allowed
          to get drunk and croon as much as they like.
          -- Terry Pratchett in Sourcery






          [Non-text portions of this message have been removed]
        • John Francini
          Ah, but Perl doesn t always flush output out to the terminal as it s generated unless you have the special variable $| (or $AUTOFLUSH if you used English) set
          Message 4 of 9 , Oct 3, 2007
          • 0 Attachment
            Ah, but Perl doesn't always flush output out to the terminal as it's
            generated unless you have the special variable $| (or $AUTOFLUSH if
            you used English) set to 1. Otherwise, it may not flush until the
            output buffer is full or the program exits.

            John


            At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
            >Hi
            >
            >I have "print STDERR..." statements throught my script, with a call to
            >`date` appended so I know when it happens.
            >I also have similar statements before and after disconnecting from MySQL
            >using DBI and undef-ing $self in my perl object.
            >
            >As far as I can tell, the script has completely finished executing, and
            >then there is about a 20 minute wait for the process to end.
            >
            >Mick
            >
            >________________________________
            >
            >From: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
            >[mailto:<mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com]
            >On Behalf Of Jenda Krynicky
            >Sent: 03 October 2007 11:12
            >To: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
            >Subject: Re: [PBML] Memory release problem?
            >
            >On 2 Oct 2007 at 9:51, Mick Watson wrote:
            >> I am not a beginner to perl, but am a beginner to this problem!
            >>
            >> I create a perl object, read about 500,000 rows from a MySQL
            >> database, populate some structures and print out a table of results.
            >> I do this using DBI. I create one database handle, which I disconnect
            >> from, and about 6 statement handles, all of which I explicitly finish.
            >>
            >> After the results have printed, the script simply dies. However, the
            >> results finish printing after about 4 minutes from execution; the
            >> process then actually finishes some 20 minutes later - what could be
            >> going on?
            >>
            >> In my object's DESTROY method, I explicitly disconnect from the db
            >> handle (takes less than one second) and explicitly undef $self (takes
            >> less than one second). So I am guessing that the rest of the 20
            >> minutes is perl returning memory to the OS...? Why would it take so
            >> long? The server has 3Gb of RAM, so this should not be a problem.
            >
            >I don't think you are looking at the right place. Though it's hard to
            >
            >suggest where to look.
            >
            >What does it do if you add something like
            >
            >END {print "Exiting the script: 1"}
            >
            >before the use statements, after the use statements and at the end of
            >
            >the script Are these messages printed immedialtely after the normal
            >output of the script or at the end of those 20 minutes of waiting?
            >
            >Jenda
            >
            >===== <mailto:Jenda%40Krynicky.cz>Jenda@...
            ><mailto:Jenda%40Krynicky.cz> ===
            ><http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz
            ><<http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz> =====
            >When it comes to wine, women and song, wizards are allowed
            >to get drunk and croon as much as they like.
            >-- Terry Pratchett in Sourcery
            >
            >[Non-text portions of this message have been removed]
            >
            >

            --
            John Francini, francini@...

            "The journey is more important than the destination -- that's part of life.
            If you only live for getting to the end, you're almost always disappointed."
            -- Donald Knuth

            [Non-text portions of this message have been removed]
          • michael watson (IAH-C)
            Hmmm, OK, I will try this, thank you, but as there are only 17000 rows of output, is it really possible perl takes 20 mins to write that to a file?!
            Message 5 of 9 , Oct 3, 2007
            • 0 Attachment
              Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
              of output, is it really possible perl takes 20 mins to write that to a
              file?!

              ________________________________

              From: John Francini [mailto:francini@...]
              Sent: 03 October 2007 14:08
              To: perl-beginner@yahoogroups.com
              Cc: michael watson (IAH-C)
              Subject: RE: [PBML] Memory release problem?


              Ah, but Perl doesn't always flush output out to the terminal as it's
              generated unless you have the special variable $| (or $AUTOFLUSH if you
              used English) set to 1. Otherwise, it may not flush until the output
              buffer is full or the program exits.

              John


              At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:

              Hi

              I have "print STDERR..." statements throught my script, with a
              call to
              `date` appended so I know when it happens.
              I also have similar statements before and after disconnecting
              from MySQL
              using DBI and undef-ing $self in my perl object.

              As far as I can tell, the script has completely finished
              executing, and
              then there is about a 20 minute wait for the process to end.

              Mick

              ________________________________

              From: perl-beginner@yahoogroups.com
              <mailto:perl-beginner%40yahoogroups.com>
              [mailto:perl-beginner@yahoogroups.com
              <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
              Sent: 03 October 2007 11:12
              To: perl-beginner@yahoogroups.com
              <mailto:perl-beginner%40yahoogroups.com>
              Subject: Re: [PBML] Memory release problem?

              On 2 Oct 2007 at 9:51, Mick Watson wrote:
              > I am not a beginner to perl, but am a beginner to this
              problem!
              >
              > I create a perl object, read about 500,000 rows from a MySQL
              > database, populate some structures and print out a table of
              results.
              > I do this using DBI. I create one database handle, which I
              disconnect
              > from, and about 6 statement handles, all of which I explicitly
              finish.
              >
              > After the results have printed, the script simply dies.
              However, the
              > results finish printing after about 4 minutes from execution;
              the
              > process then actually finishes some 20 minutes later - what
              could be
              > going on?
              >
              > In my object's DESTROY method, I explicitly disconnect from
              the db
              > handle (takes less than one second) and explicitly undef $self
              (takes
              > less than one second). So I am guessing that the rest of the
              20
              > minutes is perl returning memory to the OS...? Why would it
              take so
              > long? The server has 3Gb of RAM, so this should not be a
              problem.

              I don't think you are looking at the right place. Though it's
              hard to

              suggest where to look.

              What does it do if you add something like

              END {print "Exiting the script: 1"}

              before the use statements, after the use statements and at the
              end of

              the script Are these messages printed immedialtely after the
              normal
              output of the script or at the end of those 20 minutes of
              waiting?

              Jenda

              ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
              <mailto:Jenda%40Krynicky.cz> ===
              http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
              When it comes to wine, women and song, wizards are allowed
              to get drunk and croon as much as they like.
              -- Terry Pratchett in Sourcery

              [Non-text portions of this message have been removed]





              --
              John Francini, francini@...

              "The journey is more important than the destination -- that's part of
              life.
              If you only live for getting to the end, you're almost always
              disappointed."
              -- Donald Knuth


              [Non-text portions of this message have been removed]
            • John Francini
              Not likely; it s just that output to STDERR/STDOUT is buffered and isn t guaranteed to come out at the moment it s generated unless you ve set $| to 1 on
              Message 6 of 9 , Oct 3, 2007
              • 0 Attachment
                Not likely; it's just that output to STDERR/STDOUT is buffered and
                isn't guaranteed to come out at the moment it's generated unless
                you've set $| to 1 on STDERR and/or STDOUT.

                One way you can see if the disk I/O is indeed taking a long time is
                to watch the size of the output file, either through repeated 'ls'
                commands (Unix/Linux) or 'dir' commands (Windows), or equivalent
                means, in another window.

                John



                On 3 Oct 2007, at 10:12, michael watson (IAH-C) wrote:

                > Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                > of output, is it really possible perl takes 20 mins to write that to a
                > file?!
                >
                > ________________________________
                >
                > From: John Francini [mailto:francini@...]
                > Sent: 03 October 2007 14:08
                > To: perl-beginner@yahoogroups.com
                > Cc: michael watson (IAH-C)
                > Subject: RE: [PBML] Memory release problem?
                >
                > Ah, but Perl doesn't always flush output out to the terminal as it's
                > generated unless you have the special variable $| (or $AUTOFLUSH if
                > you
                > used English) set to 1. Otherwise, it may not flush until the output
                > buffer is full or the program exits.
                >
                > John
                >
                > At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
                >
                > Hi
                >
                > I have "print STDERR..." statements throught my script, with a
                > call to
                > `date` appended so I know when it happens.
                > I also have similar statements before and after disconnecting
                > from MySQL
                > using DBI and undef-ing $self in my perl object.
                >
                > As far as I can tell, the script has completely finished
                > executing, and
                > then there is about a 20 minute wait for the process to end.
                >
                > Mick
                >
                > ________________________________
                >
                > From: perl-beginner@yahoogroups.com
                > <mailto:perl-beginner%40yahoogroups.com>
                > [mailto:perl-beginner@yahoogroups.com
                > <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
                > Sent: 03 October 2007 11:12
                > To: perl-beginner@yahoogroups.com
                > <mailto:perl-beginner%40yahoogroups.com>
                > Subject: Re: [PBML] Memory release problem?
                >
                > On 2 Oct 2007 at 9:51, Mick Watson wrote:
                > > I am not a beginner to perl, but am a beginner to this
                > problem!
                > >
                > > I create a perl object, read about 500,000 rows from a MySQL
                > > database, populate some structures and print out a table of
                > results.
                > > I do this using DBI. I create one database handle, which I
                > disconnect
                > > from, and about 6 statement handles, all of which I explicitly
                > finish.
                > >
                > > After the results have printed, the script simply dies.
                > However, the
                > > results finish printing after about 4 minutes from execution;
                > the
                > > process then actually finishes some 20 minutes later - what
                > could be
                > > going on?
                > >
                > > In my object's DESTROY method, I explicitly disconnect from
                > the db
                > > handle (takes less than one second) and explicitly undef $self
                > (takes
                > > less than one second). So I am guessing that the rest of the
                > 20
                > > minutes is perl returning memory to the OS...? Why would it
                > take so
                > > long? The server has 3Gb of RAM, so this should not be a
                > problem.
                >
                > I don't think you are looking at the right place. Though it's
                > hard to
                >
                > suggest where to look.
                >
                > What does it do if you add something like
                >
                > END {print "Exiting the script: 1"}
                >
                > before the use statements, after the use statements and at the
                > end of
                >
                > the script Are these messages printed immedialtely after the
                > normal
                > output of the script or at the end of those 20 minutes of
                > waiting?
                >
                > Jenda
                >
                > ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
                > <mailto:Jenda%40Krynicky.cz> ===
                > http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
                > When it comes to wine, women and song, wizards are allowed
                > to get drunk and croon as much as they like.
                > -- Terry Pratchett in Sourcery
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >
                >
                > --
                > John Francini, francini@...
                >
                > "The journey is more important than the destination -- that's part of
                > life.
                > If you only live for getting to the end, you're almost always
                > disappointed."
                > -- Donald Knuth
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >



                [Non-text portions of this message have been removed]
              • merlyn@stonehenge.com
                ... michael Hmmm, OK, I will try this, thank you, but as there are only 17000 rows michael of output, is it really possible perl takes 20 mins to write that
                Message 7 of 9 , Oct 4, 2007
                • 0 Attachment
                  >>>>> "michael" == michael watson \(IAH-C\) <michael> writes:

                  michael> Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                  michael> of output, is it really possible perl takes 20 mins to write that to a
                  michael> file?!

                  When you profiled your program using Devel::DProf or Devel::SmallProf,
                  where did it show that it was spending its time? It would help to know
                  that.

                  --
                  Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
                  <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
                  Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
                  See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
                Your message has been successfully submitted and would be delivered to recipients shortly.