Loading ...
Sorry, an error occurred while loading the content.

Memory release problem?

Expand Messages
  • Mick Watson
    Hi I am not a beginner to perl, but am a beginner to this problem! I create a perl object, read about 500,000 rows from a MySQL database, populate some
    Message 1 of 9 , Oct 2, 2007
    • 0 Attachment
      Hi

      I am not a beginner to perl, but am a beginner to this problem!

      I create a perl object, read about 500,000 rows from a MySQL
      database, populate some structures and print out a table of results.
      I do this using DBI. I create one database handle, which I
      disconnect from, and about 6 statement handles, all of which I
      explicitly finish.

      After the results have printed, the script simply dies. However, the
      results finish printing after about 4 minutes from execution; the
      process then actually finishes some 20 minutes later - what could be
      going on?

      In my object's DESTROY method, I explicitly disconnect from the db
      handle (takes less than one second) and explicitly undef $self (takes
      less than one second). So I am guessing that the rest of the 20
      minutes is perl returning memory to the OS...? Why would it take so
      long? The server has 3Gb of RAM, so this should not be a problem.

      Thanks
      Mick

      I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
      Could this be the problem?
    • Ramesh Polepalli
      I Guess, The problem is due to the limitation of screen buffer. I would suggest you to, Try redirecting the output to a file instead of screen and share your
      Message 2 of 9 , Oct 2, 2007
      • 0 Attachment
        I Guess,

        The problem is due to the limitation of screen buffer.



        I would suggest you to,

        Try redirecting the output to a file instead of screen and

        share your experience.



        ________________________________

        From: perl-beginner@yahoogroups.com
        [mailto:perl-beginner@yahoogroups.com] On Behalf Of Mick Watson
        Sent: Tuesday, October 02, 2007 3:22 PM
        To: perl-beginner@yahoogroups.com
        Subject: [PBML] Memory release problem?



        Hi

        I am not a beginner to perl, but am a beginner to this problem!

        I create a perl object, read about 500,000 rows from a MySQL
        database, populate some structures and print out a table of results.
        I do this using DBI. I create one database handle, which I
        disconnect from, and about 6 statement handles, all of which I
        explicitly finish.

        After the results have printed, the script simply dies. However, the
        results finish printing after about 4 minutes from execution; the
        process then actually finishes some 20 minutes later - what could be
        going on?

        In my object's DESTROY method, I explicitly disconnect from the db
        handle (takes less than one second) and explicitly undef $self (takes
        less than one second). So I am guessing that the rest of the 20
        minutes is perl returning memory to the OS...? Why would it take so
        long? The server has 3Gb of RAM, so this should not be a problem.

        Thanks
        Mick

        I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
        Could this be the problem?







        Ramesh Polepalli









        mFormation Technologies Inc.
        Direct Line: +91 80 6620 5939
        Switchboard: +91 80 6620 5900

        http://www.mformation.com










        Visit WWW.MFORMATION.COM and see how the MFORMATION SERVICE MANAGER can help you to transform your business.

        [Non-text portions of this message have been removed]
      • michael watson (IAH-C)
        Hi Thanks for the response. The data is directed to a file, not to the screen, and there are plenty of new-lines in there so perl should be flushing it s
        Message 3 of 9 , Oct 3, 2007
        • 0 Attachment
          Hi

          Thanks for the response. The data is directed to a file, not to the
          screen, and there are plenty of new-lines in there so perl should be
          flushing it's buffers enough times.

          Mick

          ________________________________

          From: perl-beginner@yahoogroups.com
          [mailto:perl-beginner@yahoogroups.com] On Behalf Of Ramesh Polepalli
          Sent: 03 October 2007 06:37
          To: perl-beginner@yahoogroups.com
          Subject: RE: [PBML] Memory release problem?



          I Guess,

          The problem is due to the limitation of screen buffer.

          I would suggest you to,

          Try redirecting the output to a file instead of screen and

          share your experience.

          ________________________________

          From: perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com>
          [mailto:perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Mick Watson
          Sent: Tuesday, October 02, 2007 3:22 PM
          To: perl-beginner@yahoogroups.com
          <mailto:perl-beginner%40yahoogroups.com>
          Subject: [PBML] Memory release problem?

          Hi

          I am not a beginner to perl, but am a beginner to this problem!

          I create a perl object, read about 500,000 rows from a MySQL
          database, populate some structures and print out a table of results.
          I do this using DBI. I create one database handle, which I
          disconnect from, and about 6 statement handles, all of which I
          explicitly finish.

          After the results have printed, the script simply dies. However, the
          results finish printing after about 4 minutes from execution; the
          process then actually finishes some 20 minutes later - what could be
          going on?

          In my object's DESTROY method, I explicitly disconnect from the db
          handle (takes less than one second) and explicitly undef $self (takes
          less than one second). So I am guessing that the rest of the 20
          minutes is perl returning memory to the OS...? Why would it take so
          long? The server has 3Gb of RAM, so this should not be a problem.

          Thanks
          Mick

          I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
          Could this be the problem?

          Ramesh Polepalli

          mFormation Technologies Inc.
          Direct Line: +91 80 6620 5939
          Switchboard: +91 80 6620 5900

          http://www.mformation.com <http://www.mformation.com>


          Visit WWW.MFORMATION.COM and see how the MFORMATION SERVICE MANAGER can
          help you to transform your business.

          [Non-text portions of this message have been removed]






          [Non-text portions of this message have been removed]
        • Jenda Krynicky
          ... I don t think you are looking at the right place. Though it s hard to suggest where to look. What does it do if you add something like END {print Exiting
          Message 4 of 9 , Oct 3, 2007
          • 0 Attachment
            On 2 Oct 2007 at 9:51, Mick Watson wrote:
            > I am not a beginner to perl, but am a beginner to this problem!
            >
            > I create a perl object, read about 500,000 rows from a MySQL
            > database, populate some structures and print out a table of results.
            > I do this using DBI. I create one database handle, which I disconnect
            > from, and about 6 statement handles, all of which I explicitly finish.
            >
            > After the results have printed, the script simply dies. However, the
            > results finish printing after about 4 minutes from execution; the
            > process then actually finishes some 20 minutes later - what could be
            > going on?
            >
            > In my object's DESTROY method, I explicitly disconnect from the db
            > handle (takes less than one second) and explicitly undef $self (takes
            > less than one second). So I am guessing that the rest of the 20
            > minutes is perl returning memory to the OS...? Why would it take so
            > long? The server has 3Gb of RAM, so this should not be a problem.

            I don't think you are looking at the right place. Though it's hard to

            suggest where to look.

            What does it do if you add something like

            END {print "Exiting the script: 1"}

            before the use statements, after the use statements and at the end of

            the script Are these messages printed immedialtely after the normal
            output of the script or at the end of those 20 minutes of waiting?

            Jenda

            ===== Jenda@... === http://Jenda.Krynicky.cz =====
            When it comes to wine, women and song, wizards are allowed
            to get drunk and croon as much as they like.
            -- Terry Pratchett in Sourcery
          • michael watson (IAH-C)
            Hi I have print STDERR... statements throught my script, with a call to `date` appended so I know when it happens. I also have similar statements before and
            Message 5 of 9 , Oct 3, 2007
            • 0 Attachment
              Hi

              I have "print STDERR..." statements throught my script, with a call to
              `date` appended so I know when it happens.
              I also have similar statements before and after disconnecting from MySQL
              using DBI and undef-ing $self in my perl object.

              As far as I can tell, the script has completely finished executing, and
              then there is about a 20 minute wait for the process to end.

              Mick

              ________________________________

              From: perl-beginner@yahoogroups.com
              [mailto:perl-beginner@yahoogroups.com] On Behalf Of Jenda Krynicky
              Sent: 03 October 2007 11:12
              To: perl-beginner@yahoogroups.com
              Subject: Re: [PBML] Memory release problem?



              On 2 Oct 2007 at 9:51, Mick Watson wrote:
              > I am not a beginner to perl, but am a beginner to this problem!
              >
              > I create a perl object, read about 500,000 rows from a MySQL
              > database, populate some structures and print out a table of results.
              > I do this using DBI. I create one database handle, which I disconnect
              > from, and about 6 statement handles, all of which I explicitly finish.
              >
              > After the results have printed, the script simply dies. However, the
              > results finish printing after about 4 minutes from execution; the
              > process then actually finishes some 20 minutes later - what could be
              > going on?
              >
              > In my object's DESTROY method, I explicitly disconnect from the db
              > handle (takes less than one second) and explicitly undef $self (takes
              > less than one second). So I am guessing that the rest of the 20
              > minutes is perl returning memory to the OS...? Why would it take so
              > long? The server has 3Gb of RAM, so this should not be a problem.

              I don't think you are looking at the right place. Though it's hard to

              suggest where to look.

              What does it do if you add something like

              END {print "Exiting the script: 1"}

              before the use statements, after the use statements and at the end of

              the script Are these messages printed immedialtely after the normal
              output of the script or at the end of those 20 minutes of waiting?

              Jenda

              ===== Jenda@... <mailto:Jenda%40Krynicky.cz> ===
              http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
              When it comes to wine, women and song, wizards are allowed
              to get drunk and croon as much as they like.
              -- Terry Pratchett in Sourcery






              [Non-text portions of this message have been removed]
            • John Francini
              Ah, but Perl doesn t always flush output out to the terminal as it s generated unless you have the special variable $| (or $AUTOFLUSH if you used English) set
              Message 6 of 9 , Oct 3, 2007
              • 0 Attachment
                Ah, but Perl doesn't always flush output out to the terminal as it's
                generated unless you have the special variable $| (or $AUTOFLUSH if
                you used English) set to 1. Otherwise, it may not flush until the
                output buffer is full or the program exits.

                John


                At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
                >Hi
                >
                >I have "print STDERR..." statements throught my script, with a call to
                >`date` appended so I know when it happens.
                >I also have similar statements before and after disconnecting from MySQL
                >using DBI and undef-ing $self in my perl object.
                >
                >As far as I can tell, the script has completely finished executing, and
                >then there is about a 20 minute wait for the process to end.
                >
                >Mick
                >
                >________________________________
                >
                >From: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
                >[mailto:<mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com]
                >On Behalf Of Jenda Krynicky
                >Sent: 03 October 2007 11:12
                >To: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
                >Subject: Re: [PBML] Memory release problem?
                >
                >On 2 Oct 2007 at 9:51, Mick Watson wrote:
                >> I am not a beginner to perl, but am a beginner to this problem!
                >>
                >> I create a perl object, read about 500,000 rows from a MySQL
                >> database, populate some structures and print out a table of results.
                >> I do this using DBI. I create one database handle, which I disconnect
                >> from, and about 6 statement handles, all of which I explicitly finish.
                >>
                >> After the results have printed, the script simply dies. However, the
                >> results finish printing after about 4 minutes from execution; the
                >> process then actually finishes some 20 minutes later - what could be
                >> going on?
                >>
                >> In my object's DESTROY method, I explicitly disconnect from the db
                >> handle (takes less than one second) and explicitly undef $self (takes
                >> less than one second). So I am guessing that the rest of the 20
                >> minutes is perl returning memory to the OS...? Why would it take so
                >> long? The server has 3Gb of RAM, so this should not be a problem.
                >
                >I don't think you are looking at the right place. Though it's hard to
                >
                >suggest where to look.
                >
                >What does it do if you add something like
                >
                >END {print "Exiting the script: 1"}
                >
                >before the use statements, after the use statements and at the end of
                >
                >the script Are these messages printed immedialtely after the normal
                >output of the script or at the end of those 20 minutes of waiting?
                >
                >Jenda
                >
                >===== <mailto:Jenda%40Krynicky.cz>Jenda@...
                ><mailto:Jenda%40Krynicky.cz> ===
                ><http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz
                ><<http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz> =====
                >When it comes to wine, women and song, wizards are allowed
                >to get drunk and croon as much as they like.
                >-- Terry Pratchett in Sourcery
                >
                >[Non-text portions of this message have been removed]
                >
                >

                --
                John Francini, francini@...

                "The journey is more important than the destination -- that's part of life.
                If you only live for getting to the end, you're almost always disappointed."
                -- Donald Knuth

                [Non-text portions of this message have been removed]
              • michael watson (IAH-C)
                Hmmm, OK, I will try this, thank you, but as there are only 17000 rows of output, is it really possible perl takes 20 mins to write that to a file?!
                Message 7 of 9 , Oct 3, 2007
                • 0 Attachment
                  Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                  of output, is it really possible perl takes 20 mins to write that to a
                  file?!

                  ________________________________

                  From: John Francini [mailto:francini@...]
                  Sent: 03 October 2007 14:08
                  To: perl-beginner@yahoogroups.com
                  Cc: michael watson (IAH-C)
                  Subject: RE: [PBML] Memory release problem?


                  Ah, but Perl doesn't always flush output out to the terminal as it's
                  generated unless you have the special variable $| (or $AUTOFLUSH if you
                  used English) set to 1. Otherwise, it may not flush until the output
                  buffer is full or the program exits.

                  John


                  At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:

                  Hi

                  I have "print STDERR..." statements throught my script, with a
                  call to
                  `date` appended so I know when it happens.
                  I also have similar statements before and after disconnecting
                  from MySQL
                  using DBI and undef-ing $self in my perl object.

                  As far as I can tell, the script has completely finished
                  executing, and
                  then there is about a 20 minute wait for the process to end.

                  Mick

                  ________________________________

                  From: perl-beginner@yahoogroups.com
                  <mailto:perl-beginner%40yahoogroups.com>
                  [mailto:perl-beginner@yahoogroups.com
                  <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
                  Sent: 03 October 2007 11:12
                  To: perl-beginner@yahoogroups.com
                  <mailto:perl-beginner%40yahoogroups.com>
                  Subject: Re: [PBML] Memory release problem?

                  On 2 Oct 2007 at 9:51, Mick Watson wrote:
                  > I am not a beginner to perl, but am a beginner to this
                  problem!
                  >
                  > I create a perl object, read about 500,000 rows from a MySQL
                  > database, populate some structures and print out a table of
                  results.
                  > I do this using DBI. I create one database handle, which I
                  disconnect
                  > from, and about 6 statement handles, all of which I explicitly
                  finish.
                  >
                  > After the results have printed, the script simply dies.
                  However, the
                  > results finish printing after about 4 minutes from execution;
                  the
                  > process then actually finishes some 20 minutes later - what
                  could be
                  > going on?
                  >
                  > In my object's DESTROY method, I explicitly disconnect from
                  the db
                  > handle (takes less than one second) and explicitly undef $self
                  (takes
                  > less than one second). So I am guessing that the rest of the
                  20
                  > minutes is perl returning memory to the OS...? Why would it
                  take so
                  > long? The server has 3Gb of RAM, so this should not be a
                  problem.

                  I don't think you are looking at the right place. Though it's
                  hard to

                  suggest where to look.

                  What does it do if you add something like

                  END {print "Exiting the script: 1"}

                  before the use statements, after the use statements and at the
                  end of

                  the script Are these messages printed immedialtely after the
                  normal
                  output of the script or at the end of those 20 minutes of
                  waiting?

                  Jenda

                  ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
                  <mailto:Jenda%40Krynicky.cz> ===
                  http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
                  When it comes to wine, women and song, wizards are allowed
                  to get drunk and croon as much as they like.
                  -- Terry Pratchett in Sourcery

                  [Non-text portions of this message have been removed]





                  --
                  John Francini, francini@...

                  "The journey is more important than the destination -- that's part of
                  life.
                  If you only live for getting to the end, you're almost always
                  disappointed."
                  -- Donald Knuth


                  [Non-text portions of this message have been removed]
                • John Francini
                  Not likely; it s just that output to STDERR/STDOUT is buffered and isn t guaranteed to come out at the moment it s generated unless you ve set $| to 1 on
                  Message 8 of 9 , Oct 3, 2007
                  • 0 Attachment
                    Not likely; it's just that output to STDERR/STDOUT is buffered and
                    isn't guaranteed to come out at the moment it's generated unless
                    you've set $| to 1 on STDERR and/or STDOUT.

                    One way you can see if the disk I/O is indeed taking a long time is
                    to watch the size of the output file, either through repeated 'ls'
                    commands (Unix/Linux) or 'dir' commands (Windows), or equivalent
                    means, in another window.

                    John



                    On 3 Oct 2007, at 10:12, michael watson (IAH-C) wrote:

                    > Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                    > of output, is it really possible perl takes 20 mins to write that to a
                    > file?!
                    >
                    > ________________________________
                    >
                    > From: John Francini [mailto:francini@...]
                    > Sent: 03 October 2007 14:08
                    > To: perl-beginner@yahoogroups.com
                    > Cc: michael watson (IAH-C)
                    > Subject: RE: [PBML] Memory release problem?
                    >
                    > Ah, but Perl doesn't always flush output out to the terminal as it's
                    > generated unless you have the special variable $| (or $AUTOFLUSH if
                    > you
                    > used English) set to 1. Otherwise, it may not flush until the output
                    > buffer is full or the program exits.
                    >
                    > John
                    >
                    > At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
                    >
                    > Hi
                    >
                    > I have "print STDERR..." statements throught my script, with a
                    > call to
                    > `date` appended so I know when it happens.
                    > I also have similar statements before and after disconnecting
                    > from MySQL
                    > using DBI and undef-ing $self in my perl object.
                    >
                    > As far as I can tell, the script has completely finished
                    > executing, and
                    > then there is about a 20 minute wait for the process to end.
                    >
                    > Mick
                    >
                    > ________________________________
                    >
                    > From: perl-beginner@yahoogroups.com
                    > <mailto:perl-beginner%40yahoogroups.com>
                    > [mailto:perl-beginner@yahoogroups.com
                    > <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
                    > Sent: 03 October 2007 11:12
                    > To: perl-beginner@yahoogroups.com
                    > <mailto:perl-beginner%40yahoogroups.com>
                    > Subject: Re: [PBML] Memory release problem?
                    >
                    > On 2 Oct 2007 at 9:51, Mick Watson wrote:
                    > > I am not a beginner to perl, but am a beginner to this
                    > problem!
                    > >
                    > > I create a perl object, read about 500,000 rows from a MySQL
                    > > database, populate some structures and print out a table of
                    > results.
                    > > I do this using DBI. I create one database handle, which I
                    > disconnect
                    > > from, and about 6 statement handles, all of which I explicitly
                    > finish.
                    > >
                    > > After the results have printed, the script simply dies.
                    > However, the
                    > > results finish printing after about 4 minutes from execution;
                    > the
                    > > process then actually finishes some 20 minutes later - what
                    > could be
                    > > going on?
                    > >
                    > > In my object's DESTROY method, I explicitly disconnect from
                    > the db
                    > > handle (takes less than one second) and explicitly undef $self
                    > (takes
                    > > less than one second). So I am guessing that the rest of the
                    > 20
                    > > minutes is perl returning memory to the OS...? Why would it
                    > take so
                    > > long? The server has 3Gb of RAM, so this should not be a
                    > problem.
                    >
                    > I don't think you are looking at the right place. Though it's
                    > hard to
                    >
                    > suggest where to look.
                    >
                    > What does it do if you add something like
                    >
                    > END {print "Exiting the script: 1"}
                    >
                    > before the use statements, after the use statements and at the
                    > end of
                    >
                    > the script Are these messages printed immedialtely after the
                    > normal
                    > output of the script or at the end of those 20 minutes of
                    > waiting?
                    >
                    > Jenda
                    >
                    > ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
                    > <mailto:Jenda%40Krynicky.cz> ===
                    > http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
                    > When it comes to wine, women and song, wizards are allowed
                    > to get drunk and croon as much as they like.
                    > -- Terry Pratchett in Sourcery
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >
                    >
                    > --
                    > John Francini, francini@...
                    >
                    > "The journey is more important than the destination -- that's part of
                    > life.
                    > If you only live for getting to the end, you're almost always
                    > disappointed."
                    > -- Donald Knuth
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >



                    [Non-text portions of this message have been removed]
                  • merlyn@stonehenge.com
                    ... michael Hmmm, OK, I will try this, thank you, but as there are only 17000 rows michael of output, is it really possible perl takes 20 mins to write that
                    Message 9 of 9 , Oct 4, 2007
                    • 0 Attachment
                      >>>>> "michael" == michael watson \(IAH-C\) <michael> writes:

                      michael> Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                      michael> of output, is it really possible perl takes 20 mins to write that to a
                      michael> file?!

                      When you profiled your program using Devel::DProf or Devel::SmallProf,
                      where did it show that it was spending its time? It would help to know
                      that.

                      --
                      Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
                      <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
                      Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
                      See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
                    Your message has been successfully submitted and would be delivered to recipients shortly.