Loading ...
Sorry, an error occurred while loading the content.

RE: [PBML] Memory release problem?

Expand Messages
  • michael watson (IAH-C)
    Hi Thanks for the response. The data is directed to a file, not to the screen, and there are plenty of new-lines in there so perl should be flushing it s
    Message 1 of 9 , Oct 3, 2007
    • 0 Attachment
      Hi

      Thanks for the response. The data is directed to a file, not to the
      screen, and there are plenty of new-lines in there so perl should be
      flushing it's buffers enough times.

      Mick

      ________________________________

      From: perl-beginner@yahoogroups.com
      [mailto:perl-beginner@yahoogroups.com] On Behalf Of Ramesh Polepalli
      Sent: 03 October 2007 06:37
      To: perl-beginner@yahoogroups.com
      Subject: RE: [PBML] Memory release problem?



      I Guess,

      The problem is due to the limitation of screen buffer.

      I would suggest you to,

      Try redirecting the output to a file instead of screen and

      share your experience.

      ________________________________

      From: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      [mailto:perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Mick Watson
      Sent: Tuesday, October 02, 2007 3:22 PM
      To: perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com>
      Subject: [PBML] Memory release problem?

      Hi

      I am not a beginner to perl, but am a beginner to this problem!

      I create a perl object, read about 500,000 rows from a MySQL
      database, populate some structures and print out a table of results.
      I do this using DBI. I create one database handle, which I
      disconnect from, and about 6 statement handles, all of which I
      explicitly finish.

      After the results have printed, the script simply dies. However, the
      results finish printing after about 4 minutes from execution; the
      process then actually finishes some 20 minutes later - what could be
      going on?

      In my object's DESTROY method, I explicitly disconnect from the db
      handle (takes less than one second) and explicitly undef $self (takes
      less than one second). So I am guessing that the rest of the 20
      minutes is perl returning memory to the OS...? Why would it take so
      long? The server has 3Gb of RAM, so this should not be a problem.

      Thanks
      Mick

      I am running an old-ish version of SuSe Linux Enterprise Server 8.2.
      Could this be the problem?

      Ramesh Polepalli

      mFormation Technologies Inc.
      Direct Line: +91 80 6620 5939
      Switchboard: +91 80 6620 5900

      http://www.mformation.com <http://www.mformation.com>


      Visit WWW.MFORMATION.COM and see how the MFORMATION SERVICE MANAGER can
      help you to transform your business.

      [Non-text portions of this message have been removed]






      [Non-text portions of this message have been removed]
    • michael watson (IAH-C)
      Hi I have print STDERR... statements throught my script, with a call to `date` appended so I know when it happens. I also have similar statements before and
      Message 2 of 9 , Oct 3, 2007
      • 0 Attachment
        Hi

        I have "print STDERR..." statements throught my script, with a call to
        `date` appended so I know when it happens.
        I also have similar statements before and after disconnecting from MySQL
        using DBI and undef-ing $self in my perl object.

        As far as I can tell, the script has completely finished executing, and
        then there is about a 20 minute wait for the process to end.

        Mick

        ________________________________

        From: perl-beginner@yahoogroups.com
        [mailto:perl-beginner@yahoogroups.com] On Behalf Of Jenda Krynicky
        Sent: 03 October 2007 11:12
        To: perl-beginner@yahoogroups.com
        Subject: Re: [PBML] Memory release problem?



        On 2 Oct 2007 at 9:51, Mick Watson wrote:
        > I am not a beginner to perl, but am a beginner to this problem!
        >
        > I create a perl object, read about 500,000 rows from a MySQL
        > database, populate some structures and print out a table of results.
        > I do this using DBI. I create one database handle, which I disconnect
        > from, and about 6 statement handles, all of which I explicitly finish.
        >
        > After the results have printed, the script simply dies. However, the
        > results finish printing after about 4 minutes from execution; the
        > process then actually finishes some 20 minutes later - what could be
        > going on?
        >
        > In my object's DESTROY method, I explicitly disconnect from the db
        > handle (takes less than one second) and explicitly undef $self (takes
        > less than one second). So I am guessing that the rest of the 20
        > minutes is perl returning memory to the OS...? Why would it take so
        > long? The server has 3Gb of RAM, so this should not be a problem.

        I don't think you are looking at the right place. Though it's hard to

        suggest where to look.

        What does it do if you add something like

        END {print "Exiting the script: 1"}

        before the use statements, after the use statements and at the end of

        the script Are these messages printed immedialtely after the normal
        output of the script or at the end of those 20 minutes of waiting?

        Jenda

        ===== Jenda@... <mailto:Jenda%40Krynicky.cz> ===
        http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
        When it comes to wine, women and song, wizards are allowed
        to get drunk and croon as much as they like.
        -- Terry Pratchett in Sourcery






        [Non-text portions of this message have been removed]
      • John Francini
        Ah, but Perl doesn t always flush output out to the terminal as it s generated unless you have the special variable $| (or $AUTOFLUSH if you used English) set
        Message 3 of 9 , Oct 3, 2007
        • 0 Attachment
          Ah, but Perl doesn't always flush output out to the terminal as it's
          generated unless you have the special variable $| (or $AUTOFLUSH if
          you used English) set to 1. Otherwise, it may not flush until the
          output buffer is full or the program exits.

          John


          At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
          >Hi
          >
          >I have "print STDERR..." statements throught my script, with a call to
          >`date` appended so I know when it happens.
          >I also have similar statements before and after disconnecting from MySQL
          >using DBI and undef-ing $self in my perl object.
          >
          >As far as I can tell, the script has completely finished executing, and
          >then there is about a 20 minute wait for the process to end.
          >
          >Mick
          >
          >________________________________
          >
          >From: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
          >[mailto:<mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com]
          >On Behalf Of Jenda Krynicky
          >Sent: 03 October 2007 11:12
          >To: <mailto:perl-beginner%40yahoogroups.com>perl-beginner@yahoogroups.com
          >Subject: Re: [PBML] Memory release problem?
          >
          >On 2 Oct 2007 at 9:51, Mick Watson wrote:
          >> I am not a beginner to perl, but am a beginner to this problem!
          >>
          >> I create a perl object, read about 500,000 rows from a MySQL
          >> database, populate some structures and print out a table of results.
          >> I do this using DBI. I create one database handle, which I disconnect
          >> from, and about 6 statement handles, all of which I explicitly finish.
          >>
          >> After the results have printed, the script simply dies. However, the
          >> results finish printing after about 4 minutes from execution; the
          >> process then actually finishes some 20 minutes later - what could be
          >> going on?
          >>
          >> In my object's DESTROY method, I explicitly disconnect from the db
          >> handle (takes less than one second) and explicitly undef $self (takes
          >> less than one second). So I am guessing that the rest of the 20
          >> minutes is perl returning memory to the OS...? Why would it take so
          >> long? The server has 3Gb of RAM, so this should not be a problem.
          >
          >I don't think you are looking at the right place. Though it's hard to
          >
          >suggest where to look.
          >
          >What does it do if you add something like
          >
          >END {print "Exiting the script: 1"}
          >
          >before the use statements, after the use statements and at the end of
          >
          >the script Are these messages printed immedialtely after the normal
          >output of the script or at the end of those 20 minutes of waiting?
          >
          >Jenda
          >
          >===== <mailto:Jenda%40Krynicky.cz>Jenda@...
          ><mailto:Jenda%40Krynicky.cz> ===
          ><http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz
          ><<http://Jenda.Krynicky.cz>http://Jenda.Krynicky.cz> =====
          >When it comes to wine, women and song, wizards are allowed
          >to get drunk and croon as much as they like.
          >-- Terry Pratchett in Sourcery
          >
          >[Non-text portions of this message have been removed]
          >
          >

          --
          John Francini, francini@...

          "The journey is more important than the destination -- that's part of life.
          If you only live for getting to the end, you're almost always disappointed."
          -- Donald Knuth

          [Non-text portions of this message have been removed]
        • michael watson (IAH-C)
          Hmmm, OK, I will try this, thank you, but as there are only 17000 rows of output, is it really possible perl takes 20 mins to write that to a file?!
          Message 4 of 9 , Oct 3, 2007
          • 0 Attachment
            Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
            of output, is it really possible perl takes 20 mins to write that to a
            file?!

            ________________________________

            From: John Francini [mailto:francini@...]
            Sent: 03 October 2007 14:08
            To: perl-beginner@yahoogroups.com
            Cc: michael watson (IAH-C)
            Subject: RE: [PBML] Memory release problem?


            Ah, but Perl doesn't always flush output out to the terminal as it's
            generated unless you have the special variable $| (or $AUTOFLUSH if you
            used English) set to 1. Otherwise, it may not flush until the output
            buffer is full or the program exits.

            John


            At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:

            Hi

            I have "print STDERR..." statements throught my script, with a
            call to
            `date` appended so I know when it happens.
            I also have similar statements before and after disconnecting
            from MySQL
            using DBI and undef-ing $self in my perl object.

            As far as I can tell, the script has completely finished
            executing, and
            then there is about a 20 minute wait for the process to end.

            Mick

            ________________________________

            From: perl-beginner@yahoogroups.com
            <mailto:perl-beginner%40yahoogroups.com>
            [mailto:perl-beginner@yahoogroups.com
            <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
            Sent: 03 October 2007 11:12
            To: perl-beginner@yahoogroups.com
            <mailto:perl-beginner%40yahoogroups.com>
            Subject: Re: [PBML] Memory release problem?

            On 2 Oct 2007 at 9:51, Mick Watson wrote:
            > I am not a beginner to perl, but am a beginner to this
            problem!
            >
            > I create a perl object, read about 500,000 rows from a MySQL
            > database, populate some structures and print out a table of
            results.
            > I do this using DBI. I create one database handle, which I
            disconnect
            > from, and about 6 statement handles, all of which I explicitly
            finish.
            >
            > After the results have printed, the script simply dies.
            However, the
            > results finish printing after about 4 minutes from execution;
            the
            > process then actually finishes some 20 minutes later - what
            could be
            > going on?
            >
            > In my object's DESTROY method, I explicitly disconnect from
            the db
            > handle (takes less than one second) and explicitly undef $self
            (takes
            > less than one second). So I am guessing that the rest of the
            20
            > minutes is perl returning memory to the OS...? Why would it
            take so
            > long? The server has 3Gb of RAM, so this should not be a
            problem.

            I don't think you are looking at the right place. Though it's
            hard to

            suggest where to look.

            What does it do if you add something like

            END {print "Exiting the script: 1"}

            before the use statements, after the use statements and at the
            end of

            the script Are these messages printed immedialtely after the
            normal
            output of the script or at the end of those 20 minutes of
            waiting?

            Jenda

            ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
            <mailto:Jenda%40Krynicky.cz> ===
            http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
            When it comes to wine, women and song, wizards are allowed
            to get drunk and croon as much as they like.
            -- Terry Pratchett in Sourcery

            [Non-text portions of this message have been removed]





            --
            John Francini, francini@...

            "The journey is more important than the destination -- that's part of
            life.
            If you only live for getting to the end, you're almost always
            disappointed."
            -- Donald Knuth


            [Non-text portions of this message have been removed]
          • John Francini
            Not likely; it s just that output to STDERR/STDOUT is buffered and isn t guaranteed to come out at the moment it s generated unless you ve set $| to 1 on
            Message 5 of 9 , Oct 3, 2007
            • 0 Attachment
              Not likely; it's just that output to STDERR/STDOUT is buffered and
              isn't guaranteed to come out at the moment it's generated unless
              you've set $| to 1 on STDERR and/or STDOUT.

              One way you can see if the disk I/O is indeed taking a long time is
              to watch the size of the output file, either through repeated 'ls'
              commands (Unix/Linux) or 'dir' commands (Windows), or equivalent
              means, in another window.

              John



              On 3 Oct 2007, at 10:12, michael watson (IAH-C) wrote:

              > Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
              > of output, is it really possible perl takes 20 mins to write that to a
              > file?!
              >
              > ________________________________
              >
              > From: John Francini [mailto:francini@...]
              > Sent: 03 October 2007 14:08
              > To: perl-beginner@yahoogroups.com
              > Cc: michael watson (IAH-C)
              > Subject: RE: [PBML] Memory release problem?
              >
              > Ah, but Perl doesn't always flush output out to the terminal as it's
              > generated unless you have the special variable $| (or $AUTOFLUSH if
              > you
              > used English) set to 1. Otherwise, it may not flush until the output
              > buffer is full or the program exits.
              >
              > John
              >
              > At 11:27 +0100 10/3/07, michael watson (IAH-C) wrote:
              >
              > Hi
              >
              > I have "print STDERR..." statements throught my script, with a
              > call to
              > `date` appended so I know when it happens.
              > I also have similar statements before and after disconnecting
              > from MySQL
              > using DBI and undef-ing $self in my perl object.
              >
              > As far as I can tell, the script has completely finished
              > executing, and
              > then there is about a 20 minute wait for the process to end.
              >
              > Mick
              >
              > ________________________________
              >
              > From: perl-beginner@yahoogroups.com
              > <mailto:perl-beginner%40yahoogroups.com>
              > [mailto:perl-beginner@yahoogroups.com
              > <mailto:perl-beginner%40yahoogroups.com> ] On Behalf Of Jenda Krynicky
              > Sent: 03 October 2007 11:12
              > To: perl-beginner@yahoogroups.com
              > <mailto:perl-beginner%40yahoogroups.com>
              > Subject: Re: [PBML] Memory release problem?
              >
              > On 2 Oct 2007 at 9:51, Mick Watson wrote:
              > > I am not a beginner to perl, but am a beginner to this
              > problem!
              > >
              > > I create a perl object, read about 500,000 rows from a MySQL
              > > database, populate some structures and print out a table of
              > results.
              > > I do this using DBI. I create one database handle, which I
              > disconnect
              > > from, and about 6 statement handles, all of which I explicitly
              > finish.
              > >
              > > After the results have printed, the script simply dies.
              > However, the
              > > results finish printing after about 4 minutes from execution;
              > the
              > > process then actually finishes some 20 minutes later - what
              > could be
              > > going on?
              > >
              > > In my object's DESTROY method, I explicitly disconnect from
              > the db
              > > handle (takes less than one second) and explicitly undef $self
              > (takes
              > > less than one second). So I am guessing that the rest of the
              > 20
              > > minutes is perl returning memory to the OS...? Why would it
              > take so
              > > long? The server has 3Gb of RAM, so this should not be a
              > problem.
              >
              > I don't think you are looking at the right place. Though it's
              > hard to
              >
              > suggest where to look.
              >
              > What does it do if you add something like
              >
              > END {print "Exiting the script: 1"}
              >
              > before the use statements, after the use statements and at the
              > end of
              >
              > the script Are these messages printed immedialtely after the
              > normal
              > output of the script or at the end of those 20 minutes of
              > waiting?
              >
              > Jenda
              >
              > ===== Jenda@... <mailto:Jenda%40Krynicky.cz>
              > <mailto:Jenda%40Krynicky.cz> ===
              > http://Jenda.Krynicky.cz <http://Jenda.Krynicky.cz> =====
              > When it comes to wine, women and song, wizards are allowed
              > to get drunk and croon as much as they like.
              > -- Terry Pratchett in Sourcery
              >
              > [Non-text portions of this message have been removed]
              >
              >
              >
              >
              > --
              > John Francini, francini@...
              >
              > "The journey is more important than the destination -- that's part of
              > life.
              > If you only live for getting to the end, you're almost always
              > disappointed."
              > -- Donald Knuth
              >
              > [Non-text portions of this message have been removed]
              >
              >
              >



              [Non-text portions of this message have been removed]
            • merlyn@stonehenge.com
              ... michael Hmmm, OK, I will try this, thank you, but as there are only 17000 rows michael of output, is it really possible perl takes 20 mins to write that
              Message 6 of 9 , Oct 4, 2007
              • 0 Attachment
                >>>>> "michael" == michael watson \(IAH-C\) <michael> writes:

                michael> Hmmm, OK, I will try this, thank you, but as there are only 17000 rows
                michael> of output, is it really possible perl takes 20 mins to write that to a
                michael> file?!

                When you profiled your program using Devel::DProf or Devel::SmallProf,
                where did it show that it was spending its time? It would help to know
                that.

                --
                Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
                <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
                Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
                See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
              Your message has been successfully submitted and would be delivered to recipients shortly.