Loading ...
Sorry, an error occurred while loading the content.
 

Re: [Cheetahtemplate-discuss] cheetah memory leak

Expand Messages
  • Tavis Rudd
    Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(), which keeps a cache of recently used cheetah module names to prevent collisions
    Message 1 of 9 , Feb 3, 2006
      Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(),
      which keeps a cache of recently used cheetah module names to prevent
      collisions before the new modules are added to sys.path. I'd forgotten to
      clear that cache. It now only keeps the last 20 names in the cache.

      I'll look into using a lock to avoid this completely, but this seems to work
      for now.

      Thanks for catching that!

      On Friday 03 February 2006 11:12 am, Tavis Rudd wrote:
      > I'm seeing it as well. Looking into it now. It's something in the
      > compilation process as this doesn't leak:
      >
      > from Cheetah.Template import Template
      >
      > src = open('/tmp/z.py').read()
      > t = Template(src)
      > nr = 0
      > while True:
      > output = t.respond()
      >
      > nr += 1
      > if not nr % 10000:
      > print 'collect'
      > import gc
      > gc.collect()
      >
      >
      > Nor this:
      >
      >
      > from Cheetah.Template import Template
      >
      > src = open('/tmp/z.py').read()
      > tclass = Template.compile(src)
      > nr = 0
      > while True:
      > t = tclass()
      > output = t.respond()
      >
      > nr += 1
      > if not nr % 10000:
      > print 'collect'
      > import gc
      > gc.collect()
      >
      > On Friday 03 February 2006 10:24 am, Andrea Arcangeli wrote:
      > > I noticed a memory leak during benchmarks, and I tracked it down to
      > > Cheetah. Here how to reproduce: save this file to /tmp/z.py and execute
      > > it. Then monitor with 'ps v xxx', RSS will keep growing.
      > >
      > > I called gc.collect() just to be sure (I had problems before without it
      > > in some workload).
      > >
      > > I'm using SVN of a few days ago, if you can't reproduce let me know of
      > > course.
      > >
      > > Thanks.
      > >
      > > ------- cut here
      > > from Cheetah.Template import Template
      > >
      > > nr = 0
      > > while True:
      > > t = Template(file='/tmp/z.py')
      > > t.respond()
      > >
      > > nr += 1
      > > if not nr % 10000:
      > > print 'collect'
      > > import gc
      > > gc.collect()
      > >
      > >
      > > -------------------------------------------------------
      > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log
      > > files for problems? Stop! Download the new AJAX search engine that
      > > makes searching your log files as easy as surfing the web. DOWNLOAD
      > > SPLUNK!
      > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
      > > _______________________________________________
      > > Cheetahtemplate-discuss mailing list
      > > Cheetahtemplate-discuss@...
      > > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
      >
      > -------------------------------------------------------
      > This SF.net email is sponsored by: Splunk Inc. Do you grep through log
      > files for problems? Stop! Download the new AJAX search engine that makes
      > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
      > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
      > _______________________________________________
      > Cheetahtemplate-discuss mailing list
      > Cheetahtemplate-discuss@...
      > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


      -------------------------------------------------------
      This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
      for problems? Stop! Download the new AJAX search engine that makes
      searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
      http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
      _______________________________________________
      Cheetahtemplate-discuss mailing list
      Cheetahtemplate-discuss@...
      https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
    • Andrea Arcangeli
      ... That was fast! great thanks. However I get this error, it seems you forgot to commit I18n? from Cheetah.Macros.I18n import I18n ImportError: No module
      Message 2 of 9 , Feb 3, 2006
        On Fri, Feb 03, 2006 at 11:43:26AM -0800, Tavis Rudd wrote:
        > Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(),
        > which keeps a cache of recently used cheetah module names to prevent
        > collisions before the new modules are added to sys.path. I'd forgotten to
        > clear that cache. It now only keeps the last 20 names in the cache.
        > I'll look into using a lock to avoid this completely, but this seems to work
        > for now.

        That was fast! great thanks.

        However I get this error, it seems you forgot to commit I18n?

        from Cheetah.Macros.I18n import I18n
        ImportError: No module named I18n

        > Thanks for catching that!

        You're welcome.


        -------------------------------------------------------
        This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
        for problems? Stop! Download the new AJAX search engine that makes
        searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
        http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
        _______________________________________________
        Cheetahtemplate-discuss mailing list
        Cheetahtemplate-discuss@...
        https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
      • Tavis Rudd
        ... Done. Andrea, do you mind taking a quick look at the concurrency locking in Template.compile to see if there are any issues I ve missed? ... This SF.net
        Message 3 of 9 , Feb 3, 2006
          On Friday 03 February 2006 11:43 am, Tavis Rudd wrote:
          > Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(),
          > which keeps a cache of recently used cheetah module names to prevent
          > collisions before the new modules are added to sys.path. I'd forgotten to
          > clear that cache. It now only keeps the last 20 names in the cache.
          >
          > I'll look into using a lock to avoid this completely, but this seems to
          > work for now.
          Done.

          Andrea, do you mind taking a quick look at the concurrency locking in
          Template.compile to see if there are any issues I've missed?


          -------------------------------------------------------
          This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
          for problems? Stop! Download the new AJAX search engine that makes
          searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
          http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
          _______________________________________________
          Cheetahtemplate-discuss mailing list
          Cheetahtemplate-discuss@...
          https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
        • Tavis Rudd
          ... Indeed I did! It s there now. I m going to release 2.0rc5 this afternoon with these changes. ... This SF.net email is sponsored by: Splunk Inc. Do you
          Message 4 of 9 , Feb 3, 2006
            On Friday 03 February 2006 12:10 pm, Andrea Arcangeli wrote:
            > That was fast! great thanks.
            >
            > However I get this error, it seems you forgot to commit I18n?
            >
            > from Cheetah.Macros.I18n import I18n
            > ImportError: No module named I18n

            Indeed I did! It's there now.

            I'm going to release 2.0rc5 this afternoon with these changes.


            -------------------------------------------------------
            This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
            for problems? Stop! Download the new AJAX search engine that makes
            searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
            http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
            _______________________________________________
            Cheetahtemplate-discuss mailing list
            Cheetahtemplate-discuss@...
            https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
          • Andrea Arcangeli
            ... Ok I ll have a look as soon as fs syncs anon cvs ;) thanks! ... This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems?
            Message 5 of 9 , Feb 3, 2006
              On Fri, Feb 03, 2006 at 12:16:07PM -0800, Tavis Rudd wrote:
              > Andrea, do you mind taking a quick look at the concurrency locking in
              > Template.compile to see if there are any issues I've missed?

              Ok I'll have a look as soon as fs syncs anon cvs ;)

              thanks!


              -------------------------------------------------------
              This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
              for problems? Stop! Download the new AJAX search engine that makes
              searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
              http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
              _______________________________________________
              Cheetahtemplate-discuss mailing list
              Cheetahtemplate-discuss@...
              https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
            • Andrea Arcangeli
              ... btw, may I ask you if you ve some utility to track where the memory utilization goes in python, or did you fix it by reading the source? thanks. ... This
              Message 6 of 9 , Feb 3, 2006
                On Fri, Feb 03, 2006 at 11:43:26AM -0800, Tavis Rudd wrote:
                > Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(),

                btw, may I ask you if you've some utility to track where the memory
                utilization goes in python, or did you fix it by reading the source?

                thanks.


                -------------------------------------------------------
                This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
                for problems? Stop! Download the new AJAX search engine that makes
                searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
                http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
                _______________________________________________
                Cheetahtemplate-discuss mailing list
                Cheetahtemplate-discuss@...
                https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
              • Tavis Rudd
                I just read the source and experimented with likely culprits. ... This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems?
                Message 7 of 9 , Feb 3, 2006
                  I just read the source and experimented with likely culprits.

                  On Friday 03 February 2006 2:03 pm, Andrea Arcangeli wrote:
                  > On Fri, Feb 03, 2006 at 11:43:26AM -0800, Tavis Rudd wrote:
                  > > Fixed in the cvs. It was due to Cheetah.Template:_genUniqueModuleName(),
                  >
                  > btw, may I ask you if you've some utility to track where the memory
                  > utilization goes in python, or did you fix it by reading the source?
                  >
                  > thanks.
                  >
                  >
                  > -------------------------------------------------------
                  > This SF.net email is sponsored by: Splunk Inc. Do you grep through log
                  > files for problems? Stop! Download the new AJAX search engine that makes
                  > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
                  > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
                  > _______________________________________________
                  > Cheetahtemplate-discuss mailing list
                  > Cheetahtemplate-discuss@...
                  > https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss


                  -------------------------------------------------------
                  This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
                  for problems? Stop! Download the new AJAX search engine that makes
                  searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
                  http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
                  _______________________________________________
                  Cheetahtemplate-discuss mailing list
                  Cheetahtemplate-discuss@...
                  https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss
                Your message has been successfully submitted and would be delivered to recipients shortly.