Loading ...
Sorry, an error occurred while loading the content.

Re: [pcgen_developers] memory usage

Expand Messages
  • Chris Dolan
    Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the
    Message 1 of 22 , Jan 28, 2013
    • 0 Attachment
      Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

      Chris



      On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




      As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

      At load time, it will probably be faster, since it will be doing a lot less memory allocation.

      TP. 
      --
      Tom Parker


      From: Chris Dolan <chris@...>
      To: pcgen_developers@yahoogroups.com 
      Sent: Monday, January 28, 2013 5:04 PM
      Subject: Re: [pcgen_developers] memory usage

      Aha, awesome. I'll try that change this week and benchmark how much it
      helps memory vs. hurts speed. Thanks!
      Chris




    • Tom Parker
      Since the map creation should all happen at LST load, the answer *should* be no. That being said, I make no guarantees about UI behavior.  I was not involved
      Message 2 of 22 , Jan 28, 2013
      • 0 Attachment
        Since the map creation should all happen at LST load, the answer *should* be no.

        That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

        TP.
        --
        Tom Parker


        From: Chris Dolan <chris@...>
        To: pcgen_developers@yahoogroups.com
        Sent: Monday, January 28, 2013 7:32 PM
        Subject: Re: [pcgen_developers] memory usage



        Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

        Chris



        On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




        As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

        At load time, it will probably be faster, since it will be doing a lot less memory allocation.

        TP. 
        --
        Tom Parker


        From: Chris Dolan <chris@...>
        To: pcgen_developers@yahoogroups.com 
        Sent: Monday, January 28, 2013 5:04 PM
        Subject: Re: [pcgen_developers] memory usage

        Aha, awesome. I'll try that change this week and benchmark how much it
        helps memory vs. hurts speed. Thanks!
        Chris








      • James Dempsey
        Hi, After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output
        Message 3 of 22 , Jan 28, 2013
        • 0 Attachment
          Hi,

          After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output threads which keep the summary sheet, skills list and character sheet up to date.

          As Tom mentioned, these should not be updating the LST objects much though. Some cloning still remains though, which is where a small number of updates will occur. Tom is gradually eliminating those though.

          Cheers,
          James

          On 29 January 2013 11:41, Tom Parker <thpr@...> wrote:


          Since the map creation should all happen at LST load, the answer *should* be no.

          That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

          TP.
          --
          Tom Parker

          Sent: Monday, January 28, 2013 7:32 PM

          Subject: Re: [pcgen_developers] memory usage



          Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

          Chris



          On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




          As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

          At load time, it will probably be faster, since it will be doing a lot less memory allocation.

          TP. 
          --
          Tom Parker


          From: Chris Dolan <chris@...>
          To: pcgen_developers@yahoogroups.com 
          Sent: Monday, January 28, 2013 5:04 PM
          Subject: Re: [pcgen_developers] memory usage

          Aha, awesome. I'll try that change this week and benchmark how much it
          helps memory vs. hurts speed. Thanks!
          Chris



        • FerretDave
          Greetings, This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ? Cheers D
          Message 4 of 22 , Jan 30, 2013
          • 0 Attachment
            Greetings,

            This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

            Cheers
            D

            --- In pcgen_developers@yahoogroups.com, Tom Parker wrote:
            >
            >
            >
            > As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.
            >
            >
            > At load time, it will probably be faster, since it will be doing a lot less memory allocation.
            >
            >
            > TP.
            >
            > --
            > Tom Parker
            >
            >
            >
            > ________________________________
            > From: Chris Dolan
            > To: pcgen_developers@yahoogroups.com
            > Sent: Monday, January 28, 2013 5:04 PM
            > Subject: Re: [pcgen_developers] memory usage
            >
            > Aha, awesome. I'll try that change this week and benchmark how much it
            > helps memory vs. hurts speed. Thanks!
            > Chris
            >
          • Tom Parker
            Absolutely.  Like everything else it s been a matter of picking what I address first.   TP. -- Tom Parker ________________________________ From: FerretDave
            Message 5 of 22 , Jan 30, 2013
            • 0 Attachment
              Absolutely.  Like everything else it's been a matter of picking what I address first.
               
              TP.
              --
              Tom Parker


              From: FerretDave <ferret.griffin@...>
              To: pcgen_developers@yahoogroups.com
              Sent: Wednesday, January 30, 2013 4:44 AM
              Subject: [pcgen_developers] Re: memory usage

              Greetings,

              This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

              Cheers
              D


            • Henk Slaaf
              Hey all, It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have. Then we could
              Message 6 of 22 , Jan 30, 2013
              • 0 Attachment

                Hey all,

                It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.

                Then we could graph the results.

                No idea if this is actually feasible, but just dreaming :-)

                Best,

                Henk

                On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                Chris

                P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
              • Tom Parker
                also... Number of java warnings Processing time per character regression test Load errors/warnings for key dataset combinations Unit test coverage ...probably
                Message 7 of 22 , Jan 30, 2013
                • 0 Attachment
                  also...

                  Number of java warnings
                  Processing time per character regression test
                  Load errors/warnings for key dataset combinations
                  Unit test coverage

                  ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                  TP.
                  --
                  Tom Parker


                  From: Henk Slaaf <henk@...>
                  To: pcgen_developers@yahoogroups.com
                  Sent: Wednesday, January 30, 2013 12:58 PM
                  Subject: Re: [pcgen_developers] memory usage



                  Hey all,
                  It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                  Then we could graph the results.
                  No idea if this is actually feasible, but just dreaming :-)
                  Best,
                  Henk
                  On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                  As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                  My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                  Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                  I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                  BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                  AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                  Chris

                  P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                  http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                • Chris Dolan
                  For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up. Chris ... For most of those genetic
                  Message 8 of 22 , Jan 30, 2013
                  • 0 Attachment
                    For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up.
                    Chris

                    Tom Parker <thpr@...> wrote:
                    also...

                    Number of java warnings
                    Processing time per character regression test
                    Load errors/warnings for key dataset combinations
                    Unit test coverage

                    ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                    TP.
                    --
                    Tom Parker


                    From: Henk Slaaf <henk@...>
                    To: pcgen_developers@yahoogroups.com
                    Sent: Wednesday, January 30, 2013 12:58 PM
                    Subject: Re: [pcgen_developers] memory usage



                    Hey all,
                    It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                    Then we could graph the results.
                    No idea if this is actually feasible, but just dreaming :-)
                    Best,
                    Henk
                    On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                    As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                    My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                    Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                    I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                    BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                    AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                    Chris

                    P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                    http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                  Your message has been successfully submitted and would be delivered to recipients shortly.