Loading ...
Sorry, an error occurred while loading the content.

Re: [pcgen_developers] memory usage

Expand Messages
  • Martijn Verburg
    Let me know when you put this on a branch I/we can access. FYI - I m the CTO of a new start-up that s building Java perf tools, so there s some stuff I can
    Message 1 of 22 , Jan 28, 2013
    • 0 Attachment
      Let me know when you put this on a branch I/we can access.  FYI - I'm the CTO of a new start-up that's building Java perf tools, so there's some stuff I can quickly look at internally :-).


      On 28 January 2013 04:37, Chris Dolan <chris@...> wrote:
       

      A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

      Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

      As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

      Chris


      On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

      As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

      My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

      Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

      I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

      BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
      <chart4-before.png>
      AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

      <chart4-after.png>

      Chris

      P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:


    • Henk Slaaf
      Hey Chris! ... You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into
      Message 2 of 22 , Jan 28, 2013
      • 0 Attachment

        Hey Chris!


        On Mon, Jan 28, 2013 at 5:37 AM, Chris Dolan <chris@...> wrote: 

        A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

        Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.


        You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into master much easier. It also splits ideas, so they become more easily reviewed.

         
        As a si de note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

        Cool!

        If you need some Git -> SVN magic to happen, let me know. I'd be happy to pull some stuff for you and put it to SVN. 
        That is, if the code monkeys accept the patches. I myself am not qualified nor authorized to decide :)

        Best,

        Henk
      • Chris Dolan
        Sure, I ve been continuously adding my code at these locations: https://github.com/chrisdolan/pcgen-svn/tree/interned_strings
        Message 3 of 22 , Jan 28, 2013
        • 0 Attachment
          Sure, I've been continuously adding my code at these locations:
            https://github.com/chrisdolan/pcgen-android

          I've got a few hard-coded paths in the pcgen-android repo so it might not compile immediately, but there's not that much code there so it shouldn't be too hard to understand, especially the "pcgenuitest" eclipse project that I've been using for benchmarking.

          Chris

          On Jan 28, 2013, at 3:07 AM, Martijn Verburg wrote:



          Let me know when you put this on a branch I/we can access.  FYI - I'm the CTO of a new start-up that's building Java perf tools, so there's some stuff I can quickly look at internally :-).


          On 28 January 2013 04:37, Chris Dolan <chris@...> wrote:
           

          A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

          Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

          As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

          Chris


          On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

          As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

          My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

          Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

          I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

          BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
          <chart4-before.png>
          AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

          <chart4-after.png>

          Chris

          P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:






        • Chris Dolan
          ... Thanks for the reminder. Yes, I was planing to do that. ... Thanks. Anything from my chrisdolan/master branch is eligible for pulling. Chris
          Message 4 of 22 , Jan 28, 2013
          • 0 Attachment
            On Jan 28, 2013, at 6:27 AM, Henk Slaaf wrote:

            On Mon, Jan 28, 2013 at 5:37 AM, Chris Dolan <chris@...> wrote: 

            A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

            Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.


            You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into master much easier. It also splits ideas, so they become more easily reviewed.

            Thanks for the reminder. Yes, I was planing to do that.

             
            As a si de note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

            Cool!

            If you need some Git -> SVN magic to happen, let me know. I'd be happy to pull some stuff for you and put it to SVN. 
            That is, if the code monkeys accept the patches. I myself am not qualified nor authorized to decide :)

            Thanks. Anything from my chrisdolan/master branch is eligible for pulling.

            Chris

          • Tom Parker
            Do you have a sense of where the memory usage is (what type of object?)  I haven t put PCGen itno TPTP (Eclipse s Test and Performance Tools Platform) in
            Message 5 of 22 , Jan 28, 2013
            • 0 Attachment
              Do you have a sense of where the memory usage is (what type of object?)  I haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools Platform) in quite some time, so I won't claim my knowledge is current... but if you have a sense of where the issue is, I may be able to highlight pretty quickly the constructs we are using and how they might be improved.

              Thanks.

              TP.
              --
              Tom Parker


              From: Chris Dolan <chris@...>
              To: pcgen_developers@yahoogroups.com
              Sent: Sunday, January 27, 2013 11:37 PM
              Subject: Re: [pcgen_developers] memory usage



              A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

              Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

              As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

              Chris


              On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

              As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

              My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

              Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

              I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

              BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
              <chart4-before.png>
              AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

              <chart4-after.png>

              Chris

              P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
              http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string





            • Chris Dolan
              Yep, I ve been using the Eclipse Memory Analysis Tools (MAT) which analyze the .hprof heap dumps that VisualVM emits. It used to be that char[] was the biggest
              Message 6 of 22 , Jan 28, 2013
              • 0 Attachment
                Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
                the .hprof heap dumps that VisualVM emits. It used to be that char[] was
                the biggest consumer due to leaked substring parents, but I've knocked
                that one out of the top 10. The next biggest are Equipment and Spell,
                which is not at all surprising. However, my gut says that a more
                challenging problem will arise with the amount of permgen consumed by the
                ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

                So far, my simple test of running with "-Xmx64m" has been a super way to
                find out what the JVM is hanging up on. With my fixes I can get past game
                mode and campaign loading, but I run out of RAM when loading required
                sources for my sample PC, let alone loading the PC itself.

                Chris


                On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
                > Do you have a sense of where the memory usage is (what type of object?)  I
                > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
                > Platform) in quite some time, so I won't claim my knowledge is current...
                > but if you have a sense of where the issue is, I may be able to highlight
                > pretty quickly the constructs we are using and how they might be improved.
                >
                > Thanks.
                >
                >
                > TP.
                >
                > --
                > Tom Parker
                >
                >
                >
                > ________________________________
                > From: Chris Dolan <chris@...>
                > To: pcgen_developers@yahoogroups.com
                > Sent: Sunday, January 27, 2013 11:37 PM
                > Subject: Re: [pcgen_developers] memory usage
                >
                >
                >
                >
                >
                >
                > A small progress note: I think I've hit the wall in intern()ing and
                > deduplicating strings. I've gained about 10 MB of heap improvement, but
                > there's only a couple of MB left for possible improvement left via the
                > little tweaks I've been making and I'm still about 30-40 MB higher than I
                > can afford running on Android. So more radical changes are certainly
                > necessary.
                >
                > Right now, I'm still loading way more data than I need. My next experiment
                > will be to add a new GameModeFilter option to the GameModeFileLoader so I
                > can skip loading modes that are not interesting to my PC. I don't
                > currently have an intuition about how much that will save (probably not
                > enough), but it looks like just a couple hours of programming so I'm going
                > to try it. Right now, I don't have a deep enough understanding of the
                > PCGen code to risk massive speculative changes like an intermediate
                > database, but I'm reading that threads with interest.
                >
                > As a side note, I think my work has improved startup time by about 5
                > percent. I didn't expect this result, and don't fully understand it. I
                > theorize that the improvement comes from reduced time spent int he garbage
                > collector, but I'm not sure. 5% is almost certainly not noticeable to the
                > average user, but every little bit helps.
                >
                > Chris
                >
                >
                > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
                >
                > As a performance metric, I wrote a simple desktop app that loads a single
                > (hard-coded) Pathfinder PCGen character and renders it with the
                > CharacterSheetPanel. I measured time to load (about 40 seconds) and
                > resulting memory usage (about 85 MB of heap when idle).
                >>
                >>
                >>My test app is at the URL below (less than 200 lines of
                >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
                >>
                >>
                >>Then I went through the PCGen source looking at all of the Lst loading
                >> paths and I added lots of .intern() clauses to the tokenined input to
                >> detach them from the original file strings. This saved a notable amount
                >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
                >> than I hoped. My intern changes are visible on this git branch commit:
                >> 
                >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
                >>
                >>
                >>I've attached a pretty chart of the heap dominators before and after my
                >> change, according the Eclipse MAT plugin. I believe that the heap
                >> associated with the AppClassLoader is the static fields, like in
                >> SystemCollections, but I'm not sure. It's not surprising that Equipment
                >> is the #1 user.
                >>
                >>
                >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
                >> classes don't show. Maybe I made a mistake?)
                >><chart4-before.png>
                >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
                >> the others are about the same)
                >>
                >>
                >><chart4-after.png>
                >>
                >>
                >>Chris
                >>
                >>
                >>P.S. I had an interesting discussion on StackOverflow about
                >> String.intern(). Take a look:
                >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
                >
                >
                >
                >
              • Tom Parker
                A few thoughts: (1) Make sure the preferences are set to NOT create magical items by default.  I don t recall when this process actually happens (or even many
                Message 7 of 22 , Jan 28, 2013
                • 0 Attachment
                  A few thoughts:

                  (1) Make sure the preferences are set to NOT create magical items by default.  I don't recall when this process actually happens (or even many of the exact details on what it is doing), but there is an ability to create objects, and you want that off.

                  (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy instantiation on the fields.  This should shrink both Spell and Equipment, perhaps materially, since each field not instantiated is saving a HashMap, which is something on the magnitude of 100 bytes if I recall correctly.  Wouldn't surprise me if this saved a few MB across all of the objects in something like the RSRD.  (and it would take perhaps 10 minutes to implement)

                  (3) Evaluate a way to jettison the plugin classes after load.  This is easily hundreds of classes... which are loaded with an alternate classloader.  (This occurs somewhere in gmgen.* - I forget the exact class - PluginLoader?)   Once the data is loaded, can you destroy the classloader and destroy the caches in - I think it's TokenLibrary (or TokenStore or something to that effect - the code is not in front of me at the moment, but I think this is in pcgen.persistence.* somewhere).  That might then free up those classes to be garbage collected.

                  TP.
                  --
                  Tom Parker


                  From: Chris Dolan <chris@...>
                  To: pcgen_developers@yahoogroups.com
                  Sent: Monday, January 28, 2013 10:27 AM
                  Subject: Re: [pcgen_developers] memory usage

                  Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
                  the .hprof heap dumps that VisualVM emits. It used to be that char[] was
                  the biggest consumer due to leaked substring parents, but I've knocked
                  that one out of the top 10. The next biggest are Equipment and Spell,
                  which is not at all surprising. However, my gut says that a more
                  challenging problem will arise with the amount of permgen consumed by the
                  ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

                  So far, my simple test of running with "-Xmx64m" has been a super way to
                  find out what the JVM is hanging up on. With my fixes I can get past game
                  mode and campaign loading, but I run out of RAM when loading required
                  sources for my sample PC, let alone loading the PC itself.

                  Chris


                  On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
                  > Do you have a sense of where the memory usage is (what type of object?)  I
                  > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
                  > Platform) in quite some time, so I won't claim my knowledge is current...
                  > but if you have a sense of where the issue is, I may be able to highlight
                  > pretty quickly the constructs we are using and how they might be improved.
                  >
                  > Thanks.
                  >
                  >
                  > TP.
                  >
                  > --
                  > Tom Parker
                  >
                  >
                  >
                  > ________________________________
                  >  From: Chris Dolan <chris@...>
                  > To: pcgen_developers@yahoogroups.com
                  > Sent: Sunday, January 27, 2013 11:37 PM
                  > Subject: Re: [pcgen_developers] memory usage
                  >
                  >
                  >
                  >
                  >
                  >
                  > A small progress note: I think I've hit the wall in intern()ing and
                  > deduplicating strings. I've gained about 10 MB of heap improvement, but
                  > there's only a couple of MB left for possible improvement left via the
                  > little tweaks I've been making and I'm still about 30-40 MB higher than I
                  > can afford running on Android. So more radical changes are certainly
                  > necessary.
                  >
                  > Right now, I'm still loading way more data than I need. My next experiment
                  > will be to add a new GameModeFilter option to the GameModeFileLoader so I
                  > can skip loading modes that are not interesting to my PC. I don't
                  > currently have an intuition about how much that will save (probably not
                  > enough), but it looks like just a couple hours of programming so I'm going
                  > to try it. Right now, I don't have a deep enough understanding of the
                  > PCGen code to risk massive speculative changes like an intermediate
                  > database, but I'm reading that threads with interest.
                  >
                  > As a side note, I think my work has improved startup time by about 5
                  > percent. I didn't expect this result, and don't fully understand it. I
                  > theorize that the improvement comes from reduced time spent int he garbage
                  > collector, but I'm not sure. 5% is almost certainly not noticeable to the
                  > average user, but every little bit helps.
                  >
                  > Chris
                  >
                  >
                  > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
                  >
                  > As a performance metric, I wrote a simple desktop app that loads a single
                  > (hard-coded) Pathfinder PCGen character and renders it with the
                  > CharacterSheetPanel. I measured time to load (about 40 seconds) and
                  > resulting memory usage (about 85 MB of heap when idle).
                  >>
                  >>
                  >>My test app is at the URL below (less than 200 lines of
                  >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
                  >>
                  >>
                  >>Then I went through the PCGen source looking at all of the Lst loading
                  >> paths and I added lots of .intern() clauses to the tokenined input to
                  >> detach them from the original file strings. This saved a notable amount
                  >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
                  >> than I hoped. My intern changes are visible on this git branch commit:
                  >> 
                  >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
                  >>
                  >>
                  >>I've attached a pretty chart of the heap dominators before and after my
                  >> change, according the Eclipse MAT plugin. I believe that the heap
                  >> associated with the AppClassLoader is the static fields, like in
                  >> SystemCollections, but I'm not sure. It's not surprising that Equipment
                  >> is the #1 user.
                  >>
                  >>
                  >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
                  >> classes don't show. Maybe I made a mistake?)
                  >><chart4-before.png>
                  >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
                  >> the others are about the same)
                  >>
                  >>
                  >><chart4-after.png>
                  >>
                  >>
                  >>Chris
                  >>
                  >>
                  >>P.S. I had an interesting discussion on StackOverflow about
                  >> String.intern(). Take a look:
                  >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
                  >
                  >
                  >
                  >




                  ------------------------------------

                  Yahoo! Groups Links

                  <*> To visit your group on the web, go to:
                      http://groups.yahoo.com/group/pcgen_developers/

                  <*> Your email settings:
                      Individual Email | Traditional

                  <*> To change settings online go to:
                      http://groups.yahoo.com/group/pcgen_developers/join
                      (Yahoo! ID required)

                  <*> To change settings via email:
                      pcgen_developers-digest@yahoogroups.com
                      pcgen_developers-fullfeatured@yahoogroups.com

                  <*> To unsubscribe from this group, send an email to:
                      pcgen_developers-unsubscribe@yahoogroups.com

                  <*> Your use of Yahoo! Groups is subject to:
                      http://docs.yahoo.com/info/terms/



                • Chris Dolan
                  ... I think it s already off by default, but I ll double check. This would indeed be a big savings. ... I understand what you mean at a high level, but not the
                  Message 8 of 22 , Jan 28, 2013
                  • 0 Attachment
                    On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                    > A few thoughts:
                    >
                    > (1) Make sure the preferences are set to NOT create magical items by
                    > default.  I don't recall when this process actually happens (or even many
                    > of the exact details on what it is doing), but there is an ability to
                    > create objects, and you want that off.

                    I think it's already off by default, but I'll double check. This would
                    indeed be a big savings.

                    > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                    > instantiation on the fields.  This should shrink both Spell and Equipment,
                    > perhaps materially, since each field not instantiated is saving a HashMap,
                    > which is something on the magnitude of 100 bytes if I recall correctly. 
                    > Wouldn't surprise me if this saved a few MB across all of the objects in
                    > something like the RSRD.  (and it would take perhaps 10 minutes to
                    > implement)

                    I understand what you mean at a high level, but not the specifics. Are you
                    referring to the SystemCollections global? Maybe I just need to look at
                    the CDOMObject code and your meaning will be clear... (I don't have the
                    code on this machine...)

                    > (3) Evaluate a way to jettison the plugin classes after load.  This is
                    > easily hundreds of classes... which are loaded with an alternate
                    > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
                    > - PluginLoader?)   Once the data is loaded, can you destroy the
                    > classloader and destroy the caches in - I think it's TokenLibrary (or
                    > TokenStore or something to that effect - the code is not in front of me at
                    > the moment, but I think this is in pcgen.persistence.* somewhere).  That
                    > might then free up those classes to be garbage collected.

                    Excellent point. Along the same lines there are classes pinned by the
                    Spring loader's singleton cache. Without having looked at the Spring code,
                    I'll bet they're using a SoftReference or a WeakReference. I'll add
                    something like that in my NonSpringImpl that I built for Android.

                    Chris
                  • Tom Parker
                    On #2: I m referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I m recommending are all localized to that one class.  CDOMObject has a
                    Message 9 of 22 , Jan 28, 2013
                    • 0 Attachment
                      On #2:

                      I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I'm recommending are all localized to that one class. 

                      CDOMObject has a number of fields, something like:

                      private final Map<StringKey, String> stringMap = new HashMap<StringChar, String>();

                      then in methods, it is used as such:

                      public String get(StringKey sk)
                      {
                         return stringMap.get(sk);
                      }

                      public void put(StringKey sk, String s)
                      {
                         stringMap.put(sk, s);
                      }

                      This means the HashMap is instantiated as soon as the object is created ("aggressive instantiation", if you will).  What I recommend is "lazy instantiation", meaning only instantiate a map if the map is truly required (on a put).  So just change to:

                      private Map<StringKey, String> stringMap = null;

                      then in methods:

                      public String get(StringKey sk)
                      {
                         if (stringMap == null)
                         {
                            return null;
                         }
                         return stringMap.get(sk);
                      }


                      public String put(StringKey sk, String s)
                      {
                         if (stringMap == null)
                         {
                            stringMap = new HashMap<StringKey, String>();
                         }
                         stringMap.put(sk, s);
                      }

                      It adds in a whole ton of null checks across the class, making it a tiny bit less straightforward (and somewhere between infinitesimally and a tiny bit slower), but it will use less memory.

                      This is useful because simple objects that only have a key, a type, and a bonus or whatever will only use StringKey and ListKey, and none of the others.  So in any given object, a number of those maps are probably empty, and over lots of objects, that adds up.  If you assume RSRD loads 5K objects, and it saves 3 maps per object and 100 bytes per map, that's 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really never tested it.

                      p.s. My field name and method names are probably off, but you get the idea.

                      TP.
                      --
                      Tom Parker


                      From: Chris Dolan <chris@...>
                      To: pcgen_developers@yahoogroups.com
                      Sent: Monday, January 28, 2013 4:36 PM
                      Subject: Re: [pcgen_developers] memory usage

                      On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                      > A few thoughts:
                      >
                      > (1) Make sure the preferences are set to NOT create magical items by
                      > default.  I don't recall when this process actually happens (or even many
                      > of the exact details on what it is doing), but there is an ability to
                      > create objects, and you want that off.

                      I think it's already off by default, but I'll double check. This would
                      indeed be a big savings.

                      > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                      > instantiation on the fields.  This should shrink both Spell and Equipment,
                      > perhaps materially, since each field not instantiated is saving a HashMap,
                      > which is something on the magnitude of 100 bytes if I recall correctly. 
                      > Wouldn't surprise me if this saved a few MB across all of the objects in
                      > something like the RSRD.  (and it would take perhaps 10 minutes to
                      > implement)

                      I understand what you mean at a high level, but not the specifics. Are you
                      referring to the SystemCollections global? Maybe I just need to look at
                      the CDOMObject code and your meaning will be clear... (I don't have the
                      code on this machine...)

                      > (3) Evaluate a way to jettison the plugin classes after load.  This is
                      > easily hundreds of classes... which are loaded with an alternate
                      > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
                      > - PluginLoader?)   Once the data is loaded, can you destroy the
                      > classloader and destroy the caches in - I think it's TokenLibrary (or
                      > TokenStore or something to that effect - the code is not in front of me at
                      > the moment, but I think this is in pcgen.persistence.* somewhere).  That
                      > might then free up those classes to be garbage collected.

                      Excellent point. Along the same lines there are classes pinned by the
                      Spring loader's singleton cache. Without having looked at the Spring code,
                      I'll bet they're using a SoftReference or a WeakReference. I'll add
                      something like that in my NonSpringImpl that I built for Android.

                      Chris



                      ------------------------------------

                      Yahoo! Groups Links

                      <*> To visit your group on the web, go to:
                          http://groups.yahoo.com/group/pcgen_developers/

                      <*> Your email settings:
                          Individual Email | Traditional

                      <*> To change settings online go to:
                          http://groups.yahoo.com/group/pcgen_developers/join
                          (Yahoo! ID required)

                      <*> To change settings via email:
                          pcgen_developers-digest@yahoogroups.com
                          pcgen_developers-fullfeatured@yahoogroups.com

                      <*> To unsubscribe from this group, send an email to:
                          pcgen_developers-unsubscribe@yahoogroups.com

                      <*> Your use of Yahoo! Groups is subject to:
                          http://docs.yahoo.com/info/terms/



                    • Chris Dolan
                      Aha, awesome. I ll try that change this week and benchmark how much it helps memory vs. hurts speed. Thanks! Chris
                      Message 10 of 22 , Jan 28, 2013
                      • 0 Attachment
                        Aha, awesome. I'll try that change this week and benchmark how much it
                        helps memory vs. hurts speed. Thanks!
                        Chris


                        On Mon, January 28, 2013 3:56 pm, Tom Parker wrote:
                        > On #2:
                        >
                        > I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the
                        > changes I'm recommending are all localized to that one class. 
                        >
                        >
                        > CDOMObject has a number of fields, something like:
                        >
                        > private final Map<StringKey, String> stringMap = new HashMap<StringChar,
                        > String>();
                        >
                        > then in methods, it is used as such:
                        >
                        > public String get(StringKey sk)
                        > {
                        >    return stringMap.get(sk);
                        >
                        > }
                        >
                        > public void put(StringKey sk, String s)
                        > {
                        >    stringMap.put(sk, s);
                        >
                        > }
                        >
                        > This means the HashMap is instantiated as soon as the object is created
                        > ("aggressive instantiation", if you will).  What I recommend is "lazy
                        > instantiation", meaning only instantiate a map if the map is truly
                        > required (on a put).  So just change to:
                        >
                        >
                        > private Map<StringKey, String> stringMap = null;
                        >
                        > then in methods:
                        >
                        > public String get(StringKey sk)
                        > {
                        >    if (stringMap == null)
                        >    {
                        >       return null;
                        >    }
                        >
                        >    return stringMap.get(sk);
                        >
                        > }
                        >
                        >
                        > public String put(StringKey sk, String s)
                        > {
                        >    if (stringMap == null)
                        >    {
                        >       stringMap = new HashMap<StringKey, String>();
                        >    }
                        >
                        >    stringMap.put(sk, s);
                        >
                        > }
                        >
                        > It adds in a whole ton of null checks across the class, making it a tiny
                        > bit less straightforward (and somewhere between infinitesimally and a tiny
                        > bit slower), but it will use less memory.
                        >
                        >
                        > This is useful because simple objects that only have a key, a type, and a
                        > bonus or whatever will only use StringKey and ListKey, and none of the
                        > others.  So in any given object, a number of those maps are probably
                        > empty, and over lots of objects, that adds up.  If you assume RSRD loads
                        > 5K objects, and it saves 3 maps per object and 100 bytes per map, that's
                        > 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really
                        > never tested it.
                        >
                        >
                        > p.s. My field name and method names are probably off, but you get the
                        > idea.
                        >
                        > TP.
                        >
                        > --
                        > Tom Parker
                        >
                        >
                        >
                        > ________________________________
                        > From: Chris Dolan <chris@...>
                        > To: pcgen_developers@yahoogroups.com
                        > Sent: Monday, January 28, 2013 4:36 PM
                        > Subject: Re: [pcgen_developers] memory usage
                        >
                        > On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                        >> A few thoughts:
                        >>
                        >> (1) Make sure the preferences are set to NOT create magical items by
                        >> default.  I don't recall when this process actually happens (or even
                        >> many
                        >> of the exact details on what it is doing), but there is an ability to
                        >> create objects, and you want that off.
                        >
                        > I think it's already off by default, but I'll double check. This would
                        > indeed be a big savings.
                        >
                        >> (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                        >> instantiation on the fields.  This should shrink both Spell and
                        >> Equipment,
                        >> perhaps materially, since each field not instantiated is saving a
                        >> HashMap,
                        >> which is something on the magnitude of 100 bytes if I recall correctly. 
                        >> Wouldn't surprise me if this saved a few MB across all of the objects in
                        >> something like the RSRD.  (and it would take perhaps 10 minutes to
                        >> implement)
                        >
                        > I understand what you mean at a high level, but not the specifics. Are you
                        > referring to the SystemCollections global? Maybe I just need to look at
                        > the CDOMObject code and your meaning will be clear... (I don't have the
                        > code on this machine...)
                        >
                        >> (3) Evaluate a way to jettison the plugin classes after load.  This is
                        >> easily hundreds of classes... which are loaded with an alternate
                        >> classloader.  (This occurs somewhere in gmgen.* - I forget the exact
                        >> class
                        >> - PluginLoader?)   Once the data is loaded, can you destroy the
                        >> classloader and destroy the caches in - I think it's TokenLibrary (or
                        >> TokenStore or something to that effect - the code is not in front of me
                        >> at
                        >> the moment, but I think this is in pcgen.persistence.* somewhere).  That
                        >> might then free up those classes to be garbage collected.
                        >
                        > Excellent point. Along the same lines there are classes pinned by the
                        > Spring loader's singleton cache. Without having looked at the Spring code,
                        > I'll bet they're using a SoftReference or a WeakReference. I'll add
                        > something like that in my NonSpringImpl that I built for Android.
                        >
                        > Chris
                        >
                        >
                        >
                        > ------------------------------------
                        >
                        > Yahoo! Groups Links
                        >
                        >
                        >
                        >     http://docs.yahoo.com/info/terms/
                      • Tom Parker
                        As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many
                        Message 11 of 22 , Jan 28, 2013
                        • 0 Attachment

                          As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                          At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                          TP.
                          --
                          Tom Parker


                          From: Chris Dolan <chris@...>
                          To: pcgen_developers@yahoogroups.com
                          Sent: Monday, January 28, 2013 5:04 PM
                          Subject: Re: [pcgen_developers] memory usage

                          Aha, awesome. I'll try that change this week and benchmark how much it
                          helps memory vs. hurts speed. Thanks!
                          Chris

                        • Chris Dolan
                          Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the
                          Message 12 of 22 , Jan 28, 2013
                          • 0 Attachment
                            Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                            Chris



                            On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                            As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                            At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                            TP. 
                            --
                            Tom Parker


                            From: Chris Dolan <chris@...>
                            To: pcgen_developers@yahoogroups.com 
                            Sent: Monday, January 28, 2013 5:04 PM
                            Subject: Re: [pcgen_developers] memory usage

                            Aha, awesome. I'll try that change this week and benchmark how much it
                            helps memory vs. hurts speed. Thanks!
                            Chris




                          • Tom Parker
                            Since the map creation should all happen at LST load, the answer *should* be no. That being said, I make no guarantees about UI behavior.  I was not involved
                            Message 13 of 22 , Jan 28, 2013
                            • 0 Attachment
                              Since the map creation should all happen at LST load, the answer *should* be no.

                              That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                              TP.
                              --
                              Tom Parker


                              From: Chris Dolan <chris@...>
                              To: pcgen_developers@yahoogroups.com
                              Sent: Monday, January 28, 2013 7:32 PM
                              Subject: Re: [pcgen_developers] memory usage



                              Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                              Chris



                              On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                              As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                              At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                              TP. 
                              --
                              Tom Parker


                              From: Chris Dolan <chris@...>
                              To: pcgen_developers@yahoogroups.com 
                              Sent: Monday, January 28, 2013 5:04 PM
                              Subject: Re: [pcgen_developers] memory usage

                              Aha, awesome. I'll try that change this week and benchmark how much it
                              helps memory vs. hurts speed. Thanks!
                              Chris








                            • James Dempsey
                              Hi, After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output
                              Message 14 of 22 , Jan 28, 2013
                              • 0 Attachment
                                Hi,

                                After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output threads which keep the summary sheet, skills list and character sheet up to date.

                                As Tom mentioned, these should not be updating the LST objects much though. Some cloning still remains though, which is where a small number of updates will occur. Tom is gradually eliminating those though.

                                Cheers,
                                James

                                On 29 January 2013 11:41, Tom Parker <thpr@...> wrote:


                                Since the map creation should all happen at LST load, the answer *should* be no.

                                That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                                TP.
                                --
                                Tom Parker

                                Sent: Monday, January 28, 2013 7:32 PM

                                Subject: Re: [pcgen_developers] memory usage



                                Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                                Chris



                                On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                                As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                                At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                                TP. 
                                --
                                Tom Parker


                                From: Chris Dolan <chris@...>
                                To: pcgen_developers@yahoogroups.com 
                                Sent: Monday, January 28, 2013 5:04 PM
                                Subject: Re: [pcgen_developers] memory usage

                                Aha, awesome. I'll try that change this week and benchmark how much it
                                helps memory vs. hurts speed. Thanks!
                                Chris



                              • FerretDave
                                Greetings, This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ? Cheers D
                                Message 15 of 22 , Jan 30, 2013
                                • 0 Attachment
                                  Greetings,

                                  This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                                  Cheers
                                  D

                                  --- In pcgen_developers@yahoogroups.com, Tom Parker wrote:
                                  >
                                  >
                                  >
                                  > As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.
                                  >
                                  >
                                  > At load time, it will probably be faster, since it will be doing a lot less memory allocation.
                                  >
                                  >
                                  > TP.
                                  >
                                  > --
                                  > Tom Parker
                                  >
                                  >
                                  >
                                  > ________________________________
                                  > From: Chris Dolan
                                  > To: pcgen_developers@yahoogroups.com
                                  > Sent: Monday, January 28, 2013 5:04 PM
                                  > Subject: Re: [pcgen_developers] memory usage
                                  >
                                  > Aha, awesome. I'll try that change this week and benchmark how much it
                                  > helps memory vs. hurts speed. Thanks!
                                  > Chris
                                  >
                                • Tom Parker
                                  Absolutely.  Like everything else it s been a matter of picking what I address first.   TP. -- Tom Parker ________________________________ From: FerretDave
                                  Message 16 of 22 , Jan 30, 2013
                                  • 0 Attachment
                                    Absolutely.  Like everything else it's been a matter of picking what I address first.
                                     
                                    TP.
                                    --
                                    Tom Parker


                                    From: FerretDave <ferret.griffin@...>
                                    To: pcgen_developers@yahoogroups.com
                                    Sent: Wednesday, January 30, 2013 4:44 AM
                                    Subject: [pcgen_developers] Re: memory usage

                                    Greetings,

                                    This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                                    Cheers
                                    D


                                  • Henk Slaaf
                                    Hey all, It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have. Then we could
                                    Message 17 of 22 , Jan 30, 2013
                                    • 0 Attachment

                                      Hey all,

                                      It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.

                                      Then we could graph the results.

                                      No idea if this is actually feasible, but just dreaming :-)

                                      Best,

                                      Henk

                                      On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                      As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                      My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                      Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                      I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                      BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                      AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                      Chris

                                      P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                    • Tom Parker
                                      also... Number of java warnings Processing time per character regression test Load errors/warnings for key dataset combinations Unit test coverage ...probably
                                      Message 18 of 22 , Jan 30, 2013
                                      • 0 Attachment
                                        also...

                                        Number of java warnings
                                        Processing time per character regression test
                                        Load errors/warnings for key dataset combinations
                                        Unit test coverage

                                        ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                        TP.
                                        --
                                        Tom Parker


                                        From: Henk Slaaf <henk@...>
                                        To: pcgen_developers@yahoogroups.com
                                        Sent: Wednesday, January 30, 2013 12:58 PM
                                        Subject: Re: [pcgen_developers] memory usage



                                        Hey all,
                                        It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                        Then we could graph the results.
                                        No idea if this is actually feasible, but just dreaming :-)
                                        Best,
                                        Henk
                                        On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                        As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                        My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                        Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                        I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                        BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                        AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                        Chris

                                        P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                        http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                                      • Chris Dolan
                                        For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up. Chris ... For most of those genetic
                                        Message 19 of 22 , Jan 30, 2013
                                        • 0 Attachment
                                          For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up.
                                          Chris

                                          Tom Parker <thpr@...> wrote:
                                          also...

                                          Number of java warnings
                                          Processing time per character regression test
                                          Load errors/warnings for key dataset combinations
                                          Unit test coverage

                                          ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                          TP.
                                          --
                                          Tom Parker


                                          From: Henk Slaaf <henk@...>
                                          To: pcgen_developers@yahoogroups.com
                                          Sent: Wednesday, January 30, 2013 12:58 PM
                                          Subject: Re: [pcgen_developers] memory usage



                                          Hey all,
                                          It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                          Then we could graph the results.
                                          No idea if this is actually feasible, but just dreaming :-)
                                          Best,
                                          Henk
                                          On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                          As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                          My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                          Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                          I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                          BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                          AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                          Chris

                                          P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                          http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                                        Your message has been successfully submitted and would be delivered to recipients shortly.