Loading ...
Sorry, an error occurred while loading the content.

memory usage

Expand Messages
  • Chris Dolan
    As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel.
    Message 1 of 22 , Jan 25, 2013
    • 0 Attachment
      As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

      My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

      Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

      I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

      BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
      AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


      Chris

      P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
    • Henk Slaaf
      Hey Chris! Keep up the good work. It would be great if we could cut memory usage some more. I really appreciate how you persue something you are obviously
      Message 2 of 22 , Jan 25, 2013
      • 0 Attachment

        Hey Chris!

        Keep up the good work. It would be great if we could cut memory usage some more.


        I really appreciate how you persue something you are obviously passionate about. Go Chris!

        Best,

        Henk
      • Chris Dolan
        A small progress note: I think I ve hit the wall in intern()ing and deduplicating strings. I ve gained about 10 MB of heap improvement, but there s only a
        Message 3 of 22 , Jan 27, 2013
        • 0 Attachment
          A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

          Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

          As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

          Chris


          On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

          As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

          My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

          Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

          I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

          BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
          <chart4-before.png>
          AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

          <chart4-after.png>

          Chris

          P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:

        • Martijn Verburg
          Let me know when you put this on a branch I/we can access. FYI - I m the CTO of a new start-up that s building Java perf tools, so there s some stuff I can
          Message 4 of 22 , Jan 28, 2013
          • 0 Attachment
            Let me know when you put this on a branch I/we can access.  FYI - I'm the CTO of a new start-up that's building Java perf tools, so there's some stuff I can quickly look at internally :-).


            On 28 January 2013 04:37, Chris Dolan <chris@...> wrote:
             

            A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

            Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

            As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

            Chris


            On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

            As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

            My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

            Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

            I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

            BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
            <chart4-before.png>
            AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

            <chart4-after.png>

            Chris

            P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:


          • Henk Slaaf
            Hey Chris! ... You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into
            Message 5 of 22 , Jan 28, 2013
            • 0 Attachment

              Hey Chris!


              On Mon, Jan 28, 2013 at 5:37 AM, Chris Dolan <chris@...> wrote: 

              A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

              Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.


              You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into master much easier. It also splits ideas, so they become more easily reviewed.

               
              As a si de note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

              Cool!

              If you need some Git -> SVN magic to happen, let me know. I'd be happy to pull some stuff for you and put it to SVN. 
              That is, if the code monkeys accept the patches. I myself am not qualified nor authorized to decide :)

              Best,

              Henk
            • Chris Dolan
              Sure, I ve been continuously adding my code at these locations: https://github.com/chrisdolan/pcgen-svn/tree/interned_strings
              Message 6 of 22 , Jan 28, 2013
              • 0 Attachment
                Sure, I've been continuously adding my code at these locations:
                  https://github.com/chrisdolan/pcgen-android

                I've got a few hard-coded paths in the pcgen-android repo so it might not compile immediately, but there's not that much code there so it shouldn't be too hard to understand, especially the "pcgenuitest" eclipse project that I've been using for benchmarking.

                Chris

                On Jan 28, 2013, at 3:07 AM, Martijn Verburg wrote:



                Let me know when you put this on a branch I/we can access.  FYI - I'm the CTO of a new start-up that's building Java perf tools, so there's some stuff I can quickly look at internally :-).


                On 28 January 2013 04:37, Chris Dolan <chris@...> wrote:
                 

                A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

                Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

                As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

                Chris


                On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

                As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                <chart4-before.png>
                AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

                <chart4-after.png>

                Chris

                P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:






              • Chris Dolan
                ... Thanks for the reminder. Yes, I was planing to do that. ... Thanks. Anything from my chrisdolan/master branch is eligible for pulling. Chris
                Message 7 of 22 , Jan 28, 2013
                • 0 Attachment
                  On Jan 28, 2013, at 6:27 AM, Henk Slaaf wrote:

                  On Mon, Jan 28, 2013 at 5:37 AM, Chris Dolan <chris@...> wrote: 

                  A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

                  Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.


                  You might want to put that code in a different branch from the memory fixes. Branching is free, but makes switching tasks and merging them into master much easier. It also splits ideas, so they become more easily reviewed.

                  Thanks for the reminder. Yes, I was planing to do that.

                   
                  As a si de note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

                  Cool!

                  If you need some Git -> SVN magic to happen, let me know. I'd be happy to pull some stuff for you and put it to SVN. 
                  That is, if the code monkeys accept the patches. I myself am not qualified nor authorized to decide :)

                  Thanks. Anything from my chrisdolan/master branch is eligible for pulling.

                  Chris

                • Tom Parker
                  Do you have a sense of where the memory usage is (what type of object?)  I haven t put PCGen itno TPTP (Eclipse s Test and Performance Tools Platform) in
                  Message 8 of 22 , Jan 28, 2013
                  • 0 Attachment
                    Do you have a sense of where the memory usage is (what type of object?)  I haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools Platform) in quite some time, so I won't claim my knowledge is current... but if you have a sense of where the issue is, I may be able to highlight pretty quickly the constructs we are using and how they might be improved.

                    Thanks.

                    TP.
                    --
                    Tom Parker


                    From: Chris Dolan <chris@...>
                    To: pcgen_developers@yahoogroups.com
                    Sent: Sunday, January 27, 2013 11:37 PM
                    Subject: Re: [pcgen_developers] memory usage



                    A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

                    Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

                    As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

                    Chris


                    On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

                    As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                    My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                    Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                    I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                    BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                    <chart4-before.png>
                    AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

                    <chart4-after.png>

                    Chris

                    P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                    http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string





                  • Chris Dolan
                    Yep, I ve been using the Eclipse Memory Analysis Tools (MAT) which analyze the .hprof heap dumps that VisualVM emits. It used to be that char[] was the biggest
                    Message 9 of 22 , Jan 28, 2013
                    • 0 Attachment
                      Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
                      the .hprof heap dumps that VisualVM emits. It used to be that char[] was
                      the biggest consumer due to leaked substring parents, but I've knocked
                      that one out of the top 10. The next biggest are Equipment and Spell,
                      which is not at all surprising. However, my gut says that a more
                      challenging problem will arise with the amount of permgen consumed by the
                      ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

                      So far, my simple test of running with "-Xmx64m" has been a super way to
                      find out what the JVM is hanging up on. With my fixes I can get past game
                      mode and campaign loading, but I run out of RAM when loading required
                      sources for my sample PC, let alone loading the PC itself.

                      Chris


                      On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
                      > Do you have a sense of where the memory usage is (what type of object?)  I
                      > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
                      > Platform) in quite some time, so I won't claim my knowledge is current...
                      > but if you have a sense of where the issue is, I may be able to highlight
                      > pretty quickly the constructs we are using and how they might be improved.
                      >
                      > Thanks.
                      >
                      >
                      > TP.
                      >
                      > --
                      > Tom Parker
                      >
                      >
                      >
                      > ________________________________
                      > From: Chris Dolan <chris@...>
                      > To: pcgen_developers@yahoogroups.com
                      > Sent: Sunday, January 27, 2013 11:37 PM
                      > Subject: Re: [pcgen_developers] memory usage
                      >
                      >
                      >
                      >
                      >
                      >
                      > A small progress note: I think I've hit the wall in intern()ing and
                      > deduplicating strings. I've gained about 10 MB of heap improvement, but
                      > there's only a couple of MB left for possible improvement left via the
                      > little tweaks I've been making and I'm still about 30-40 MB higher than I
                      > can afford running on Android. So more radical changes are certainly
                      > necessary.
                      >
                      > Right now, I'm still loading way more data than I need. My next experiment
                      > will be to add a new GameModeFilter option to the GameModeFileLoader so I
                      > can skip loading modes that are not interesting to my PC. I don't
                      > currently have an intuition about how much that will save (probably not
                      > enough), but it looks like just a couple hours of programming so I'm going
                      > to try it. Right now, I don't have a deep enough understanding of the
                      > PCGen code to risk massive speculative changes like an intermediate
                      > database, but I'm reading that threads with interest.
                      >
                      > As a side note, I think my work has improved startup time by about 5
                      > percent. I didn't expect this result, and don't fully understand it. I
                      > theorize that the improvement comes from reduced time spent int he garbage
                      > collector, but I'm not sure. 5% is almost certainly not noticeable to the
                      > average user, but every little bit helps.
                      >
                      > Chris
                      >
                      >
                      > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
                      >
                      > As a performance metric, I wrote a simple desktop app that loads a single
                      > (hard-coded) Pathfinder PCGen character and renders it with the
                      > CharacterSheetPanel. I measured time to load (about 40 seconds) and
                      > resulting memory usage (about 85 MB of heap when idle).
                      >>
                      >>
                      >>My test app is at the URL below (less than 200 lines of
                      >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
                      >>
                      >>
                      >>Then I went through the PCGen source looking at all of the Lst loading
                      >> paths and I added lots of .intern() clauses to the tokenined input to
                      >> detach them from the original file strings. This saved a notable amount
                      >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
                      >> than I hoped. My intern changes are visible on this git branch commit:
                      >> 
                      >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
                      >>
                      >>
                      >>I've attached a pretty chart of the heap dominators before and after my
                      >> change, according the Eclipse MAT plugin. I believe that the heap
                      >> associated with the AppClassLoader is the static fields, like in
                      >> SystemCollections, but I'm not sure. It's not surprising that Equipment
                      >> is the #1 user.
                      >>
                      >>
                      >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
                      >> classes don't show. Maybe I made a mistake?)
                      >><chart4-before.png>
                      >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
                      >> the others are about the same)
                      >>
                      >>
                      >><chart4-after.png>
                      >>
                      >>
                      >>Chris
                      >>
                      >>
                      >>P.S. I had an interesting discussion on StackOverflow about
                      >> String.intern(). Take a look:
                      >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
                      >
                      >
                      >
                      >
                    • Tom Parker
                      A few thoughts: (1) Make sure the preferences are set to NOT create magical items by default.  I don t recall when this process actually happens (or even many
                      Message 10 of 22 , Jan 28, 2013
                      • 0 Attachment
                        A few thoughts:

                        (1) Make sure the preferences are set to NOT create magical items by default.  I don't recall when this process actually happens (or even many of the exact details on what it is doing), but there is an ability to create objects, and you want that off.

                        (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy instantiation on the fields.  This should shrink both Spell and Equipment, perhaps materially, since each field not instantiated is saving a HashMap, which is something on the magnitude of 100 bytes if I recall correctly.  Wouldn't surprise me if this saved a few MB across all of the objects in something like the RSRD.  (and it would take perhaps 10 minutes to implement)

                        (3) Evaluate a way to jettison the plugin classes after load.  This is easily hundreds of classes... which are loaded with an alternate classloader.  (This occurs somewhere in gmgen.* - I forget the exact class - PluginLoader?)   Once the data is loaded, can you destroy the classloader and destroy the caches in - I think it's TokenLibrary (or TokenStore or something to that effect - the code is not in front of me at the moment, but I think this is in pcgen.persistence.* somewhere).  That might then free up those classes to be garbage collected.

                        TP.
                        --
                        Tom Parker


                        From: Chris Dolan <chris@...>
                        To: pcgen_developers@yahoogroups.com
                        Sent: Monday, January 28, 2013 10:27 AM
                        Subject: Re: [pcgen_developers] memory usage

                        Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
                        the .hprof heap dumps that VisualVM emits. It used to be that char[] was
                        the biggest consumer due to leaked substring parents, but I've knocked
                        that one out of the top 10. The next biggest are Equipment and Spell,
                        which is not at all surprising. However, my gut says that a more
                        challenging problem will arise with the amount of permgen consumed by the
                        ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

                        So far, my simple test of running with "-Xmx64m" has been a super way to
                        find out what the JVM is hanging up on. With my fixes I can get past game
                        mode and campaign loading, but I run out of RAM when loading required
                        sources for my sample PC, let alone loading the PC itself.

                        Chris


                        On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
                        > Do you have a sense of where the memory usage is (what type of object?)  I
                        > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
                        > Platform) in quite some time, so I won't claim my knowledge is current...
                        > but if you have a sense of where the issue is, I may be able to highlight
                        > pretty quickly the constructs we are using and how they might be improved.
                        >
                        > Thanks.
                        >
                        >
                        > TP.
                        >
                        > --
                        > Tom Parker
                        >
                        >
                        >
                        > ________________________________
                        >  From: Chris Dolan <chris@...>
                        > To: pcgen_developers@yahoogroups.com
                        > Sent: Sunday, January 27, 2013 11:37 PM
                        > Subject: Re: [pcgen_developers] memory usage
                        >
                        >
                        >
                        >
                        >
                        >
                        > A small progress note: I think I've hit the wall in intern()ing and
                        > deduplicating strings. I've gained about 10 MB of heap improvement, but
                        > there's only a couple of MB left for possible improvement left via the
                        > little tweaks I've been making and I'm still about 30-40 MB higher than I
                        > can afford running on Android. So more radical changes are certainly
                        > necessary.
                        >
                        > Right now, I'm still loading way more data than I need. My next experiment
                        > will be to add a new GameModeFilter option to the GameModeFileLoader so I
                        > can skip loading modes that are not interesting to my PC. I don't
                        > currently have an intuition about how much that will save (probably not
                        > enough), but it looks like just a couple hours of programming so I'm going
                        > to try it. Right now, I don't have a deep enough understanding of the
                        > PCGen code to risk massive speculative changes like an intermediate
                        > database, but I'm reading that threads with interest.
                        >
                        > As a side note, I think my work has improved startup time by about 5
                        > percent. I didn't expect this result, and don't fully understand it. I
                        > theorize that the improvement comes from reduced time spent int he garbage
                        > collector, but I'm not sure. 5% is almost certainly not noticeable to the
                        > average user, but every little bit helps.
                        >
                        > Chris
                        >
                        >
                        > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
                        >
                        > As a performance metric, I wrote a simple desktop app that loads a single
                        > (hard-coded) Pathfinder PCGen character and renders it with the
                        > CharacterSheetPanel. I measured time to load (about 40 seconds) and
                        > resulting memory usage (about 85 MB of heap when idle).
                        >>
                        >>
                        >>My test app is at the URL below (less than 200 lines of
                        >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
                        >>
                        >>
                        >>Then I went through the PCGen source looking at all of the Lst loading
                        >> paths and I added lots of .intern() clauses to the tokenined input to
                        >> detach them from the original file strings. This saved a notable amount
                        >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
                        >> than I hoped. My intern changes are visible on this git branch commit:
                        >> 
                        >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
                        >>
                        >>
                        >>I've attached a pretty chart of the heap dominators before and after my
                        >> change, according the Eclipse MAT plugin. I believe that the heap
                        >> associated with the AppClassLoader is the static fields, like in
                        >> SystemCollections, but I'm not sure. It's not surprising that Equipment
                        >> is the #1 user.
                        >>
                        >>
                        >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
                        >> classes don't show. Maybe I made a mistake?)
                        >><chart4-before.png>
                        >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
                        >> the others are about the same)
                        >>
                        >>
                        >><chart4-after.png>
                        >>
                        >>
                        >>Chris
                        >>
                        >>
                        >>P.S. I had an interesting discussion on StackOverflow about
                        >> String.intern(). Take a look:
                        >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
                        >
                        >
                        >
                        >




                        ------------------------------------

                        Yahoo! Groups Links

                        <*> To visit your group on the web, go to:
                            http://groups.yahoo.com/group/pcgen_developers/

                        <*> Your email settings:
                            Individual Email | Traditional

                        <*> To change settings online go to:
                            http://groups.yahoo.com/group/pcgen_developers/join
                            (Yahoo! ID required)

                        <*> To change settings via email:
                            pcgen_developers-digest@yahoogroups.com
                            pcgen_developers-fullfeatured@yahoogroups.com

                        <*> To unsubscribe from this group, send an email to:
                            pcgen_developers-unsubscribe@yahoogroups.com

                        <*> Your use of Yahoo! Groups is subject to:
                            http://docs.yahoo.com/info/terms/



                      • Chris Dolan
                        ... I think it s already off by default, but I ll double check. This would indeed be a big savings. ... I understand what you mean at a high level, but not the
                        Message 11 of 22 , Jan 28, 2013
                        • 0 Attachment
                          On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                          > A few thoughts:
                          >
                          > (1) Make sure the preferences are set to NOT create magical items by
                          > default.  I don't recall when this process actually happens (or even many
                          > of the exact details on what it is doing), but there is an ability to
                          > create objects, and you want that off.

                          I think it's already off by default, but I'll double check. This would
                          indeed be a big savings.

                          > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                          > instantiation on the fields.  This should shrink both Spell and Equipment,
                          > perhaps materially, since each field not instantiated is saving a HashMap,
                          > which is something on the magnitude of 100 bytes if I recall correctly. 
                          > Wouldn't surprise me if this saved a few MB across all of the objects in
                          > something like the RSRD.  (and it would take perhaps 10 minutes to
                          > implement)

                          I understand what you mean at a high level, but not the specifics. Are you
                          referring to the SystemCollections global? Maybe I just need to look at
                          the CDOMObject code and your meaning will be clear... (I don't have the
                          code on this machine...)

                          > (3) Evaluate a way to jettison the plugin classes after load.  This is
                          > easily hundreds of classes... which are loaded with an alternate
                          > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
                          > - PluginLoader?)   Once the data is loaded, can you destroy the
                          > classloader and destroy the caches in - I think it's TokenLibrary (or
                          > TokenStore or something to that effect - the code is not in front of me at
                          > the moment, but I think this is in pcgen.persistence.* somewhere).  That
                          > might then free up those classes to be garbage collected.

                          Excellent point. Along the same lines there are classes pinned by the
                          Spring loader's singleton cache. Without having looked at the Spring code,
                          I'll bet they're using a SoftReference or a WeakReference. I'll add
                          something like that in my NonSpringImpl that I built for Android.

                          Chris
                        • Tom Parker
                          On #2: I m referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I m recommending are all localized to that one class.  CDOMObject has a
                          Message 12 of 22 , Jan 28, 2013
                          • 0 Attachment
                            On #2:

                            I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I'm recommending are all localized to that one class. 

                            CDOMObject has a number of fields, something like:

                            private final Map<StringKey, String> stringMap = new HashMap<StringChar, String>();

                            then in methods, it is used as such:

                            public String get(StringKey sk)
                            {
                               return stringMap.get(sk);
                            }

                            public void put(StringKey sk, String s)
                            {
                               stringMap.put(sk, s);
                            }

                            This means the HashMap is instantiated as soon as the object is created ("aggressive instantiation", if you will).  What I recommend is "lazy instantiation", meaning only instantiate a map if the map is truly required (on a put).  So just change to:

                            private Map<StringKey, String> stringMap = null;

                            then in methods:

                            public String get(StringKey sk)
                            {
                               if (stringMap == null)
                               {
                                  return null;
                               }
                               return stringMap.get(sk);
                            }


                            public String put(StringKey sk, String s)
                            {
                               if (stringMap == null)
                               {
                                  stringMap = new HashMap<StringKey, String>();
                               }
                               stringMap.put(sk, s);
                            }

                            It adds in a whole ton of null checks across the class, making it a tiny bit less straightforward (and somewhere between infinitesimally and a tiny bit slower), but it will use less memory.

                            This is useful because simple objects that only have a key, a type, and a bonus or whatever will only use StringKey and ListKey, and none of the others.  So in any given object, a number of those maps are probably empty, and over lots of objects, that adds up.  If you assume RSRD loads 5K objects, and it saves 3 maps per object and 100 bytes per map, that's 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really never tested it.

                            p.s. My field name and method names are probably off, but you get the idea.

                            TP.
                            --
                            Tom Parker


                            From: Chris Dolan <chris@...>
                            To: pcgen_developers@yahoogroups.com
                            Sent: Monday, January 28, 2013 4:36 PM
                            Subject: Re: [pcgen_developers] memory usage

                            On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                            > A few thoughts:
                            >
                            > (1) Make sure the preferences are set to NOT create magical items by
                            > default.  I don't recall when this process actually happens (or even many
                            > of the exact details on what it is doing), but there is an ability to
                            > create objects, and you want that off.

                            I think it's already off by default, but I'll double check. This would
                            indeed be a big savings.

                            > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                            > instantiation on the fields.  This should shrink both Spell and Equipment,
                            > perhaps materially, since each field not instantiated is saving a HashMap,
                            > which is something on the magnitude of 100 bytes if I recall correctly. 
                            > Wouldn't surprise me if this saved a few MB across all of the objects in
                            > something like the RSRD.  (and it would take perhaps 10 minutes to
                            > implement)

                            I understand what you mean at a high level, but not the specifics. Are you
                            referring to the SystemCollections global? Maybe I just need to look at
                            the CDOMObject code and your meaning will be clear... (I don't have the
                            code on this machine...)

                            > (3) Evaluate a way to jettison the plugin classes after load.  This is
                            > easily hundreds of classes... which are loaded with an alternate
                            > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
                            > - PluginLoader?)   Once the data is loaded, can you destroy the
                            > classloader and destroy the caches in - I think it's TokenLibrary (or
                            > TokenStore or something to that effect - the code is not in front of me at
                            > the moment, but I think this is in pcgen.persistence.* somewhere).  That
                            > might then free up those classes to be garbage collected.

                            Excellent point. Along the same lines there are classes pinned by the
                            Spring loader's singleton cache. Without having looked at the Spring code,
                            I'll bet they're using a SoftReference or a WeakReference. I'll add
                            something like that in my NonSpringImpl that I built for Android.

                            Chris



                            ------------------------------------

                            Yahoo! Groups Links

                            <*> To visit your group on the web, go to:
                                http://groups.yahoo.com/group/pcgen_developers/

                            <*> Your email settings:
                                Individual Email | Traditional

                            <*> To change settings online go to:
                                http://groups.yahoo.com/group/pcgen_developers/join
                                (Yahoo! ID required)

                            <*> To change settings via email:
                                pcgen_developers-digest@yahoogroups.com
                                pcgen_developers-fullfeatured@yahoogroups.com

                            <*> To unsubscribe from this group, send an email to:
                                pcgen_developers-unsubscribe@yahoogroups.com

                            <*> Your use of Yahoo! Groups is subject to:
                                http://docs.yahoo.com/info/terms/



                          • Chris Dolan
                            Aha, awesome. I ll try that change this week and benchmark how much it helps memory vs. hurts speed. Thanks! Chris
                            Message 13 of 22 , Jan 28, 2013
                            • 0 Attachment
                              Aha, awesome. I'll try that change this week and benchmark how much it
                              helps memory vs. hurts speed. Thanks!
                              Chris


                              On Mon, January 28, 2013 3:56 pm, Tom Parker wrote:
                              > On #2:
                              >
                              > I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the
                              > changes I'm recommending are all localized to that one class. 
                              >
                              >
                              > CDOMObject has a number of fields, something like:
                              >
                              > private final Map<StringKey, String> stringMap = new HashMap<StringChar,
                              > String>();
                              >
                              > then in methods, it is used as such:
                              >
                              > public String get(StringKey sk)
                              > {
                              >    return stringMap.get(sk);
                              >
                              > }
                              >
                              > public void put(StringKey sk, String s)
                              > {
                              >    stringMap.put(sk, s);
                              >
                              > }
                              >
                              > This means the HashMap is instantiated as soon as the object is created
                              > ("aggressive instantiation", if you will).  What I recommend is "lazy
                              > instantiation", meaning only instantiate a map if the map is truly
                              > required (on a put).  So just change to:
                              >
                              >
                              > private Map<StringKey, String> stringMap = null;
                              >
                              > then in methods:
                              >
                              > public String get(StringKey sk)
                              > {
                              >    if (stringMap == null)
                              >    {
                              >       return null;
                              >    }
                              >
                              >    return stringMap.get(sk);
                              >
                              > }
                              >
                              >
                              > public String put(StringKey sk, String s)
                              > {
                              >    if (stringMap == null)
                              >    {
                              >       stringMap = new HashMap<StringKey, String>();
                              >    }
                              >
                              >    stringMap.put(sk, s);
                              >
                              > }
                              >
                              > It adds in a whole ton of null checks across the class, making it a tiny
                              > bit less straightforward (and somewhere between infinitesimally and a tiny
                              > bit slower), but it will use less memory.
                              >
                              >
                              > This is useful because simple objects that only have a key, a type, and a
                              > bonus or whatever will only use StringKey and ListKey, and none of the
                              > others.  So in any given object, a number of those maps are probably
                              > empty, and over lots of objects, that adds up.  If you assume RSRD loads
                              > 5K objects, and it saves 3 maps per object and 100 bytes per map, that's
                              > 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really
                              > never tested it.
                              >
                              >
                              > p.s. My field name and method names are probably off, but you get the
                              > idea.
                              >
                              > TP.
                              >
                              > --
                              > Tom Parker
                              >
                              >
                              >
                              > ________________________________
                              > From: Chris Dolan <chris@...>
                              > To: pcgen_developers@yahoogroups.com
                              > Sent: Monday, January 28, 2013 4:36 PM
                              > Subject: Re: [pcgen_developers] memory usage
                              >
                              > On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                              >> A few thoughts:
                              >>
                              >> (1) Make sure the preferences are set to NOT create magical items by
                              >> default.  I don't recall when this process actually happens (or even
                              >> many
                              >> of the exact details on what it is doing), but there is an ability to
                              >> create objects, and you want that off.
                              >
                              > I think it's already off by default, but I'll double check. This would
                              > indeed be a big savings.
                              >
                              >> (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                              >> instantiation on the fields.  This should shrink both Spell and
                              >> Equipment,
                              >> perhaps materially, since each field not instantiated is saving a
                              >> HashMap,
                              >> which is something on the magnitude of 100 bytes if I recall correctly. 
                              >> Wouldn't surprise me if this saved a few MB across all of the objects in
                              >> something like the RSRD.  (and it would take perhaps 10 minutes to
                              >> implement)
                              >
                              > I understand what you mean at a high level, but not the specifics. Are you
                              > referring to the SystemCollections global? Maybe I just need to look at
                              > the CDOMObject code and your meaning will be clear... (I don't have the
                              > code on this machine...)
                              >
                              >> (3) Evaluate a way to jettison the plugin classes after load.  This is
                              >> easily hundreds of classes... which are loaded with an alternate
                              >> classloader.  (This occurs somewhere in gmgen.* - I forget the exact
                              >> class
                              >> - PluginLoader?)   Once the data is loaded, can you destroy the
                              >> classloader and destroy the caches in - I think it's TokenLibrary (or
                              >> TokenStore or something to that effect - the code is not in front of me
                              >> at
                              >> the moment, but I think this is in pcgen.persistence.* somewhere).  That
                              >> might then free up those classes to be garbage collected.
                              >
                              > Excellent point. Along the same lines there are classes pinned by the
                              > Spring loader's singleton cache. Without having looked at the Spring code,
                              > I'll bet they're using a SoftReference or a WeakReference. I'll add
                              > something like that in my NonSpringImpl that I built for Android.
                              >
                              > Chris
                              >
                              >
                              >
                              > ------------------------------------
                              >
                              > Yahoo! Groups Links
                              >
                              >
                              >
                              >     http://docs.yahoo.com/info/terms/
                            • Tom Parker
                              As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many
                              Message 14 of 22 , Jan 28, 2013
                              • 0 Attachment

                                As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                                At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                                TP.
                                --
                                Tom Parker


                                From: Chris Dolan <chris@...>
                                To: pcgen_developers@yahoogroups.com
                                Sent: Monday, January 28, 2013 5:04 PM
                                Subject: Re: [pcgen_developers] memory usage

                                Aha, awesome. I'll try that change this week and benchmark how much it
                                helps memory vs. hurts speed. Thanks!
                                Chris

                              • Chris Dolan
                                Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the
                                Message 15 of 22 , Jan 28, 2013
                                • 0 Attachment
                                  Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                                  Chris



                                  On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                                  As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                                  At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                                  TP. 
                                  --
                                  Tom Parker


                                  From: Chris Dolan <chris@...>
                                  To: pcgen_developers@yahoogroups.com 
                                  Sent: Monday, January 28, 2013 5:04 PM
                                  Subject: Re: [pcgen_developers] memory usage

                                  Aha, awesome. I'll try that change this week and benchmark how much it
                                  helps memory vs. hurts speed. Thanks!
                                  Chris




                                • Tom Parker
                                  Since the map creation should all happen at LST load, the answer *should* be no. That being said, I make no guarantees about UI behavior.  I was not involved
                                  Message 16 of 22 , Jan 28, 2013
                                  • 0 Attachment
                                    Since the map creation should all happen at LST load, the answer *should* be no.

                                    That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                                    TP.
                                    --
                                    Tom Parker


                                    From: Chris Dolan <chris@...>
                                    To: pcgen_developers@yahoogroups.com
                                    Sent: Monday, January 28, 2013 7:32 PM
                                    Subject: Re: [pcgen_developers] memory usage



                                    Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                                    Chris



                                    On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                                    As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                                    At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                                    TP. 
                                    --
                                    Tom Parker


                                    From: Chris Dolan <chris@...>
                                    To: pcgen_developers@yahoogroups.com 
                                    Sent: Monday, January 28, 2013 5:04 PM
                                    Subject: Re: [pcgen_developers] memory usage

                                    Aha, awesome. I'll try that change this week and benchmark how much it
                                    helps memory vs. hurts speed. Thanks!
                                    Chris








                                  • James Dempsey
                                    Hi, After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output
                                    Message 17 of 22 , Jan 28, 2013
                                    • 0 Attachment
                                      Hi,

                                      After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output threads which keep the summary sheet, skills list and character sheet up to date.

                                      As Tom mentioned, these should not be updating the LST objects much though. Some cloning still remains though, which is where a small number of updates will occur. Tom is gradually eliminating those though.

                                      Cheers,
                                      James

                                      On 29 January 2013 11:41, Tom Parker <thpr@...> wrote:


                                      Since the map creation should all happen at LST load, the answer *should* be no.

                                      That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                                      TP.
                                      --
                                      Tom Parker

                                      Sent: Monday, January 28, 2013 7:32 PM

                                      Subject: Re: [pcgen_developers] memory usage



                                      Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                                      Chris



                                      On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                                      As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                                      At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                                      TP. 
                                      --
                                      Tom Parker


                                      From: Chris Dolan <chris@...>
                                      To: pcgen_developers@yahoogroups.com 
                                      Sent: Monday, January 28, 2013 5:04 PM
                                      Subject: Re: [pcgen_developers] memory usage

                                      Aha, awesome. I'll try that change this week and benchmark how much it
                                      helps memory vs. hurts speed. Thanks!
                                      Chris



                                    • FerretDave
                                      Greetings, This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ? Cheers D
                                      Message 18 of 22 , Jan 30, 2013
                                      • 0 Attachment
                                        Greetings,

                                        This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                                        Cheers
                                        D

                                        --- In pcgen_developers@yahoogroups.com, Tom Parker wrote:
                                        >
                                        >
                                        >
                                        > As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.
                                        >
                                        >
                                        > At load time, it will probably be faster, since it will be doing a lot less memory allocation.
                                        >
                                        >
                                        > TP.
                                        >
                                        > --
                                        > Tom Parker
                                        >
                                        >
                                        >
                                        > ________________________________
                                        > From: Chris Dolan
                                        > To: pcgen_developers@yahoogroups.com
                                        > Sent: Monday, January 28, 2013 5:04 PM
                                        > Subject: Re: [pcgen_developers] memory usage
                                        >
                                        > Aha, awesome. I'll try that change this week and benchmark how much it
                                        > helps memory vs. hurts speed. Thanks!
                                        > Chris
                                        >
                                      • Tom Parker
                                        Absolutely.  Like everything else it s been a matter of picking what I address first.   TP. -- Tom Parker ________________________________ From: FerretDave
                                        Message 19 of 22 , Jan 30, 2013
                                        • 0 Attachment
                                          Absolutely.  Like everything else it's been a matter of picking what I address first.
                                           
                                          TP.
                                          --
                                          Tom Parker


                                          From: FerretDave <ferret.griffin@...>
                                          To: pcgen_developers@yahoogroups.com
                                          Sent: Wednesday, January 30, 2013 4:44 AM
                                          Subject: [pcgen_developers] Re: memory usage

                                          Greetings,

                                          This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                                          Cheers
                                          D


                                        • Henk Slaaf
                                          Hey all, It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have. Then we could
                                          Message 20 of 22 , Jan 30, 2013
                                          • 0 Attachment

                                            Hey all,

                                            It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.

                                            Then we could graph the results.

                                            No idea if this is actually feasible, but just dreaming :-)

                                            Best,

                                            Henk

                                            On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                            As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                            My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                            Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                            I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                            BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                            AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                            Chris

                                            P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                          • Tom Parker
                                            also... Number of java warnings Processing time per character regression test Load errors/warnings for key dataset combinations Unit test coverage ...probably
                                            Message 21 of 22 , Jan 30, 2013
                                            • 0 Attachment
                                              also...

                                              Number of java warnings
                                              Processing time per character regression test
                                              Load errors/warnings for key dataset combinations
                                              Unit test coverage

                                              ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                              TP.
                                              --
                                              Tom Parker


                                              From: Henk Slaaf <henk@...>
                                              To: pcgen_developers@yahoogroups.com
                                              Sent: Wednesday, January 30, 2013 12:58 PM
                                              Subject: Re: [pcgen_developers] memory usage



                                              Hey all,
                                              It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                              Then we could graph the results.
                                              No idea if this is actually feasible, but just dreaming :-)
                                              Best,
                                              Henk
                                              On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                              As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                              My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                              Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                              I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                              BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                              AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                              Chris

                                              P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                              http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                                            • Chris Dolan
                                              For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up. Chris ... For most of those genetic
                                              Message 22 of 22 , Jan 30, 2013
                                              • 0 Attachment
                                                For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up.
                                                Chris

                                                Tom Parker <thpr@...> wrote:
                                                also...

                                                Number of java warnings
                                                Processing time per character regression test
                                                Load errors/warnings for key dataset combinations
                                                Unit test coverage

                                                ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                                TP.
                                                --
                                                Tom Parker


                                                From: Henk Slaaf <henk@...>
                                                To: pcgen_developers@yahoogroups.com
                                                Sent: Wednesday, January 30, 2013 12:58 PM
                                                Subject: Re: [pcgen_developers] memory usage



                                                Hey all,
                                                It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                                Then we could graph the results.
                                                No idea if this is actually feasible, but just dreaming :-)
                                                Best,
                                                Henk
                                                On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                                As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                                My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                                Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                                I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                                BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                                AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                                Chris

                                                P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                                http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                                              Your message has been successfully submitted and would be delivered to recipients shortly.