Loading ...
Sorry, an error occurred while loading the content.

Re: [pcgen_developers] memory usage

Expand Messages
  • Tom Parker
    Do you have a sense of where the memory usage is (what type of object?)  I haven t put PCGen itno TPTP (Eclipse s Test and Performance Tools Platform) in
    Message 1 of 22 , Jan 28, 2013
    • 0 Attachment
      Do you have a sense of where the memory usage is (what type of object?)  I haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools Platform) in quite some time, so I won't claim my knowledge is current... but if you have a sense of where the issue is, I may be able to highlight pretty quickly the constructs we are using and how they might be improved.

      Thanks.

      TP.
      --
      Tom Parker


      From: Chris Dolan <chris@...>
      To: pcgen_developers@yahoogroups.com
      Sent: Sunday, January 27, 2013 11:37 PM
      Subject: Re: [pcgen_developers] memory usage



      A small progress note: I think I've hit the wall in intern()ing and deduplicating strings. I've gained about 10 MB of heap improvement, but there's only a couple of MB left for possible improvement left via the little tweaks I've been making and I'm still about 30-40 MB higher than I can afford running on Android. So more radical changes are certainly necessary.

      Right now, I'm still loading way more data than I need. My next experiment will be to add a new GameModeFilter option to the GameModeFileLoader so I can skip loading modes that are not interesting to my PC. I don't currently have an intuition about how much that will save (probably not enough), but it looks like just a couple hours of programming so I'm going to try it. Right now, I don't have a deep enough understanding of the PCGen code to risk massive speculative changes like an intermediate database, but I'm reading that threads with interest.

      As a side note, I think my work has improved startup time by about 5 percent. I didn't expect this result, and don't fully understand it. I theorize that the improvement comes from reduced time spent int he garbage collector, but I'm not sure. 5% is almost certainly not noticeable to the average user, but every little bit helps.

      Chris


      On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:

      As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

      My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

      Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

      I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

      BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
      <chart4-before.png>
      AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)

      <chart4-after.png>

      Chris

      P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
      http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string





    • Chris Dolan
      Yep, I ve been using the Eclipse Memory Analysis Tools (MAT) which analyze the .hprof heap dumps that VisualVM emits. It used to be that char[] was the biggest
      Message 2 of 22 , Jan 28, 2013
      • 0 Attachment
        Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
        the .hprof heap dumps that VisualVM emits. It used to be that char[] was
        the biggest consumer due to leaked substring parents, but I've knocked
        that one out of the top 10. The next biggest are Equipment and Spell,
        which is not at all surprising. However, my gut says that a more
        challenging problem will arise with the amount of permgen consumed by the
        ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

        So far, my simple test of running with "-Xmx64m" has been a super way to
        find out what the JVM is hanging up on. With my fixes I can get past game
        mode and campaign loading, but I run out of RAM when loading required
        sources for my sample PC, let alone loading the PC itself.

        Chris


        On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
        > Do you have a sense of where the memory usage is (what type of object?)  I
        > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
        > Platform) in quite some time, so I won't claim my knowledge is current...
        > but if you have a sense of where the issue is, I may be able to highlight
        > pretty quickly the constructs we are using and how they might be improved.
        >
        > Thanks.
        >
        >
        > TP.
        >
        > --
        > Tom Parker
        >
        >
        >
        > ________________________________
        > From: Chris Dolan <chris@...>
        > To: pcgen_developers@yahoogroups.com
        > Sent: Sunday, January 27, 2013 11:37 PM
        > Subject: Re: [pcgen_developers] memory usage
        >
        >
        >
        >
        >
        >
        > A small progress note: I think I've hit the wall in intern()ing and
        > deduplicating strings. I've gained about 10 MB of heap improvement, but
        > there's only a couple of MB left for possible improvement left via the
        > little tweaks I've been making and I'm still about 30-40 MB higher than I
        > can afford running on Android. So more radical changes are certainly
        > necessary.
        >
        > Right now, I'm still loading way more data than I need. My next experiment
        > will be to add a new GameModeFilter option to the GameModeFileLoader so I
        > can skip loading modes that are not interesting to my PC. I don't
        > currently have an intuition about how much that will save (probably not
        > enough), but it looks like just a couple hours of programming so I'm going
        > to try it. Right now, I don't have a deep enough understanding of the
        > PCGen code to risk massive speculative changes like an intermediate
        > database, but I'm reading that threads with interest.
        >
        > As a side note, I think my work has improved startup time by about 5
        > percent. I didn't expect this result, and don't fully understand it. I
        > theorize that the improvement comes from reduced time spent int he garbage
        > collector, but I'm not sure. 5% is almost certainly not noticeable to the
        > average user, but every little bit helps.
        >
        > Chris
        >
        >
        > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
        >
        > As a performance metric, I wrote a simple desktop app that loads a single
        > (hard-coded) Pathfinder PCGen character and renders it with the
        > CharacterSheetPanel. I measured time to load (about 40 seconds) and
        > resulting memory usage (about 85 MB of heap when idle).
        >>
        >>
        >>My test app is at the URL below (less than 200 lines of
        >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
        >>
        >>
        >>Then I went through the PCGen source looking at all of the Lst loading
        >> paths and I added lots of .intern() clauses to the tokenined input to
        >> detach them from the original file strings. This saved a notable amount
        >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
        >> than I hoped. My intern changes are visible on this git branch commit:
        >> 
        >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
        >>
        >>
        >>I've attached a pretty chart of the heap dominators before and after my
        >> change, according the Eclipse MAT plugin. I believe that the heap
        >> associated with the AppClassLoader is the static fields, like in
        >> SystemCollections, but I'm not sure. It's not surprising that Equipment
        >> is the #1 user.
        >>
        >>
        >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
        >> classes don't show. Maybe I made a mistake?)
        >><chart4-before.png>
        >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
        >> the others are about the same)
        >>
        >>
        >><chart4-after.png>
        >>
        >>
        >>Chris
        >>
        >>
        >>P.S. I had an interesting discussion on StackOverflow about
        >> String.intern(). Take a look:
        >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
        >
        >
        >
        >
      • Tom Parker
        A few thoughts: (1) Make sure the preferences are set to NOT create magical items by default.  I don t recall when this process actually happens (or even many
        Message 3 of 22 , Jan 28, 2013
        • 0 Attachment
          A few thoughts:

          (1) Make sure the preferences are set to NOT create magical items by default.  I don't recall when this process actually happens (or even many of the exact details on what it is doing), but there is an ability to create objects, and you want that off.

          (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy instantiation on the fields.  This should shrink both Spell and Equipment, perhaps materially, since each field not instantiated is saving a HashMap, which is something on the magnitude of 100 bytes if I recall correctly.  Wouldn't surprise me if this saved a few MB across all of the objects in something like the RSRD.  (and it would take perhaps 10 minutes to implement)

          (3) Evaluate a way to jettison the plugin classes after load.  This is easily hundreds of classes... which are loaded with an alternate classloader.  (This occurs somewhere in gmgen.* - I forget the exact class - PluginLoader?)   Once the data is loaded, can you destroy the classloader and destroy the caches in - I think it's TokenLibrary (or TokenStore or something to that effect - the code is not in front of me at the moment, but I think this is in pcgen.persistence.* somewhere).  That might then free up those classes to be garbage collected.

          TP.
          --
          Tom Parker


          From: Chris Dolan <chris@...>
          To: pcgen_developers@yahoogroups.com
          Sent: Monday, January 28, 2013 10:27 AM
          Subject: Re: [pcgen_developers] memory usage

          Yep, I've been using the Eclipse Memory Analysis Tools (MAT) which analyze
          the .hprof heap dumps that VisualVM emits. It used to be that char[] was
          the biggest consumer due to leaked substring parents, but I've knocked
          that one out of the top 10. The next biggest are Equipment and Spell,
          which is not at all surprising. However, my gut says that a more
          challenging problem will arise with the amount of permgen consumed by the
          ~5000 .class files that pcgen loads (including JRE and 3rd party jars).

          So far, my simple test of running with "-Xmx64m" has been a super way to
          find out what the JVM is hanging up on. With my fixes I can get past game
          mode and campaign loading, but I run out of RAM when loading required
          sources for my sample PC, let alone loading the PC itself.

          Chris


          On Mon, January 28, 2013 8:47 am, Tom Parker wrote:
          > Do you have a sense of where the memory usage is (what type of object?)  I
          > haven't put PCGen itno TPTP (Eclipse's Test and Performance Tools
          > Platform) in quite some time, so I won't claim my knowledge is current...
          > but if you have a sense of where the issue is, I may be able to highlight
          > pretty quickly the constructs we are using and how they might be improved.
          >
          > Thanks.
          >
          >
          > TP.
          >
          > --
          > Tom Parker
          >
          >
          >
          > ________________________________
          >  From: Chris Dolan <chris@...>
          > To: pcgen_developers@yahoogroups.com
          > Sent: Sunday, January 27, 2013 11:37 PM
          > Subject: Re: [pcgen_developers] memory usage
          >
          >
          >
          >
          >
          >
          > A small progress note: I think I've hit the wall in intern()ing and
          > deduplicating strings. I've gained about 10 MB of heap improvement, but
          > there's only a couple of MB left for possible improvement left via the
          > little tweaks I've been making and I'm still about 30-40 MB higher than I
          > can afford running on Android. So more radical changes are certainly
          > necessary.
          >
          > Right now, I'm still loading way more data than I need. My next experiment
          > will be to add a new GameModeFilter option to the GameModeFileLoader so I
          > can skip loading modes that are not interesting to my PC. I don't
          > currently have an intuition about how much that will save (probably not
          > enough), but it looks like just a couple hours of programming so I'm going
          > to try it. Right now, I don't have a deep enough understanding of the
          > PCGen code to risk massive speculative changes like an intermediate
          > database, but I'm reading that threads with interest.
          >
          > As a side note, I think my work has improved startup time by about 5
          > percent. I didn't expect this result, and don't fully understand it. I
          > theorize that the improvement comes from reduced time spent int he garbage
          > collector, but I'm not sure. 5% is almost certainly not noticeable to the
          > average user, but every little bit helps.
          >
          > Chris
          >
          >
          > On Jan 25, 2013, at 2:25 AM, Chris Dolan wrote:
          >
          > As a performance metric, I wrote a simple desktop app that loads a single
          > (hard-coded) Pathfinder PCGen character and renders it with the
          > CharacterSheetPanel. I measured time to load (about 40 seconds) and
          > resulting memory usage (about 85 MB of heap when idle).
          >>
          >>
          >>My test app is at the URL below (less than 200 lines of
          >> code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java
          >>
          >>
          >>Then I went through the PCGen source looking at all of the Lst loading
          >> paths and I added lots of .intern() clauses to the tokenined input to
          >> detach them from the original file strings. This saved a notable amount
          >> of memory (now 790 MB instead of 85 MB), but much less of an improvement
          >> than I hoped. My intern changes are visible on this git branch commit:
          >> 
          >>  https://github.com/chrisdolan/pcgen-svn/commit/e44a23ddf264ef7c6651ec61a2bed261671674db
          >>
          >>
          >>I've attached a pretty chart of the heap dominators before and after my
          >> change, according the Eclipse MAT plugin. I believe that the heap
          >> associated with the AppClassLoader is the static fields, like in
          >> SystemCollections, but I'm not sure. It's not surprising that Equipment
          >> is the #1 user.
          >>
          >>
          >>BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML
          >> classes don't show. Maybe I made a mistake?)
          >><chart4-before.png>
          >>AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and
          >> the others are about the same)
          >>
          >>
          >><chart4-after.png>
          >>
          >>
          >>Chris
          >>
          >>
          >>P.S. I had an interesting discussion on StackOverflow about
          >> String.intern(). Take a look:
          >>http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string
          >
          >
          >
          >




          ------------------------------------

          Yahoo! Groups Links

          <*> To visit your group on the web, go to:
              http://groups.yahoo.com/group/pcgen_developers/

          <*> Your email settings:
              Individual Email | Traditional

          <*> To change settings online go to:
              http://groups.yahoo.com/group/pcgen_developers/join
              (Yahoo! ID required)

          <*> To change settings via email:
              pcgen_developers-digest@yahoogroups.com
              pcgen_developers-fullfeatured@yahoogroups.com

          <*> To unsubscribe from this group, send an email to:
              pcgen_developers-unsubscribe@yahoogroups.com

          <*> Your use of Yahoo! Groups is subject to:
              http://docs.yahoo.com/info/terms/



        • Chris Dolan
          ... I think it s already off by default, but I ll double check. This would indeed be a big savings. ... I understand what you mean at a high level, but not the
          Message 4 of 22 , Jan 28, 2013
          • 0 Attachment
            On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
            > A few thoughts:
            >
            > (1) Make sure the preferences are set to NOT create magical items by
            > default.  I don't recall when this process actually happens (or even many
            > of the exact details on what it is doing), but there is an ability to
            > create objects, and you want that off.

            I think it's already off by default, but I'll double check. This would
            indeed be a big savings.

            > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
            > instantiation on the fields.  This should shrink both Spell and Equipment,
            > perhaps materially, since each field not instantiated is saving a HashMap,
            > which is something on the magnitude of 100 bytes if I recall correctly. 
            > Wouldn't surprise me if this saved a few MB across all of the objects in
            > something like the RSRD.  (and it would take perhaps 10 minutes to
            > implement)

            I understand what you mean at a high level, but not the specifics. Are you
            referring to the SystemCollections global? Maybe I just need to look at
            the CDOMObject code and your meaning will be clear... (I don't have the
            code on this machine...)

            > (3) Evaluate a way to jettison the plugin classes after load.  This is
            > easily hundreds of classes... which are loaded with an alternate
            > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
            > - PluginLoader?)   Once the data is loaded, can you destroy the
            > classloader and destroy the caches in - I think it's TokenLibrary (or
            > TokenStore or something to that effect - the code is not in front of me at
            > the moment, but I think this is in pcgen.persistence.* somewhere).  That
            > might then free up those classes to be garbage collected.

            Excellent point. Along the same lines there are classes pinned by the
            Spring loader's singleton cache. Without having looked at the Spring code,
            I'll bet they're using a SoftReference or a WeakReference. I'll add
            something like that in my NonSpringImpl that I built for Android.

            Chris
          • Tom Parker
            On #2: I m referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I m recommending are all localized to that one class.  CDOMObject has a
            Message 5 of 22 , Jan 28, 2013
            • 0 Attachment
              On #2:

              I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the changes I'm recommending are all localized to that one class. 

              CDOMObject has a number of fields, something like:

              private final Map<StringKey, String> stringMap = new HashMap<StringChar, String>();

              then in methods, it is used as such:

              public String get(StringKey sk)
              {
                 return stringMap.get(sk);
              }

              public void put(StringKey sk, String s)
              {
                 stringMap.put(sk, s);
              }

              This means the HashMap is instantiated as soon as the object is created ("aggressive instantiation", if you will).  What I recommend is "lazy instantiation", meaning only instantiate a map if the map is truly required (on a put).  So just change to:

              private Map<StringKey, String> stringMap = null;

              then in methods:

              public String get(StringKey sk)
              {
                 if (stringMap == null)
                 {
                    return null;
                 }
                 return stringMap.get(sk);
              }


              public String put(StringKey sk, String s)
              {
                 if (stringMap == null)
                 {
                    stringMap = new HashMap<StringKey, String>();
                 }
                 stringMap.put(sk, s);
              }

              It adds in a whole ton of null checks across the class, making it a tiny bit less straightforward (and somewhere between infinitesimally and a tiny bit slower), but it will use less memory.

              This is useful because simple objects that only have a key, a type, and a bonus or whatever will only use StringKey and ListKey, and none of the others.  So in any given object, a number of those maps are probably empty, and over lots of objects, that adds up.  If you assume RSRD loads 5K objects, and it saves 3 maps per object and 100 bytes per map, that's 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really never tested it.

              p.s. My field name and method names are probably off, but you get the idea.

              TP.
              --
              Tom Parker


              From: Chris Dolan <chris@...>
              To: pcgen_developers@yahoogroups.com
              Sent: Monday, January 28, 2013 4:36 PM
              Subject: Re: [pcgen_developers] memory usage

              On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
              > A few thoughts:
              >
              > (1) Make sure the preferences are set to NOT create magical items by
              > default.  I don't recall when this process actually happens (or even many
              > of the exact details on what it is doing), but there is an ability to
              > create objects, and you want that off.

              I think it's already off by default, but I'll double check. This would
              indeed be a big savings.

              > (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
              > instantiation on the fields.  This should shrink both Spell and Equipment,
              > perhaps materially, since each field not instantiated is saving a HashMap,
              > which is something on the magnitude of 100 bytes if I recall correctly. 
              > Wouldn't surprise me if this saved a few MB across all of the objects in
              > something like the RSRD.  (and it would take perhaps 10 minutes to
              > implement)

              I understand what you mean at a high level, but not the specifics. Are you
              referring to the SystemCollections global? Maybe I just need to look at
              the CDOMObject code and your meaning will be clear... (I don't have the
              code on this machine...)

              > (3) Evaluate a way to jettison the plugin classes after load.  This is
              > easily hundreds of classes... which are loaded with an alternate
              > classloader.  (This occurs somewhere in gmgen.* - I forget the exact class
              > - PluginLoader?)   Once the data is loaded, can you destroy the
              > classloader and destroy the caches in - I think it's TokenLibrary (or
              > TokenStore or something to that effect - the code is not in front of me at
              > the moment, but I think this is in pcgen.persistence.* somewhere).  That
              > might then free up those classes to be garbage collected.

              Excellent point. Along the same lines there are classes pinned by the
              Spring loader's singleton cache. Without having looked at the Spring code,
              I'll bet they're using a SoftReference or a WeakReference. I'll add
              something like that in my NonSpringImpl that I built for Android.

              Chris



              ------------------------------------

              Yahoo! Groups Links

              <*> To visit your group on the web, go to:
                  http://groups.yahoo.com/group/pcgen_developers/

              <*> Your email settings:
                  Individual Email | Traditional

              <*> To change settings online go to:
                  http://groups.yahoo.com/group/pcgen_developers/join
                  (Yahoo! ID required)

              <*> To change settings via email:
                  pcgen_developers-digest@yahoogroups.com
                  pcgen_developers-fullfeatured@yahoogroups.com

              <*> To unsubscribe from this group, send an email to:
                  pcgen_developers-unsubscribe@yahoogroups.com

              <*> Your use of Yahoo! Groups is subject to:
                  http://docs.yahoo.com/info/terms/



            • Chris Dolan
              Aha, awesome. I ll try that change this week and benchmark how much it helps memory vs. hurts speed. Thanks! Chris
              Message 6 of 22 , Jan 28, 2013
              • 0 Attachment
                Aha, awesome. I'll try that change this week and benchmark how much it
                helps memory vs. hurts speed. Thanks!
                Chris


                On Mon, January 28, 2013 3:56 pm, Tom Parker wrote:
                > On #2:
                >
                > I'm referring to the contents of pcgen.cdom.base.CDOMObject, so the
                > changes I'm recommending are all localized to that one class. 
                >
                >
                > CDOMObject has a number of fields, something like:
                >
                > private final Map<StringKey, String> stringMap = new HashMap<StringChar,
                > String>();
                >
                > then in methods, it is used as such:
                >
                > public String get(StringKey sk)
                > {
                >    return stringMap.get(sk);
                >
                > }
                >
                > public void put(StringKey sk, String s)
                > {
                >    stringMap.put(sk, s);
                >
                > }
                >
                > This means the HashMap is instantiated as soon as the object is created
                > ("aggressive instantiation", if you will).  What I recommend is "lazy
                > instantiation", meaning only instantiate a map if the map is truly
                > required (on a put).  So just change to:
                >
                >
                > private Map<StringKey, String> stringMap = null;
                >
                > then in methods:
                >
                > public String get(StringKey sk)
                > {
                >    if (stringMap == null)
                >    {
                >       return null;
                >    }
                >
                >    return stringMap.get(sk);
                >
                > }
                >
                >
                > public String put(StringKey sk, String s)
                > {
                >    if (stringMap == null)
                >    {
                >       stringMap = new HashMap<StringKey, String>();
                >    }
                >
                >    stringMap.put(sk, s);
                >
                > }
                >
                > It adds in a whole ton of null checks across the class, making it a tiny
                > bit less straightforward (and somewhere between infinitesimally and a tiny
                > bit slower), but it will use less memory.
                >
                >
                > This is useful because simple objects that only have a key, a type, and a
                > bonus or whatever will only use StringKey and ListKey, and none of the
                > others.  So in any given object, a number of those maps are probably
                > empty, and over lots of objects, that adds up.  If you assume RSRD loads
                > 5K objects, and it saves 3 maps per object and 100 bytes per map, that's
                > 5000*3*100 = 1.5MB of memory.  It may be more than that... I've really
                > never tested it.
                >
                >
                > p.s. My field name and method names are probably off, but you get the
                > idea.
                >
                > TP.
                >
                > --
                > Tom Parker
                >
                >
                >
                > ________________________________
                > From: Chris Dolan <chris@...>
                > To: pcgen_developers@yahoogroups.com
                > Sent: Monday, January 28, 2013 4:36 PM
                > Subject: Re: [pcgen_developers] memory usage
                >
                > On Mon, January 28, 2013 3:21 pm, Tom Parker wrote:
                >> A few thoughts:
                >>
                >> (1) Make sure the preferences are set to NOT create magical items by
                >> default.  I don't recall when this process actually happens (or even
                >> many
                >> of the exact details on what it is doing), but there is an ability to
                >> create objects, and you want that off.
                >
                > I think it's already off by default, but I'll double check. This would
                > indeed be a big savings.
                >
                >> (2) As per a previous note, change pcgen.cdom.base.CDOMObject to do lazy
                >> instantiation on the fields.  This should shrink both Spell and
                >> Equipment,
                >> perhaps materially, since each field not instantiated is saving a
                >> HashMap,
                >> which is something on the magnitude of 100 bytes if I recall correctly. 
                >> Wouldn't surprise me if this saved a few MB across all of the objects in
                >> something like the RSRD.  (and it would take perhaps 10 minutes to
                >> implement)
                >
                > I understand what you mean at a high level, but not the specifics. Are you
                > referring to the SystemCollections global? Maybe I just need to look at
                > the CDOMObject code and your meaning will be clear... (I don't have the
                > code on this machine...)
                >
                >> (3) Evaluate a way to jettison the plugin classes after load.  This is
                >> easily hundreds of classes... which are loaded with an alternate
                >> classloader.  (This occurs somewhere in gmgen.* - I forget the exact
                >> class
                >> - PluginLoader?)   Once the data is loaded, can you destroy the
                >> classloader and destroy the caches in - I think it's TokenLibrary (or
                >> TokenStore or something to that effect - the code is not in front of me
                >> at
                >> the moment, but I think this is in pcgen.persistence.* somewhere).  That
                >> might then free up those classes to be garbage collected.
                >
                > Excellent point. Along the same lines there are classes pinned by the
                > Spring loader's singleton cache. Without having looked at the Spring code,
                > I'll bet they're using a SoftReference or a WeakReference. I'll add
                > something like that in my NonSpringImpl that I built for Android.
                >
                > Chris
                >
                >
                >
                > ------------------------------------
                >
                > Yahoo! Groups Links
                >
                >
                >
                >     http://docs.yahoo.com/info/terms/
              • Tom Parker
                As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many
                Message 7 of 22 , Jan 28, 2013
                • 0 Attachment

                  As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                  At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                  TP.
                  --
                  Tom Parker


                  From: Chris Dolan <chris@...>
                  To: pcgen_developers@yahoogroups.com
                  Sent: Monday, January 28, 2013 5:04 PM
                  Subject: Re: [pcgen_developers] memory usage

                  Aha, awesome. I'll try that change this week and benchmark how much it
                  helps memory vs. hurts speed. Thanks!
                  Chris

                • Chris Dolan
                  Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the
                  Message 8 of 22 , Jan 28, 2013
                  • 0 Attachment
                    Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                    Chris



                    On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                    As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                    At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                    TP. 
                    --
                    Tom Parker


                    From: Chris Dolan <chris@...>
                    To: pcgen_developers@yahoogroups.com 
                    Sent: Monday, January 28, 2013 5:04 PM
                    Subject: Re: [pcgen_developers] memory usage

                    Aha, awesome. I'll try that change this week and benchmark how much it
                    helps memory vs. hurts speed. Thanks!
                    Chris




                  • Tom Parker
                    Since the map creation should all happen at LST load, the answer *should* be no. That being said, I make no guarantees about UI behavior.  I was not involved
                    Message 9 of 22 , Jan 28, 2013
                    • 0 Attachment
                      Since the map creation should all happen at LST load, the answer *should* be no.

                      That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                      TP.
                      --
                      Tom Parker


                      From: Chris Dolan <chris@...>
                      To: pcgen_developers@yahoogroups.com
                      Sent: Monday, January 28, 2013 7:32 PM
                      Subject: Re: [pcgen_developers] memory usage



                      Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                      Chris



                      On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                      As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                      At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                      TP. 
                      --
                      Tom Parker


                      From: Chris Dolan <chris@...>
                      To: pcgen_developers@yahoogroups.com 
                      Sent: Monday, January 28, 2013 5:04 PM
                      Subject: Re: [pcgen_developers] memory usage

                      Aha, awesome. I'll try that change this week and benchmark how much it
                      helps memory vs. hurts speed. Thanks!
                      Chris








                    • James Dempsey
                      Hi, After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output
                      Message 10 of 22 , Jan 28, 2013
                      • 0 Attachment
                        Hi,

                        After load, the UI can have multiple threads accessing the objects. These are the AWT event thread (used for most character updates) and the HTML output threads which keep the summary sheet, skills list and character sheet up to date.

                        As Tom mentioned, these should not be updating the LST objects much though. Some cloning still remains though, which is where a small number of updates will occur. Tom is gradually eliminating those though.

                        Cheers,
                        James

                        On 29 January 2013 11:41, Tom Parker <thpr@...> wrote:


                        Since the map creation should all happen at LST load, the answer *should* be no.

                        That being said, I make no guarantees about UI behavior.  I was not involved in that project.  James or Connor might be able to help with what the UI is doing during LST load.

                        TP.
                        --
                        Tom Parker

                        Sent: Monday, January 28, 2013 7:32 PM

                        Subject: Re: [pcgen_developers] memory usage



                        Do I need to worry about synchronization of the lazy instantiation? It looks like most of PCGen is two-threaded: one for loading and the UI thread, and the flow of data is purely one-way. So I suspect the answer is no, synchronization is not important.

                        Chris



                        On Jan 28, 2013, at 4:53 PM, Tom Parker wrote:




                        As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.

                        At load time, it will probably be faster, since it will be doing a lot less memory allocation.

                        TP. 
                        --
                        Tom Parker


                        From: Chris Dolan <chris@...>
                        To: pcgen_developers@yahoogroups.com 
                        Sent: Monday, January 28, 2013 5:04 PM
                        Subject: Re: [pcgen_developers] memory usage

                        Aha, awesome. I'll try that change this week and benchmark how much it
                        helps memory vs. hurts speed. Thanks!
                        Chris



                      • FerretDave
                        Greetings, This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ? Cheers D
                        Message 11 of 22 , Jan 30, 2013
                        • 0 Attachment
                          Greetings,

                          This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                          Cheers
                          D

                          --- In pcgen_developers@yahoogroups.com, Tom Parker wrote:
                          >
                          >
                          >
                          > As a note, the speed I was referring to is at runtime [when a character is created/viewed/modified].  Though on second thought that may get faster as in many cases it may save a method call... vs some cases where it's a useless null check... (which is the penalty of lazy instantiation)... but either way, you will be lucky to even be able to measure the difference since the things I'm talking about are a handful of CPU cycles, and not really material in the grand scheme.
                          >
                          >
                          > At load time, it will probably be faster, since it will be doing a lot less memory allocation.
                          >
                          >
                          > TP.
                          >
                          > --
                          > Tom Parker
                          >
                          >
                          >
                          > ________________________________
                          > From: Chris Dolan
                          > To: pcgen_developers@yahoogroups.com
                          > Sent: Monday, January 28, 2013 5:04 PM
                          > Subject: Re: [pcgen_developers] memory usage
                          >
                          > Aha, awesome. I'll try that change this week and benchmark how much it
                          > helps memory vs. hurts speed. Thanks!
                          > Chris
                          >
                        • Tom Parker
                          Absolutely.  Like everything else it s been a matter of picking what I address first.   TP. -- Tom Parker ________________________________ From: FerretDave
                          Message 12 of 22 , Jan 30, 2013
                          • 0 Attachment
                            Absolutely.  Like everything else it's been a matter of picking what I address first.
                             
                            TP.
                            --
                            Tom Parker


                            From: FerretDave <ferret.griffin@...>
                            To: pcgen_developers@yahoogroups.com
                            Sent: Wednesday, January 30, 2013 4:44 AM
                            Subject: [pcgen_developers] Re: memory usage

                            Greetings,

                            This sounds like it could be of benefit to the core/desktop version, never mind just the mobile viewer project ?

                            Cheers
                            D


                          • Henk Slaaf
                            Hey all, It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have. Then we could
                            Message 13 of 22 , Jan 30, 2013
                            • 0 Attachment

                              Hey all,

                              It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.

                              Then we could graph the results.

                              No idea if this is actually feasible, but just dreaming :-)

                              Best,

                              Henk

                              On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                              As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                              My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                              Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                              I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                              BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                              AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                              Chris

                              P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                            • Tom Parker
                              also... Number of java warnings Processing time per character regression test Load errors/warnings for key dataset combinations Unit test coverage ...probably
                              Message 14 of 22 , Jan 30, 2013
                              • 0 Attachment
                                also...

                                Number of java warnings
                                Processing time per character regression test
                                Load errors/warnings for key dataset combinations
                                Unit test coverage

                                ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                TP.
                                --
                                Tom Parker


                                From: Henk Slaaf <henk@...>
                                To: pcgen_developers@yahoogroups.com
                                Sent: Wednesday, January 30, 2013 12:58 PM
                                Subject: Re: [pcgen_developers] memory usage



                                Hey all,
                                It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                Then we could graph the results.
                                No idea if this is actually feasible, but just dreaming :-)
                                Best,
                                Henk
                                On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                Chris

                                P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                              • Chris Dolan
                                For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up. Chris ... For most of those genetic
                                Message 15 of 22 , Jan 30, 2013
                                • 0 Attachment
                                  For most of those genetic metrics, I recommend Sonar. It needs maven pom.xml but otherwise is really easy to set up.
                                  Chris

                                  Tom Parker <thpr@...> wrote:
                                  also...

                                  Number of java warnings
                                  Processing time per character regression test
                                  Load errors/warnings for key dataset combinations
                                  Unit test coverage

                                  ...probably a lot more, but there are a lot of interesting things we could graph over time if someone was up to the task.  Like many other things, just not my top priority.

                                  TP.
                                  --
                                  Tom Parker


                                  From: Henk Slaaf <henk@...>
                                  To: pcgen_developers@yahoogroups.com
                                  Sent: Wednesday, January 30, 2013 12:58 PM
                                  Subject: Re: [pcgen_developers] memory usage



                                  Hey all,
                                  It would be cool if Jenkins could run a memory benchmark of every commit for a few sample characters to see what effects commits have.
                                  Then we could graph the results.
                                  No idea if this is actually feasible, but just dreaming :-)
                                  Best,
                                  Henk
                                  On Jan 25, 2013 9:25 AM, "Chris Dolan" <chris@...> wrote:
                                  As a performance metric, I wrote a simple desktop app that loads a single (hard-coded) Pathfinder PCGen character and renders it with the CharacterSheetPanel. I measured time to load (about 40 seconds) and resulting memory usage (about 85 MB of heap when idle).

                                  My test app is at the URL below (less than 200 lines of code): https://github.com/chrisdolan/pcgen-android/blob/acf2de4e874697329246b9378525f7fc4852f344/pcgenuitest/src/net/chrisdolan/pcgen/viewer/uitest/HtmlSheet.java

                                  Then I went through the PCGen source looking at all of the Lst loading paths and I added lots of .intern() clauses to the tokenined input to detach them from the original file strings. This saved a notable amount of memory (now 790 MB instead of 85 MB), but much less of an improvement than I hoped. My intern changes are visible on this git branch commit:

                                  I've attached a pretty chart of the heap dominators before and after my change, according the Eclipse MAT plugin. I believe that the heap associated with the AppClassLoader is the static fields, like in SystemCollections, but I'm not sure. It's not surprising that Equipment is the #1 user.

                                  BEFORE: (note that "char[]" is the 4th biggest, not sure why the HTML classes don't show. Maybe I made a mistake?)
                                  AFTER: (note that "char[]" has vanished, AppClassLoader has shrunk, and the others are about the same)


                                  Chris

                                  P.S. I had an interesting discussion on StackOverflow about String.intern(). Take a look:
                                  http://stackoverflow.com/questions/14516635/how-do-i-reclaim-memory-after-parsing-via-substrings-intern-or-new-string




                                Your message has been successfully submitted and would be delivered to recipients shortly.