Loading ...
Sorry, an error occurred while loading the content.

Re: halting my Android project

Expand Messages
  • FerretDave
    Sorry to hear that, hopefully your improvements can help the desktop version though, and fingers crossed somebody can look into the other improvements. Fully
    Message 1 of 8 , Feb 20, 2013
    • 0 Attachment
      Sorry to hear that, hopefully your improvements can help the desktop version though, and fingers crossed somebody can look into the other improvements.

      Fully understand the issues with not enough time...

      --- In pcgen_developers@yahoogroups.com, Chris Dolan <chris@...> wrote:
      >
      > This will be my last update for a while: I'm shelving my Android port of PCGen for now. I got it to the point where it would load and render a PC. However, it's about a factor of 100 too slow for real-world use (7 minutes on my tablet, 35 minutes on my phone). I think it might be possible to achieve that factor of 100 with lots of caching and trimmed-down loading, but it's too much work for my hobby programming hours so I'm stopping. Realistically, the whole approach to LST parsing would need to be revisited (and perhaps database-driven as has been discussed on this list).
      >
      > If anyone else wants to pick up my work, my two git repos are here:
      > https://github.com/chrisdolan/pcgen-svn
      > https://github.com/chrisdolan/pcgen-android
      >
      > Many of my changes, especially in my "interned_strings" branch, would be useful for the desktop PCGen too. With those changes I achieved a speedup of 2x on my Mac before I quit, largely because of reduced garbage collection time.
      >
      > I would be very happy to answer any questions about my research. And if someone were to make a breakthrough, I'd be thrilled to jump back into the project.
      >
      > Chris
      >
    • James Dempsey
      Hi Chris, I ve been thinking about this project a bit more of late. This has been particularly around reducing the memory footprint of the program when
      Message 2 of 8 , May 5, 2013
      • 0 Attachment
        Hi Chris,

        I've been thinking about this project a bit more of late. This has been
        particularly around reducing the memory footprint of the program when
        operating in character displayer mode. I'm assuming that the memory
        impact is causing more slow down that the actual processing PCGen is
        trying to do.

        The main way of doing this would be to reduce the number of rules items
        loaded (and retained) by the system. Looking at the pathfinder for
        players dataset, it is really dominated by abilities and equipment. Of
        approx 5800 loaded rules items, 3078 are abilities and 2052 are equipment.

        Now equipment could easily be filtered as it is all listed in the pcg.
        You'd need to consider both the named equipment and the base equipment,
        but if you filtered for those you'd get down to maybe 20-100 items per
        character, which would be a significant saving.

        Filtering abilities would be a bit harder, but if the equipment showed a
        reduction in memory footprint, it could be worth the further investigation.

        Cheers,
        James.

        On 20/02/2013 1:13 PM Chris Dolan wrote
        > This will be my last update for a while: I'm shelving my Android port of PCGen for now. I got it to the point where it would load and render a PC. However, it's about a factor of 100 too slow for real-world use (7 minutes on my tablet, 35 minutes on my phone). I think it might be possible to achieve that factor of 100 with lots of caching and trimmed-down loading, but it's too much work for my hobby programming hours so I'm stopping. Realistically, the whole approach to LST parsing would need to be revisited (and perhaps database-driven as has been discussed on this list).
        >
        > If anyone else wants to pick up my work, my two git repos are here:
        > https://github.com/chrisdolan/pcgen-svn
        > https://github.com/chrisdolan/pcgen-android
        >
        > Many of my changes, especially in my "interned_strings" branch, would be useful for the desktop PCGen too. With those changes I achieved a speedup of 2x on my Mac before I quit, largely because of reduced garbage collection time.
        >
        > I would be very happy to answer any questions about my research. And if someone were to make a breakthrough, I'd be thrilled to jump back into the project.
        >
        > Chris
        >
        >
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
      • Chris Dolan
        Thanks for the thoughts, James. Yes, on Android, the majority of the egregious startup time is spent in the garbage collector. When I started, the major waste
        Message 3 of 8 , May 5, 2013
        • 0 Attachment
          Thanks for the thoughts, James.

          Yes, on Android, the majority of the egregious startup time is spent in the garbage collector. When I started, the major waste of memory was not the equipment per se, but in that we read the whole .lst files into RAM before tearing them apart into strings. Before my fixes, the whole .lst file syntax was leaked into the Java heap. My fix threw that away, but those MB of .lst data are still being allocated and thrown away, thus churning the GC.

          So, yes, reducing the amount of equipment/abilities would help linearly reduce the startup time - I can imagine that we could achieve an improvement of 5x with only minor/modest architectural changes. But I really needed 100x improvement to make the app feasible. (and furthermore, I wanted the whole equipment list and rule set eventually to be available in my app for tabletop modification) To reach that goal, I concluded that we'd need to replace .lst completely with a format that's readable without manual string parsing. I theorized that either XML (which already has a highly-optimized parser in Java) or a SQLite or other binary store would be necessary to reduce the GC churn.

          Thus I think your ideas are worth pursuing, but are likely not adequate for my goals. So I'm going to leave it tabled on my end.

          Chris

          On May 5, 2013, at 5:55 PM, James Dempsey <jdempsey@...> wrote:

          > Hi Chris,
          >
          > I've been thinking about this project a bit more of late. This has been
          > particularly around reducing the memory footprint of the program when
          > operating in character displayer mode. I'm assuming that the memory
          > impact is causing more slow down that the actual processing PCGen is
          > trying to do.
          >
          > The main way of doing this would be to reduce the number of rules items
          > loaded (and retained) by the system. Looking at the pathfinder for
          > players dataset, it is really dominated by abilities and equipment. Of
          > approx 5800 loaded rules items, 3078 are abilities and 2052 are equipment.
          >
          > Now equipment could easily be filtered as it is all listed in the pcg.
          > You'd need to consider both the named equipment and the base equipment,
          > but if you filtered for those you'd get down to maybe 20-100 items per
          > character, which would be a significant saving.
          >
          > Filtering abilities would be a bit harder, but if the equipment showed a
          > reduction in memory footprint, it could be worth the further investigation.
          >
          > Cheers,
          > James.
          >
          > On 20/02/2013 1:13 PM Chris Dolan wrote
          >> This will be my last update for a while: I'm shelving my Android port of PCGen for now. I got it to the point where it would load and render a PC. However, it's about a factor of 100 too slow for real-world use (7 minutes on my tablet, 35 minutes on my phone). I think it might be possible to achieve that factor of 100 with lots of caching and trimmed-down loading, but it's too much work for my hobby programming hours so I'm stopping. Realistically, the whole approach to LST parsing would need to be revisited (and perhaps database-driven as has been discussed on this list).
          >>
          >> If anyone else wants to pick up my work, my two git repos are here:
          >> https://github.com/chrisdolan/pcgen-svn
          >> https://github.com/chrisdolan/pcgen-android
          >>
          >> Many of my changes, especially in my "interned_strings" branch, would be useful for the desktop PCGen too. With those changes I achieved a speedup of 2x on my Mac before I quit, largely because of reduced garbage collection time.
          >>
          >> I would be very happy to answer any questions about my research. And if someone were to make a breakthrough, I'd be thrilled to jump back into the project.
          >>
          >> Chris
          >>
          >>
          >>
          >> ------------------------------------
          >>
          >> Yahoo! Groups Links
          >>
          >>
          >>
          >>
          >
          >
          >
          > ------------------------------------
          >
          > Yahoo! Groups Links
          >
          >
          >
        • Devon Jones
          My thought is that the solution on android likely means adding a shim and having the data load go into an sqlite3 database, but at that point you need to do
          Message 4 of 8 , May 5, 2013
          • 0 Attachment
            My thought is that the solution on android likely means adding a shim and having the data load go into an sqlite3 database, but at that point you need to do significant work on they system to make it all functional.

            Devon


            On Sun, May 5, 2013 at 5:58 PM, Chris Dolan <chris@...> wrote:
             

            Thanks for the thoughts, James.

            Yes, on Android, the majority of the egregious startup time is spent in the garbage collector. When I started, the major waste of memory was not the equipment per se, but in that we read the whole .lst files into RAM before tearing them apart into strings. Before my fixes, the whole .lst file syntax was leaked into the Java heap. My fix threw that away, but those MB of .lst data are still being allocated and thrown away, thus churning the GC.

            So, yes, reducing the amount of equipment/abilities would help linearly reduce the startup time - I can imagine that we could achieve an improvement of 5x with only minor/modest architectural changes. But I really needed 100x improvement to make the app feasible. (and furthermore, I wanted the whole equipment list and rule set eventually to be available in my app for tabletop modification) To reach that goal, I concluded that we'd need to replace .lst completely with a format that's readable without manual string parsing. I theorized that either XML (which already has a highly-optimized parser in Java) or a SQLite or other binary store would be necessary to reduce the GC churn.

            Thus I think your ideas are worth pursuing, but are likely not adequate for my goals. So I'm going to leave it tabled on my end.

            Chris



            On May 5, 2013, at 5:55 PM, James Dempsey <jdempsey@...> wrote:

            > Hi Chris,
            >
            > I've been thinking about this project a bit more of late. This has been
            > particularly around reducing the memory footprint of the program when
            > operating in character displayer mode. I'm assuming that the memory
            > impact is causing more slow down that the actual processing PCGen is
            > trying to do.
            >
            > The main way of doing this would be to reduce the number of rules items
            > loaded (and retained) by the system. Looking at the pathfinder for
            > players dataset, it is really dominated by abilities and equipment. Of
            > approx 5800 loaded rules items, 3078 are abilities and 2052 are equipment.
            >
            > Now equipment could easily be filtered as it is all listed in the pcg.
            > You'd need to consider both the named equipment and the base equipment,
            > but if you filtered for those you'd get down to maybe 20-100 items per
            > character, which would be a significant saving.
            >
            > Filtering abilities would be a bit harder, but if the equipment showed a
            > reduction in memory footprint, it could be worth the further investigation.
            >
            > Cheers,
            > James.
            >
            > On 20/02/2013 1:13 PM Chris Dolan wrote
            >> This will be my last update for a while: I'm shelving my Android port of PCGen for now. I got it to the point where it would load and render a PC. However, it's about a factor of 100 too slow for real-world use (7 minutes on my tablet, 35 minutes on my phone). I think it might be possible to achieve that factor of 100 with lots of caching and trimmed-down loading, but it's too much work for my hobby programming hours so I'm stopping. Realistically, the whole approach to LST parsing would need to be revisited (and perhaps database-driven as has been discussed on this list).
            >>
            >> If anyone else wants to pick up my work, my two git repos are here:
            >> https://github.com/chrisdolan/pcgen-svn
            >> https://github.com/chrisdolan/pcgen-android
            >>
            >> Many of my changes, especially in my "interned_strings" branch, would be useful for the desktop PCGen too. With those changes I achieved a speedup of 2x on my Mac before I quit, largely because of reduced garbage collection time.
            >>
            >> I would be very happy to answer any questions about my research. And if someone were to make a breakthrough, I'd be thrilled to jump back into the project.
            >>
            >> Chris
            >>
            >>
            >>
            >> ------------------------------------
            >>
            >> Yahoo! Groups Links
            >>
            >>
            >>
            >>
            >
            >
            >
            > ------------------------------------
            >
            > Yahoo! Groups Links
            >
            >
            >


          • Javier Ortiz
            I would suggest abstracting the storage means behind an API that can be implemented by the current approach and then easily replaced later (i.e. database).
            Message 5 of 8 , May 6, 2013
            • 0 Attachment
              I would suggest abstracting the storage means behind an API that can be implemented by the current approach and then easily replaced later (i.e. database). Still lots of work to do. I wish I had the time...


              On Sun, May 5, 2013 at 10:18 PM, Devon Jones <devon.jones@...> wrote:
               

              My thought is that the solution on android likely means adding a shim and having the data load go into an sqlite3 database, but at that point you need to do significant work on they system to make it all functional.

              Devon


              On Sun, May 5, 2013 at 5:58 PM, Chris Dolan <chris@...> wrote:
               

              Thanks for the thoughts, James.

              Yes, on Android, the majority of the egregious startup time is spent in the garbage collector. When I started, the major waste of memory was not the equipment per se, but in that we read the whole .lst files into RAM before tearing them apart into strings. Before my fixes, the whole .lst file syntax was leaked into the Java heap. My fix threw that away, but those MB of .lst data are still being allocated and thrown away, thus churning the GC.

              So, yes, reducing the amount of equipment/abilities would help linearly reduce the startup time - I can imagine that we could achieve an improvement of 5x with only minor/modest architectural changes. But I really needed 100x improvement to make the app feasible. (and furthermore, I wanted the whole equipment list and rule set eventually to be available in my app for tabletop modification) To reach that goal, I concluded that we'd need to replace .lst completely with a format that's readable without manual string parsing. I theorized that either XML (which already has a highly-optimized parser in Java) or a SQLite or other binary store would be necessary to reduce the GC churn.

              Thus I think your ideas are worth pursuing, but are likely not adequate for my goals. So I'm going to leave it tabled on my end.

              Chris



              On May 5, 2013, at 5:55 PM, James Dempsey <jdempsey@...> wrote:

              > Hi Chris,
              >
              > I've been thinking about this project a bit more of late. This has been
              > particularly around reducing the memory footprint of the program when
              > operating in character displayer mode. I'm assuming that the memory
              > impact is causing more slow down that the actual processing PCGen is
              > trying to do.
              >
              > The main way of doing this would be to reduce the number of rules items
              > loaded (and retained) by the system. Looking at the pathfinder for
              > players dataset, it is really dominated by abilities and equipment. Of
              > approx 5800 loaded rules items, 3078 are abilities and 2052 are equipment.
              >
              > Now equipment could easily be filtered as it is all listed in the pcg.
              > You'd need to consider both the named equipment and the base equipment,
              > but if you filtered for those you'd get down to maybe 20-100 items per
              > character, which would be a significant saving.
              >
              > Filtering abilities would be a bit harder, but if the equipment showed a
              > reduction in memory footprint, it could be worth the further investigation.
              >
              > Cheers,
              > James.
              >
              > On 20/02/2013 1:13 PM Chris Dolan wrote
              >> This will be my last update for a while: I'm shelving my Android port of PCGen for now. I got it to the point where it would load and render a PC. However, it's about a factor of 100 too slow for real-world use (7 minutes on my tablet, 35 minutes on my phone). I think it might be possible to achieve that factor of 100 with lots of caching and trimmed-down loading, but it's too much work for my hobby programming hours so I'm stopping. Realistically, the whole approach to LST parsing would need to be revisited (and perhaps database-driven as has been discussed on this list).
              >>
              >> If anyone else wants to pick up my work, my two git repos are here:
              >> https://github.com/chrisdolan/pcgen-svn
              >> https://github.com/chrisdolan/pcgen-android
              >>
              >> Many of my changes, especially in my "interned_strings" branch, would be useful for the desktop PCGen too. With those changes I achieved a speedup of 2x on my Mac before I quit, largely because of reduced garbage collection time.
              >>
              >> I would be very happy to answer any questions about my research. And if someone were to make a breakthrough, I'd be thrilled to jump back into the project.
              >>
              >> Chris
              >>
              >>
              >>
              >> ------------------------------------
              >>
              >> Yahoo! Groups Links
              >>
              >>
              >>
              >>
              >
              >
              >
              > ------------------------------------
              >
              > Yahoo! Groups Links
              >
              >
              >



            • thpr
              ... Chris, Do you have info on how much memory was taken up (on average) per object in Abilities and Equipment and how many there were of those object types in
              Message 6 of 8 , May 6, 2013
              • 0 Attachment
                --- In pcgen_developers@yahoogroups.com, Chris Dolan <chris@...> wrote:
                > So, yes, reducing the amount of equipment/abilities would help linearly reduce the startup time - I can imagine that we could achieve an improvement of 5x with only minor/modest architectural changes.

                Chris,

                Do you have info on how much memory was taken up (on average) per object in Abilities and Equipment and how many there were of those object types in the data you were loading?

                TP.
              • thpr
                You can ignore this - though through the consequences of an on demand load and do not believe that is practical. TP.
                Message 7 of 8 , May 6, 2013
                • 0 Attachment
                  You can ignore this - though through the consequences of an "on demand" load and do not believe that is practical.

                  TP.

                  --- In pcgen_developers@yahoogroups.com, "thpr" <thpr@...> wrote:
                  >
                  >
                  >
                  > --- In pcgen_developers@yahoogroups.com, Chris Dolan <chris@> wrote:
                  > > So, yes, reducing the amount of equipment/abilities would help linearly reduce the startup time - I can imagine that we could achieve an improvement of 5x with only minor/modest architectural changes.
                  >
                  > Chris,
                  >
                  > Do you have info on how much memory was taken up (on average) per object in Abilities and Equipment and how many there were of those object types in the data you were loading?
                  >
                  > TP.
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.