Loading ...
Sorry, an error occurred while loading the content.
 

[ARCH] Rules Persistence System Information & Projects

Expand Messages
  • Tom Parker
    At this point, I believe I ve asserted and provided a bit of background for a few things: - A quick introduction to the Prerequisite/Requirement problem (I
    Message 1 of 5 , Jun 19, 2009
      At this point, I believe I've asserted and provided a bit of background for a few things:
      - A quick introduction to the Prerequisite/Requirement problem (I will often refer to this as the PRE/REQ issue), this needs more explanation as there are some additional nuances here
      - Separation of static and dynamic information to avoid cloning (and all of the side effects of that cloning to gain performance benefit)
      - Graph structure and reconvergence (to be covered a bit more here)
      - Separating knowledge of the persistent data format (LST) from the internal memory format

      I would note on this last item that the Wiki already has a decent introduction to the Rules Persistence System. I unfortunately referred to the "Token/Loader System" in my initial note, which is depicted in the diagram as the Rules Persistence System. While there is already documentation on the Wiki for this system:
      http://www.pcgen-test.org/wiki/index.php?title=Rules_Persistence_System and http://www.pcgen-test.org/wiki/index.php?title=Load_Commit_Subsystem

      I think its good to cover a few key highlights here and provide a few potential projects.

      One of the primary goals is to _Catch Errors Early_. The intent should be to catch any situation which is guaranteed to be a problem, and someday to identify all situations which might be a problem and identify those with a warning or informational message. This is greatly preferable to encountering errors at runtime. The Rules Persistence System is designed around this goal.

      One interesting result of this is that mis-associations in data - such as attempting to load a Feat called Dodeg (note the misspelling) - should be caught in some way. We also want to separate the LST syntax from the internal memory structure, so all references to objects are identified and subsequently resolved during the loading process. This catches errors and guarantees uniqueness (to avoid ambiguity). If you see "Unconstructed Reference" errors it may be due to a misspelling (the message is saying you referred to something that was never constructed)

      Another point in the Rules Persistence System (and particularly in the Token/Loader System) is flexibility. We have a few tokens that are rather complex (such as CHOOSE) which have a series of subtokens. We want each of these subtokens to be separate plugins, so that we can easily add, modify and test! those input systems. There is therefore a standardized system for loading subtokens that is used across several different tokens.

      (One note here: You may find people referring to both tokens and tags. It is highly unlikely that I keep a distinction between the two terms, and it is likely I will use them interchangeably. This is unfortunate, since it makes it harder for people to pick things up, but there is enough history with both that I don't think we have a consistent definition, and even if we do, the distinction is very subtle)

      The other major thing I would note is that only valid Tokens may impact the Rules Data Store. This was a later design decision and helped to decrease the complexity of the original tokens. If the token writer (programmer) has to queue up changes to the internal structure in order to ensure the input is valid, that makes for lots of temporary data structures and harder code to understand. By absolving the programmer of that responsibility (simply returning false will throw out any changes initiated by the token), the tokens are much easier to understand. This is simply a "unit of work" design pattern applied to the loading of our data files.

      There is a lot of work to do in the Rules Persistence System, and some of this would be great work for a new programmer to PCGen:

      (1) There are a seies of line-based files that are capable of unparse, but the overall wrapper system in the converter is not capable of unparsing a Kit file.
      Specifically, if one looks at the converter, you can see code in pcgen.gui.converter.LstConverter.java:
      loaders.add(new CopyLoader(ListKey.LICENSE_FILE));
      loaders.add(new CopyLoader(ListKey.FILE_KIT));
      loaders.add(new CopyLoader(ListKey.FILE_BIO_SET));
      loaders.add(new CopyLoader(ListKey.FILE_PCC));
      All 5 of these files types are just copied without PCGen parsing the contents. Longer term this limits our ability to convert this information. While this isn't a problem for the license files (that should stay copy), the Kit and BioSet files are particular concerns for me in things we may want to convert in the next version or two. This involves developing a coverter system that is smart enough to queue a group of lines together for these file types before writing them back out.

      (2) In the same pcgen.gui.converter.LstConverter.java file, there is also:
      loaders.add(new CopyLoader(ListKey.FILE_ABILITY_CATEGORY));
      This has a conversion problem as well, but Ability Category is token based. The problem is that the ability category tokens (in plugin.lsttokens.gamemode.abilitycategory) are "old style" tokens that only have a parse and getTokenName method. This contrasts with "new style" tokens that have an unparse (and a few other) methods. A good project here is to do two things: Convert pcgen.persistence.lst.AbilityCategoryLoader.java to handle "new style tokens" and then to convert the tokens that implement AbilityCategoryLstToken into tokens that implement CDOMPrimaryToken<AbilityCategory>. For a Loader that bridges the old (map) and new (context.parse) methods, you can look at the 5.16 version of PCStatLoader.java ( http://pcgen.svn.sourceforge.net/viewvc/pcgen/branches/5.16.x/pcgen/code/src/java/pcgen/persistence/lst/PCStatLoader.java?revision=9635&view=markup ) and compare that to the 5.14 version (which will have only the old style) or the Trunk version (which has only the new style). Using this bridge method, only one token has to be converted at a time, so it can be gradual and a lot less pressure on the developer.

      (3) Cross-token validity is not evaluated. In particular, structures like this:
      MyObject <tab> TOKEN:Foo <tab> TOKEN:.CLEAR
      ...are not currently detected by PCGen. This is a weakness of the existing Token/Loader system. A reasonably simple solution to this problem exists. Currently the Load system has the tokens feed a single Tracking strategy which commits to a final Tracking (Editor) or Consolidated (Runtime) strategy. The solution to this problem is to have a Token-Tracking Strategy commit to a Line-Tracking Strategy, which commits to a final Tracking (Editor) or Consolidated (Runtime) strategy. The only additional code required is a set of detection methods for the (Line-)Tracking strategy to identify when non-sensical situations (.CLEAR after addition) occur.

      (4) Currently, the StringKey clearing system is not handled well. It is possible (likely?) that the globalClearSet map should be expanded to hold ANY of the *Key objects used in CDOMObject, including IntegerKey, StringKey, etc. rather than just ListKey.

      (5) There are a bunch of tokens where some simple additional features would be helpful. Some of these are easy to get started on, and there are quite a few of them. One example is support for .CLEAR for the TARGETAREA token (FREQ 2413630). This is an isolated change that requires changing only one class, and there are other tokens with the features to show you what needs to be changed - so another good startup experience here.

      If anyone is interested in any of those, I'm happy to help people get started. The Rules Persistence System is some of the newer code, and (hopefully at least) somewhat cleaner code, so this is probably a good place for new folks to get started.

      Those are bigger projects I can easily identify. There are also cleanup projects (reducing code warnings, refactoring support) that also exist. I'm not highlighting those, as they are likely to be less interesting. If some of the new devs are just getting started with Java, those may be better options for getting familiar with the language and with the project, so I'm happy to point projects like that out as well (as can James, the code lead).
    • Tom Parker
      One additional note I should add: The Rules Data Store as it is being built in the new structure is avoiding as much as possible global information. The
      Message 2 of 5 , Jun 19, 2009
        One additional note I should add:

        The Rules Data Store as it is being built in the new structure is avoiding as much as possible global information. The information is kept to a single "Load Context".

        While we are still bound by pcgen.core.Globals and by the GameMode system being static and making static assumptions, the goal is to be able to load multiple game modes at the same time.

        Thus, the only limitation on how many "modes" could be loaded at once should be memory (and the CDOM internal workings are designed to be a lot less memory hungry than the current design)

        TP.
      • Martijn Verburg
        Hi Tom, ... Do we have a FREQs for these 5 already (The last one partially yes)? I had a quick scan through but couldn t find them. I m happy to make
        Message 3 of 5 , Jul 21, 2009
          Hi Tom,

          <snip>

          > There is a lot of work to do in the Rules Persistence System, and
          > some of this would be great work for a new programmer to PCGen:
          >
          > (1) There are a series of line-based files that are capable of
          > unparse, but the overall wrapper system in the converter is not
          > capable of unparsing a Kit file.
          > Specifically, if one looks at the converter, you can see code in
          > pcgen.gui.converter.LstConverter.java:
          > loaders.add(new CopyLoader(ListKey.LICENSE_FILE));
          > loaders.add(new CopyLoader(ListKey.FILE_KIT));
          > loaders.add(new CopyLoader(ListKey.FILE_BIO_SET));
          > loaders.add(new CopyLoader(ListKey.FILE_PCC));
          > All 5 of these files types are just copied without PCGen parsing
          > the contents. Longer term this limits our ability to convert this
          > information. While this isn't a problem for the license files
          > (that should stay copy), the Kit and BioSet files are particular
          > concerns for me in things we may want to convert in the next
          > version or two. This involves developing a coverter system that is
          > smart enough to queue a group of lines together for these file
          > types before writing them back out.
          >
          > (2) In the same pcgen.gui.converter.LstConverter.java file, there
          > is also:
          > loaders.add(new CopyLoader(ListKey.FILE_ABILITY_CATEGORY));
          > This has a conversion problem as well, but Ability Category is
          > token based. The problem is that the ability category tokens (in
          > plugin.lsttokens.gamemode.abilitycategory) are "old style" tokens
          > that only have a parse and getTokenName method. This contrasts
          > with "new style" tokens that have an unparse (and a few other)
          > methods. A good project here is to do two things: Convert
          > pcgen.persistence.lst.AbilityCategoryLoader.java to handle "new
          > style tokens" and then to convert the tokens that implement
          > AbilityCategoryLstToken into tokens that implement
          > CDOMPrimaryToken<AbilityCategory>. For a Loader that bridges the
          > old (map) and new (context.parse) methods, you can look at the 5.16
          > version of PCStatLoader.java ( http://pcgen.svn.sourceforge.net/viewvc/pcgen/branches/5.16.x/pcgen/code/src/java/pcgen/persistence/lst/PCStatLoader.java?revision=9635&view=markup ) and compare that
          > to the 5.14 version (which will have only the old style) or the
          > Trunk version (which has only the new style). Using this bridge
          > method, only one token has to be converted at a time, so it can be
          > gradual and a lot less pressure on the developer.
          >
          > (3) Cross-token validity is not evaluated. In particular,
          > structures like this:
          > MyObject <tab> TOKEN:Foo <tab> TOKEN:.CLEAR
          > ...are not currently detected by PCGen. This is a weakness of the
          > existing Token/Loader system. A reasonably simple solution to this
          > problem exists. Currently the Load system has the tokens feed a
          > single Tracking strategy which commits to a final Tracking (Editor)
          > or Consolidated (Runtime) strategy. The solution to this problem is
          > to have a Token-Tracking Strategy commit to a Line-Tracking
          > Strategy, which commits to a final Tracking (Editor) or
          > Consolidated (Runtime) strategy. The only additional code required
          > is a set of detection methods for the (Line-)Tracking strategy to
          > identify when non-sensical situations (.CLEAR after addition)
          > occur.
          >
          > (4) Currently, the StringKey clearing system is not handled well.
          > It is possible (likely?) that the globalClearSet map should be
          > expanded to hold ANY of the *Key objects used in CDOMObject,
          > including IntegerKey, StringKey, etc. rather than just ListKey.
          >
          > (5) There are a bunch of tokens where some simple additional
          > features would be helpful. Some of these are easy to get started
          > on, and there are quite a few of them. One example is support for
          > .CLEAR for the TARGETAREA token (FREQ 2413630). This is an
          > isolated change that requires changing only one class, and there
          > are other tokens with the features to show you what needs to be
          > changed - so another good startup experience here.

          Do we have a FREQs for these 5 already (The last one partially yes)? I had a quick scan through but couldn't find them. I'm happy to make them if required.

          > If anyone is interested in any of those, I'm happy to help people
          > get started. The Rules Persistence System is some of the newer
          > code, and (hopefully at least) somewhat cleaner code, so this is
          > probably a good place for new folks to get started.
          >
          > Those are bigger projects I can easily identify. There are also
          > cleanup projects (reducing code warnings, refactoring support) that
          > also exist. I'm not highlighting those, as they are likely to be
          > less interesting. If some of the new devs are just getting started
          > with Java, those may be better options for getting familiar with
          > the language and with the project, so I'm happy to point projects
          > like that out as well (as can James, the code lead).

          So developers!! Do we have any volunteers for some of these items? As Tom points out there are some really small ones that can help get you familiar with Java itself and/or the PCGen code base.

          We currently have 38 members on this list and while I know not all are Java developers I'm sure we've got a few new ones out there, please don't hesitate to get in touch!

          We can definitely even meet you on IRC to help walk you through some code changes.

          K
        • Tom Parker
          ... The last one is really a handful (or more) of FREQs, all of which exist. If someone is interested, I m happy to read through and find them all. ... I
          Message 4 of 5 , Jul 21, 2009
            --- In pcgen_developers@yahoogroups.com, "Martijn Verburg" <martijnverburg@...> wrote:
            > Do we have a FREQs for these 5 already (The last one partially yes)?

            The last one is really a handful (or more) of FREQs, all of which exist. If someone is interested, I'm happy to read through and find them all.

            > I had a quick scan through but couldn't find them. I'm happy to make them if required.

            I don't think the other 4 are existing FREQs.

            TP.
          • Martijn Verburg
            Hi all, ... Entered as 2831547 -- 2831550 K
            Message 5 of 5 , Aug 3, 2009
              Hi all,

              > > I had a quick scan through but couldn't find them. I'm happy to
              > > make them if required.
              >
              > I don't think the other 4 are existing FREQs.

              Entered as 2831547 --> 2831550

              K
            Your message has been successfully submitted and would be delivered to recipients shortly.