Loading ...
Sorry, an error occurred while loading the content.

Output Tokens - Topic for Code Team Meeting Nov 13/14

Expand Messages
  • thpr
    During the last code team meeting, I mentioned that I thought we should discuss output sheets and output tokens at the next meeting. This is a set of thoughts
    Message 1 of 5 , Nov 6, 2009
    View Source
    • 0 Attachment
      During the last code team meeting, I mentioned that I thought we should discuss output sheets and output tokens at the next meeting.

      This is a set of thoughts that I have developed to spur discussion on this topic in preparation for the next code team meeting.

      The general presumption here is that the output token system will remain relative unchanged from the perspective of what appears in the output sheets. (e.g. "TEMPLATE.X.NAME" will remain "TEMPLATE.X.NAME").

      From my understanding, and based on discussions with James, this assumption holds even if we move to a templating engine, e.g. Apache Velocity. If anyone has reason to believe that is not the case, I'd appreciate knowing :)

      In my opinion there are a few challenges within the output tokens system that I would like to address.

      First, we are using a very large IF statement structure in order to do processing of the tokens. This can be a performance hit due to the size of the structure, and is better done as a Map lookup. I'd also like to have the "secondary" tokens be separate for flexibility (similar to how we have the subtokens of CHOOSE, ADD, AUTO, etc. as separate token files.

      Second, we are forced to have some code internal to PCGen (in pcgen.*) while some export tokens appear in the plugin system (where they all should be IMHO). It would be nice to get to a system that allows all export tokens to be plugins.

      Third, the export tokens are not leveraging existing code. In some cases, we have a number of sets of duplicate code in the UI vs. export tokens (or in multiple export tokens). These should use consistent processing in order to reduce duplicate code and risk of having bugs fixed in only one place.

      I also believe there are opportunities for further flexibility in the output system, similar to what is underway for CHOOSE. Specifically with CHOOSE, if a new primitive for Equipment is developed (e.g. WEAPONHANDS=n), that new primitive would be available in both the EQUIPMENT qualifier of CHOOSE:WEAPONPROF as well as as a primitive within CHOOSE:EQUIPMENT (and anywhere else an Equipment primitive could be used). This reuse/flexibility reduces the amount of work necessary to add new function.

      With that vision, here are some ideas around output token structure from a code perspective.

      Let me continue to use "TEMPLATE.X.NAME" as an initial example.

      For clarity in the description:
      I use "object" to refer to CDOMObjects (e.g. Templates and Races)
      I use "item" to refer to the period-separated items in the output token. The above example as 3 items. First, "TEMPLATE" defines a starting list. "X" then selects a single item from the list, and "NAME" fetches a characteristic of the item. (This presumes that output token is contained within an IF output token that is incrementing X from 0 to N)

      The first point is to look at the list of items in an output token as a pipeline of items that will pass output to each other (and eventually return a String)

      The possible starting lists are pretty obvious, and mainly line up with the basic object types: ABILITY, WEAPONPROF, LANGUAGE, TEMPLATE, CLASS, SKILL, DOMAIN, CHECK, etc.
      Some tokens may not start with a list, but a single object, e.g. ALIGNMENT, RACE, DEITY, etc.

      "X" (I use this generally to refer to a number or an object's KEY) can then reduce to an object from that list (in order, starting at 0, or direct by KEY).

      There are then "result" items that can convert from a single object into a characteristic of that object, such as NAME, DESCRIPTION, SOURCE, etc.

      One would note two other types of item that can appear in the pipeline:

      Filters can reduce the number of objects. For simplicity, I'll make up an example (NOT valid in current OS)
      CLASS.SPELLCASTER.X.NAME would be using a filter called "SPELLCASTER" that can be applied to lists of CLASS objects.

      Expanders can take a single object and expand that to a list of other objects. For example (also made up, NOT VALID in current OS):
      DEITY.DOMAINS.X.NAME would be expanding the "DEITY" of the PC to get the DOMAINS of that Deity.

      So in summary, we have 6 different possible situations, with their restrictions:
      Starting List (e.g. TEMPLATE) ... MUST appear as the first item in a token, returns a list
      Starting Object (e.g. RACE) .... MUST appear as the first item in a token, returns an object
      Filter (e.g. SPELLCASTER) ... MUST not appear as the first item, MUST start with a list, and returns a list
      Expander (e.g. DOMAIN) ... MUST not appear as the first item, MUST start with a single object and returns a list
      Reducer (e.g. X) ... MUST not appear as the first item, MUST start with a list and returns an object
      Characteristic (e.g. NAME) ... MUST not appear as the first item, MUST start with a single object and returns a String.

      As a first pass:

      Starting List/Starting Object tokens implement PrimaryExportToken, an interface with one method:
      String process(ExportController controller);

      In order to "transfer control" to the secondary tokens, we put methods on the ExportController:
      <T extends CDOMObject> T processList(List<T> list);
      <T extends CDOMObject> String processItem(T t);

      processList performs 2 functions:
      (1) Looks at the next item in the chain to determine if it is a number. If so, returns object X from the list.
      (2) Looks at the next item in the chain to determine if it is a filter. If so, runs the filter.

      Filters have to implement ExportFilter<T extends CDOMObject>:
      List<T> processList(List<T> list);

      Note that with processList having to do these two things (and with the only way to reduce from a list to a single object through the use of a number or key), it can simply loop through Filters as it sees them and then use the # or KEY to reduce to a single object that can be returned.

      Expanders and Characteristics both implement SecondaryExportToken<T extends CDOMObject>
      String processItem(ExportController ec, T t);

      A Characteristic token only needs t, and can simply call appropriate methods on T to return the (String) result.

      An Expander can use the included ExportController. Thinking again about DEITY.DOMAINS.X.NAME as an example, DOMAINS is a SecondaryExportToken<Deity>, which calls something like getDomains() to get a Domain list. This list of Domains can then be passes to the processList method of the ExportController.

      The ExportController is responsible for knowing the "next" item in the original token that needs to be processed. By passing the ExportController around, this accomplishes the processing of an output token while keeping the different filters/changes as very "atomic" sets of code (which should be easily testable independent of a full-fledged output system).

      Things I don't like or aren't done:

      I've typed this up before really finishing, and this description has an architectural flaw: The ExportController knows about the SecondaryExportToken interface, and the SecondaryExportToken interface knows about the ExportController. That circular reference is not good. Thus, while the passing of responsibility between items in this way seems to meet the requirements of the processing we do in export tokens, this *exact* class structure is not ideal, and we should find another way to do it. (One simple method would be to have a separate interface for Expanders; just pass in the PlayerCharacter/CharID rather than the ExportController and return a List<C> that is then passed into processList)

      This presumes the ExportController knows the active PC. That's probably true, but impacts of that need to be considered.

      This also presumes ExportController is a good name, which it may have been in my first rendition, but its responsibilities have changed over a few plane flights, so it probably needs a new name.

      We have some information that is "game mode" info, so it works differently. Height Unit, for example, is probably a PrimaryExportToken that works differently than described above.

      We have some stateful tokens (e.g. MANUALWHITESPACE) that we need to consider if there are reasons they don't work in this architecture.

      This does not address - at least not better than the current system - the necessary caching due to incrementing across lists. This structure would still require the tokens to cache by themselves (and detect a different PC or different sequence # on a PC). Architectures that have the ability to hold that list state and increment across it would be very cool. I think it would require some more behavior from ExportController (to know about multiple IF statements that have been encountered), but haven't completely thought through how to do it.
    • karianna03
      Hi Tom/all, I may not be able to make the code team meeting, so my thoughts here. ... Sounds good. ... OK ... _Yes_, I d love to see the export code all
      Message 2 of 5 , Nov 10, 2009
      View Source
      • 0 Attachment
        Hi Tom/all,

        I may not be able to make the code team meeting, so my thoughts here.

        <snip>

        > The general presumption here is that the output token system will
        > remain relative unchanged from the perspective of what appears in
        > the output sheets. (e.g. "TEMPLATE.X.NAME" will remain
        > "TEMPLATE.X.NAME").
        >
        > From my understanding, and based on discussions with James, this
        > assumption holds even if we move to a templating engine, e.g. Apache
        > Velocity. If anyone has reason to believe that is not the case, I'd
        > appreciate knowing :)

        Sounds good.

        > In my opinion there are a few challenges within the output tokens
        > system that I would like to address.
        >
        > First, we are using a very large IF statement structure in order to
        > do processing of the tokens. This can be a performance hit due to
        > the size of the structure, and is better done as a Map lookup. I'd
        > also like to have the "secondary" tokens be separate for flexibility
        > (similar to how we have the subtokens of CHOOSE, ADD, AUTO, etc. as
        > separate token files.

        OK

        > Second, we are forced to have some code internal to PCGen (in
        > pcgen.*) while some export tokens appear in the plugin system (where
        > they all should be IMHO). It would be nice to get to a system that
        > allows all export tokens to be plugins.

        _Yes_, I'd love to see the export code all move to plugin export tokens.

        > Third, the export tokens are not leveraging existing code. In some
        > cases, we have a number of sets of duplicate code in the UI vs.
        > export tokens (or in multiple export tokens). These should use
        > consistent processing in order to reduce duplicate code and risk of
        > having bugs fixed in only one place.

        Again correct, I have seen many examples of this in my initial scan of the OS tokens. There is a static code analysis code report we use in our maven build which can highlight actual duplicate code to assist in this.

        > I also believe there are opportunities for further flexibility in
        > the output system, similar to what is underway for CHOOSE.
        > Specifically with CHOOSE, if a new primitive for Equipment is
        > developed (e.g. WEAPONHANDS=n), that new primitive would be
        > available in both the EQUIPMENT qualifier of CHOOSE:WEAPONPROF as
        > well as as a primitive within CHOOSE:EQUIPMENT (and anywhere else an
        > Equipment primitive could be used). This reuse/flexibility reduces
        > the amount of work necessary to add new function.

        OK

        <snip>

        > So in summary, we have 6 different possible situations, with their
        > restrictions:
        > Starting List (e.g. TEMPLATE) ... MUST appear as the first item
        > in a token, returns a list
        > Starting Object (e.g. RACE) .... MUST appear as the first item
        > in a token, returns an object
        > Filter (e.g. SPELLCASTER) ... MUST not appear as the first item,
        > MUST start with a list, and returns a list
        > Expander (e.g. DOMAIN) ... MUST not appear as the first item,
        > MUST start with a single object and returns a list
        > Reducer (e.g. X) ... MUST not appear as the first item, MUST
        > start with a list and returns an object
        > Characteristic (e.g. NAME) ... MUST not appear as the first item,
        > MUST start with a single object and returns a String.
        >
        > As a first pass:
        >
        > Starting List/Starting Object tokens implement PrimaryExportToken,
        > an interface with one method:
        > String process(ExportController controller);
        >
        > In order to "transfer control" to the secondary tokens, we put
        > methods on the ExportController:
        > <T extends CDOMObject> T processList(List<T> list);
        > <T extends CDOMObject> String processItem(T t);
        >
        > processList performs 2 functions:
        > (1) Looks at the next item in the chain to determine if it is a
        > number. If so, returns object X from the list.
        > (2) Looks at the next item in the chain to determine if it is a
        > filter. If so, runs the filter.
        >
        > Filters have to implement ExportFilter<T extends CDOMObject>:
        > List<T> processList(List<T> list);
        >
        > Note that with processList having to do these two things (and with
        > the only way to reduce from a list to a single object through the
        > use of a number or key), it can simply loop through Filters as it
        > sees them and then use the # or KEY to reduce to a single object
        > that can be returned.
        >
        > Expanders and Characteristics both implement SecondaryExportToken<T
        > extends CDOMObject> String processItem(ExportController ec, T t);
        >
        > A Characteristic token only needs t, and can simply call appropriate
        > methods on T to return the (String) result.
        >
        > An Expander can use the included ExportController. Thinking again
        > about DEITY.DOMAINS.X.NAME as an example, DOMAINS is a
        > SecondaryExportToken<Deity>, which calls something like getDomains()
        > to get a Domain list. This list of Domains can then be passes to the
        > processList method of the ExportController.
        >
        > The ExportController is responsible for knowing the "next" item in
        > the original token that needs to be processed. By passing the
        > ExportController around, this accomplishes the processing of an
        > output token while keeping the different filters/changes as very
        > "atomic" sets of code (which should be easily testable independent
        > of a full-fledged output system).
        >
        > Things I don't like or aren't done:
        >
        > I've typed this up before really finishing, and this description has
        > an architectural flaw: The ExportController knows about the
        > SecondaryExportToken interface, and the SecondaryExportToken
        > interface knows about the ExportController. That circular reference
        > is not good. Thus, while the passing of responsibility between
        > items in this way seems to meet the requirements of the processing
        > we do in export tokens, this *exact* class structure is not ideal,
        > and we should find another way to do it. (One simple method would
        > be to have a separate interface for Expanders; just pass in the
        > PlayerCharacter/CharID rather than the ExportController and return a
        > List<C> that is then passed into processList)

        Yeah the circular reference thing is causing us pain at the moment with the passing in of the ExportHandler to some tokens. So it would be good to avoid that sort of behaviour in the future.

        > This presumes the ExportController knows the active PC. That's
        > probably true, but impacts of that need to be considered.
        >
        > This also presumes ExportController is a good name, which it may
        > have been in my first rendition, but its responsibilities have
        > changed over a few plane flights, so it probably needs a new name.
        >
        > We have some information that is "game mode" info, so it works
        > differently. Height Unit, for example, is probably a
        > PrimaryExportToken that works differently than described above.
        >
        > We have some stateful tokens (e.g. MANUALWHITESPACE) that we need to
        > consider if there are reasons they don't work in this architecture.

        With a new templating engine the use of MANUALWHITESPACE and similar tokens may become redundant.

        > This does not address - at least not better than the current system
        > - the necessary caching due to incrementing across lists. This
        > structure would still require the tokens to cache by themselves (and
        > detect a different PC or different sequence # on a PC).
        > Architectures that have the ability to hold that list state and
        > increment across it would be very cool. I think it would require
        > some more behavior from ExportController (to know about multiple IF
        > statements that have been encountered), but haven't completely
        > thought through how to do it.

        OK

        A good beginning Tom, it's certainly not the easiest thing in the world to try and condense down the OS tokens into types of behaviour, considering they're even more wild west than the original LST tokens :)

        K
      • thpr
        ... I want to make sure I m following. Are you referring to re-entrant processing (the discussion we had on caching with AbilityToken) or code cycles/code
        Message 3 of 5 , Nov 10, 2009
        View Source
        • 0 Attachment
          --- In pcgen_developers@yahoogroups.com, "karianna03" <martijnverburg@...> wrote:
          > Yeah the circular reference thing is causing us pain at the moment with the passing in of the ExportHandler to some tokens. So it would be good to avoid that sort of behaviour in the future.

          I want to make sure I'm following. Are you referring to re-entrant processing (the discussion we had on caching with AbilityToken) or code cycles/code tangles (because ExportHandler specifically instantiates the tokens in pcgen.io.export)?

          TP.
        • karianna03
          Hi Tom, ... The code tangles bother me more, I understand the need for the re-entrant processing. K
          Message 4 of 5 , Nov 11, 2009
          View Source
          • 0 Attachment
            Hi Tom,

            > --- In pcgen_developers@yahoogroups.com, "karianna03"
            > <martijnverburg@> wrote:
            > > Yeah the circular reference thing is causing us pain at the moment
            > > with the passing in of the ExportHandler to some tokens. So it
            > > would be good to avoid that sort of behaviour in the future.
            >
            > I want to make sure I'm following. Are you referring to re-entrant
            > processing (the discussion we had on caching with AbilityToken) or
            > code cycles/code tangles (because ExportHandler specifically
            > instantiates the tokens in pcgen.io.export)?

            The code tangles bother me more, I understand the need for the re-entrant processing.

            K
          • Tom Parker
            Good - just wanted to double check we had the same concern TP. -- Tom Parker thpr@yahoo.com and tppublic@comcast.net ... From: karianna03
            Message 5 of 5 , Nov 11, 2009
            View Source
            • 0 Attachment
              Good - just wanted to double check we had the same concern

              TP.
              --
              Tom Parker
              thpr@... and tppublic@...

              --- On Wed, 11/11/09, karianna03 <martijnverburg@...> wrote:

              From: karianna03 <martijnverburg@...>
              Subject: [pcgen_developers] Re: Output Tokens - Topic for Code Team Meeting Nov 13/14
              To: pcgen_developers@yahoogroups.com
              Date: Wednesday, November 11, 2009, 4:14 AM

              Hi Tom,

              > --- In pcgen_developers@yahoogroups.com, "karianna03"
              > <martijnverburg@> wrote:
              > > Yeah the circular reference thing is causing us pain at the moment
              > > with the passing in of the ExportHandler to some tokens.  So it
              > > would be good to avoid that sort of behaviour in the future.
              >
              > I want to make sure I'm following.  Are you referring to re-entrant
              > processing (the discussion we had on caching with AbilityToken) or
              > code cycles/code tangles (because ExportHandler specifically
              > instantiates the tokens in pcgen.io.export)?

              The code tangles bother me more, I understand the need for the re-entrant processing.

              K



              ------------------------------------

              Yahoo! Groups Links

              <*> To visit your group on the web, go to:
                  http://groups.yahoo.com/group/pcgen_developers/

              <*> Your email settings:
                  Individual Email | Traditional

              <*> To change settings online go to:
                  http://groups.yahoo.com/group/pcgen_developers/join
                  (Yahoo! ID required)

              <*> To change settings via email:
                  pcgen_developers-digest@yahoogroups.com
                  pcgen_developers-fullfeatured@yahoogroups.com

              <*> To unsubscribe from this group, send an email to:
                  pcgen_developers-unsubscribe@yahoogroups.com

              <*> Your use of Yahoo! Groups is subject to:
                  http://docs.yahoo.com/info/terms/


            Your message has been successfully submitted and would be delivered to recipients shortly.