Loading ...
Sorry, an error occurred while loading the content.

Re: [pcgen_developers] Re: [PCGD] Code Meeting April 2009 - Log

Expand Messages
  • Andrew Wilson
    ... Yeah, I see those as separate formulae. What I really hate is the way we seem to need multiple calls to and fro between JEP and the old parser. ... I see
    Message 1 of 15 , May 2, 2009
    • 0 Attachment
      2009/4/30 Tom Parker <thpr@...>:
      > I would add:
      > (d) and complain at LST load (vs at runtime) if a formula is invalid (you
      > had addressed this later on in your note), which means it also has the
      > ability to detect and warn at LST load for deprecated formula styles, e.g.
      > 2MAXINT
      > and
      > (e) Consider whether we have a formula factory that can detect situations
      > where a formula like 1+INT is used multiple places in the LST.  These should
      > be reduced via a cache to a single Formula object (since a Formula is
      > immutable) to save memory and hopefully reduce LST load time.
      >
      > I'd also note that (c) is difficult, since some of that recursiveness is
      > necessary based on our BONUS structure.  So while I agree that's part of the
      > vision, whether it's part of scope in 5.17 might need to be up for
      > discussion (I defer to you on evaluating/defining that scope)

      Yeah, I see those as separate formulae. What I really hate is the way we
      seem to need multiple calls to and fro between JEP and the old parser.

      > To be honest this cycle I think I'd only have three new term objects one
      > that
      > know how to look up a LST variable, one that knows how to evaluate an OS
      > token and one that can process a JEP function.
      >
      > For clarity - how would:
      > 1+MYVAR
      >
      > be evaluated with the 3 terms you describe above?  Specifically, is the goal
      > for 5.17 still to have JEP doing the evaluation of the +?

      I see that as two terms the 1 is a constant that evaluates to itself, the MYVAR
      would use the variable lookup term. Then the formula evaluator would
      walk the tree,
      evaluating and combining as it goes.

      > I think we can eliminate
      > the multiple recursive calls to the old parser and JEP on each formula
      > evaluation.
      >
      > Good, and I agree this is the right path.  Just make sure we have some unit
      > tests for dependent items, e.g.:
      > DEFINE:MYVAR|5
      > BONUS:VAR|MYVAR|3
      > DEFINE:OTHERVAR|MYVAR+2
      >
      > (there are some internal variables that are even worse)

      noted.

      > The overall formula would be parsed once.
      >
      > This is how I noticed the SCORE replacement (specifically the String
      > replacement in StatList's getModForNumber(int, int) for those who want to
      > look) which makes me wonder how often we can only parse once vs. the formula
      > being string modified before it goes into the evaluator.  I know equipment
      > (and equipment modifiers especially) are ugly for String replacement right
      > before formula evaluation.
      >
      > I like your term system, but as I've gone through the core, there are places
      > we need to expand what is a term vs. what is string replacement.  Not to be
      > addressed as part of this conversation, but awareness to folks that faster
      > runtime performance on formulas doesn't solve all of our problems in one
      > shot.  Also need to be conscious with our own parser that we don't change
      > the rules to the point where the replacement would break.
      >
      > If we get that far we can eliminate all of JEP, except the bits we added to
      > it.
      >
      > This is where I get lost and what triggered the question on + above.

      When I formulated my original ideas I was planning to use JEP to
      do the parsing and evaluating. The caveat was that the formula
      was going to need heavily preprocessed before handing to JEP.
      I was planning to take something like:

      1+CL=monkey (code)/(CLASS.0.LEVEL*MYVAR)

      and turn that into

      1+VAR001/(VAR002*VAR003)

      then when JEP processed it, it'd find 3 variables

      VAR001 => CL=monkey (code)
      VAR002 => CLASS.0.LEVEL
      VAR003 => MYVAR

      I was then planning to hijack JEP's symbol lookup so that when it looked for a
      value for for VAR001 it found a term object that knows how to evaluate CL=
      VAR002 is a term object that can look up an output tag and VAR003 a term
      object that looks up a LST variable.

      Now I'm not so sure that's the best option if we are planning to ditch
      JEP anyway, then I don't need to bolt my system onto both sides of it.
      I think I'd like to look into whether we can carve up the current JEP
      system and maybe only pinch bits of the parser and get it to build
      a tree of term objects. Using bits of JEP will probably mean we can
      integrate it's function handling. One of the major obstacles is the
      use of things that should be reserved characters but aren't, () are
      nasty. classlevel("code-monkey") means you can't just split on
      operators, 20/(5*classlevel("monkey (code)")) actually has three
      different uses of parens. 20/(5*somefunction(")")) actually does
      have balanced parens. Parsing this is not a trivial problem,
      but we need to deal with it at some point.

      > Does that sound compatible with your vision for the architecture?
      >
      > Sounds right on, with the caveat that I'd like to comprehend what is doing
      > the mathematical operator work to produce the tree in the first place (new
      > code?).

      Probably new code, possibly heavily influenced by JEP's parser.

      > Having said that, under the assumption that we're going to be developing a
      > parser for ourselves... and thus having to write something to do evaluation
      > of the tree the parser produces as well as testing the parser and
      > evaluator... then I have an idea I would like to float (or sink, as the case
      > may be).
      >
      > So this is Tom as code contributor (not even Code Chimp, since James might
      > demote me for something this reckless :D ), not Tom in Architecture
      > Silverback mode:
      >
      > For various reasons, I wrote a formula parser some years ago (2004-2005).
      > It uses JJTree to produce a .jj file which is then compiled to .java files
      > through javacc.  Connor recently did work to eliminate javacc, which is
      > probably controversy #1 in reintroducing derived code and javacc as an ant
      > task (Though I believe it's the most reliable way to produce a parser, so if
      > we go our own way from JEP, this should be the path)
      >
      > A portion of the project at the time was learning the inner workings of
      > compilers, Java and JavaCC, so I was also playing with ASM (potential
      > controversy #2, adding yet another library)
      >
      > What I produced is code that can take a formula (e.g. "5+T") and compile it
      > directly into Java bytecode (and then load that bytecode as a class).  This
      > usually raises hair on the back of people's necks, because dynamic code is
      > considered heresy under certain design rules (controversy #3).
      >
      > Hey, at least it's not self-modifying code.  I could have done that :P
      >
      > I dropped it as a project about the time I joined PCGen, but it does work
      > for many of the basics (perhaps enough for PCGen?) and can even simplify
      > constants like 1+3 so the resolve method is simply load 4 onto the stack and
      > return.
      >
      > By compiling to bytecode it is extremely fast (some items - terms in
      > Andrew's parlance - simply become a method call to another Formula, while
      > variables can be referred to through a resolving object) at runtime.
      >
      > I haven't looked at it in a while, so it may be a bit of a pain to integrate
      > with your Term system, but I'm happy to hand it over to you (and others) to
      > work on, so it's an option we can evaluate if people don't think it's
      > completely nuts.

      I'd certainly like to take a look at it. It sounds horrendously complex and
      to be honest I think this problem is complex enough already, but as I said
      I'd be interested in seeing it.

      > Note also for any non-JEP system, we produce the complication that the
      > existing functions (e.g. count) we have would need to be rewritten into the
      > new system vs the JEP functions we have today.

      Yes, I know. I wrote the count function it only really works for Abilty
      objects at the moment. If we can get a system that understands our
      hardcoded stuff like COUNT[] then we can probably implement term objects
      to handle that and deprecate the count() function. There are other JEP
      functions that are useful. var() would become obsolete, since var("MYVAR")
      and MYVAR should do identical things (the former might be less efficient).

      andrew
    • Tom Parker
      ... One of the major obstacles is the use of things that should be reserved characters but aren t, () are nasty. I don t think this is as nasty as you think it
      Message 2 of 15 , May 2, 2009
      • 0 Attachment

        --- On Sat, 5/2/09, Andrew Wilson <affable@...> wrote:
        One of the major obstacles is the
        use of things that should be reserved characters but aren't, () are
        nasty.

        I don't think this is as nasty as you think it is... as long as they can be distinguished, it will work in a parser system like jjtree/javacc.  I've written a few compilers (not the depth of complexity of modern languages, but that type of thing), and it's not bad, and arguably easier than how we separate tokens in the LST system today.

        I'd certainly like to take a look at it.  It sounds horrendously
         complex
        and
        to be honest I think this problem is complex enough already, but as I said
        I'd be interested in seeing it.

        Complexity is relative.  I think it actually transfers a lot of complexity into javacc.  The main complexity is mostly in the optimization (making 1+3 into 4), which we could really tear out if it is too confusing or too complex.  With that removed, you may be surprised at how simple a parser/compiler really is.

        I'll dig it up and upload it into a sandbox.

        TP.
        --
        Tom Parker
        thpr@... and tppublic@...


      • Tom Parker
        ... This is now loaded into sandbox/FormulaCompiler Yes, it s a bit of a heavy user of System.err at the moment, it s still a bit broken, and the tests aren t
        Message 3 of 15 , May 2, 2009
        • 0 Attachment
          --- In pcgen_developers@yahoogroups.com, Tom Parker <thpr@...> wrote:
          > I'll dig it up and upload it into a sandbox.

          This is now loaded into sandbox/FormulaCompiler

          Yes, it's a bit of a heavy user of System.err at the moment, it's still a bit broken, and the tests aren't *really* tests in the jUnit sense, but this really wasn't originally intended for public consumption...

          TP.
        • Andrew Wilson
          Thanks, I ll have a look. andrew ...
          Message 4 of 15 , May 2, 2009
          • 0 Attachment
            Thanks, I'll have a look.

            andrew

            2009/5/2 Tom Parker <thpr@...>:
            > --- In pcgen_developers@yahoogroups.com, Tom Parker <thpr@...> wrote:
            >> I'll dig it up and upload it into a sandbox.
            >
            > This is now loaded into sandbox/FormulaCompiler
            >
            > Yes, it's a bit of a heavy user of System.err at the moment, it's still a bit broken, and the tests aren't *really* tests in the jUnit sense, but this really wasn't originally intended for public consumption...
            >
            > TP.
            >
            >
            >
            >
            > ------------------------------------
            >
            > Yahoo! Groups Links
            >
            >
            >
            >

            <div><br></div>
          • Martijn Verburg
            Hi All, Searched for alternatives to JEP and also looked for Forks of that library, and what do you know! I found a discussion between myself and their main
            Message 5 of 15 , May 3, 2009
            • 0 Attachment
              Hi All,

              Searched for alternatives to JEP and also looked for Forks of that library, and what do you know! I found a discussion between myself and their main developer about forking the project.

              It is a valid option if we want to do that as well.

              Thoughts?

              K

              > Thanks, I'll have a look.
              >
              > andrew
              >
              > 2009/5/2 Tom Parker <thpr@...>:
              > > --- In pcgen_developers@yahoogroups.com, Tom Parker <thpr@> wrote:
              > >> I'll dig it up and upload it into a sandbox.
              > >
              > > This is now loaded into sandbox/FormulaCompiler
              > >
              > > Yes, it's a bit of a heavy user of System.err at the moment, it's still a bit broken, and the tests aren't *really* tests in the jUnit sense, but this really wasn't originally intended for public consumption...
              > >
              > > TP.
              > >
              > >
              > >
              > >
              > > ------------------------------------
              > >
              > > Yahoo! Groups Links
              > >
              > >
              > >
              > >
              >
              > <div><br></div>
              >
            • Tom Parker
              My biggest concern with JEP is that we are operating under a license exception.  We would need to be *very* clear on how licensing would work in a fork - His
              Message 6 of 15 , May 3, 2009
              • 0 Attachment
                My biggest concern with JEP is that we are operating under a license exception.  We would need to be *very* clear on how licensing would work in a fork - His project is GPL2, which causes problems for us.  We are using Apache libraries that are not compatible with GPL2. 

                I would strongly vote against any fork which was still operating under an exception - there is *way* to much risk involved since it starts to impact our ability to use code we would have to develop/modify.  The fork would have to be LGPL or Apache licensed (he would have to explicitly license it that way in a public forum, not just in an email to us) or it's a non-starter IMHO.

                I also happen to agree with Andrew - what we're looking for is much simpler than JEP.  The effort spent taking out the complexity we don't want is better spent building up what we DO want.  Parsers really aren't hard.  In fact, we may need to take complexity OUT of the (much simpler) parser I wrote (though we need to add handling of some additional characters inside quotes)

                I also think a fork is actually worse, unless we get extensive test code along with his code.  If we have to spend years developing a test library, then it's better & faster to start from scratch so we can know what we are dealing with and only have to test the subset of what we want & develop (not what people could use if they knew JEP inside and out)

                TP.
                --
                Tom Parker
                thpr@... and tppublic@...

                --- On Sun, 5/3/09, Martijn Verburg <martijnverburg@...> wrote:
                From: Martijn Verburg <martijnverburg@...>
                Subject: [pcgen_developers] Re: [PCGD] Code Meeting April 2009 - Log
                To: pcgen_developers@yahoogroups.com
                Date: Sunday, May 3, 2009, 4:49 AM

                Hi All,

                Searched for alternatives to JEP and also looked for Forks of that library, and
                what do you know! I found a discussion between myself and their main developer
                about forking the project.

                It is a valid option if we want to do that as well.

                Thoughts?

                K

                > Thanks, I'll have a look.
                >
                > andrew
                >
                > 2009/5/2 Tom Parker <thpr@...>:
                > > --- In pcgen_developers@yahoogroups.com, Tom Parker <thpr@>
                wrote:
                > >> I'll dig it up and upload it into a sandbox.
                > >
                > > This is now loaded into sandbox/FormulaCompiler
                > >
                > > Yes, it's a bit of a heavy user of System.err at the moment,
                it's still a bit broken, and the tests aren't *really* tests in the
                jUnit sense, but this really wasn't originally intended for public
                consumption...
                > >
                > > TP.
                > >
                > >
                > >
                > >
                > > ------------------------------------
                > >
                > > Yahoo! Groups Links
                > >
                > >
                > >
                > >
                >
                > <div><br></div>
                >




                ------------------------------------

                Yahoo! Groups Links

                <*> To visit your group on the web, go to:
                http://groups.yahoo.com/group/pcgen_developers/

                <*> Your email settings:
                Individual Email | Traditional

                <*> To change settings online go to:
                http://groups.yahoo.com/group/pcgen_developers/join
                (Yahoo! ID required)

                <*> To change settings via email:
                mailto:pcgen_developers-digest@yahoogroups.com
                mailto:pcgen_developers-fullfeatured@yahoogroups.com

                <*> To unsubscribe from this group, send an email to:
                pcgen_developers-unsubscribe@yahoogroups.com

                <*> Your use of Yahoo! Groups is subject to:
                http://docs.yahoo.com/info/terms/


              • Eric Jarman
                Here s a BSD licensed expression parser: http://sourceforge.net/projects/expression-tree/
                Message 7 of 15 , May 3, 2009
                • 0 Attachment
                  Here's a BSD licensed expression parser:  http://sourceforge.net/projects/expression-tree/


                  On Sun, May 3, 2009 at 11:36 AM, Tom Parker <thpr@...> wrote:


                  My biggest concern with JEP is that we are operating under a license exception.  We would need to be *very* clear on how licensing would work in a fork - His project is GPL2, which causes problems for us.  We are using Apache libraries that are not compatible with GPL2. 

                  I would strongly vote against any fork which was still operating under an exception - there is *way* to much risk involved since it starts to impact our ability to use code we would have to develop/modify.  The fork would have to be LGPL or Apache licensed (he would have to explicitly license it that way in a public forum, not just in an email to us) or it's a non-starter IMHO.

                  I also happen to agree with Andrew - what we're looking for is much simpler than JEP.  The effort spent taking out the complexity we don't want is better spent building up what we DO want.  Parsers really aren't hard.  In fact, we may need to take complexity OUT of the (much simpler) parser I wrote (though we need to add handling of some additional characters inside quotes)

                  I also think a fork is actually worse, unless we get extensive test code along with his code.  If we have to spend years developing a test library, then it's better & faster to start from scratch so we can know what we are dealing with and only have to test the subset of what we want & develop (not what people could use if they knew JEP inside and out)

                  TP.
                  --
                  Tom Parker
                  thpr@... and tppublic@...

                  --- On Sun, 5/3/09, Martijn Verburg <martijnverburg@...> wrote:
                  From: Martijn Verburg <martijnverburg@...>
                  Subject: [pcgen_developers] Re: [PCGD] Code Meeting April 2009 - Log
                  To: pcgen_developers@yahoogroups.com
                  Date: Sunday, May 3, 2009, 4:49 AM


                  Hi All,

                  Searched for alternatives to JEP and also looked for Forks of that library, and
                  what do you know! I found a discussion between myself and their main developer
                  about forking the project.

                  It is a valid option if we want to do that as well.

                  Thoughts?

                  K

                  > Thanks, I'll have a look.
                  >
                  > andrew
                  >
                  > 2009/5/2 Tom Parker <thpr@...>:
                  > > --- In pcgen_developers@yahoogroups.com, Tom Parker <thpr@>
                  wrote:
                  > >> I'll dig it up and upload it into a sandbox.
                  > >
                  > > This is now loaded into sandbox/FormulaCompiler
                  > >
                  > > Yes, it's a bit of a heavy user of System.err at the moment,
                  it's still a bit broken, and the tests aren't *really* tests in the
                  jUnit sense, but this really wasn't originally intended for public
                  consumption...
                  > >
                  > > TP.
                  > >
                  > >
                  > >
                  > >
                  > > ------------------------------------
                  > >
                  > > Yahoo! Groups Links
                  > >
                  > >
                  > >
                  > >
                  >
                  > <div><br></div>
                  >




                  ------------------------------------

                  Yahoo! Groups Links

                  <*> To visit your group on the web, go to:
                  http://groups.yahoo.com/group/pcgen_developers/

                  <*> Your email settings:
                  Individual Email | Traditional

                  <*> To change settings online go to:
                  http://groups.yahoo.com/group/pcgen_developers/join
                  (Yahoo! ID required)

                  <*> To change settings via email:
                  mailto:pcgen_developers-digest@yahoogroups.com
                  mailto:pcgen_developers-fullfeatured@yahoogroups.com

                  <*> To unsubscribe from this group, send an email to:
                  pcgen_developers-unsubscribe@yahoogroups.com

                  <*> Your use of Yahoo! Groups is subject to:
                  http://docs.yahoo.com/info/terms/





                • Tom Parker
                  ... Eric, Interesting find, thanks. I think there may be some things we can learn from this even if we can t use it (such as writing expressions back out to
                  Message 8 of 15 , May 3, 2009
                  • 0 Attachment
                    --- In pcgen_developers@yahoogroups.com, Eric Jarman <ejarman@...> wrote:
                    >
                    > Here's a BSD licensed expression parser:
                    > http://sourceforge.net/projects/expression-tree/
                    >

                    Eric,

                    Interesting find, thanks. I think there may be some things we can learn from this even if we can't use it (such as writing expressions back out to Strings).

                    My biggest concern with this design is that it is one big method (lots of ifs, etc.) that does the work.

                    This is fragile (consider how many times specific operators like '+' appear in the method and how hard it thus is to change). It also seems variable names with numbers are not supported (potentially a problem for us, and also subject to the 'fragile method' problem).

                    I think the test cases are useful for us to use as a demonstration, but would be reluctant to recommend this vs. the use of a parser generated by JJTree/JavaCC.

                    I think Andrew should take a look at this too, however, as I think he'll be leading up the formula work.

                    TP.
                  • Andrew Wilson
                    ... I glanced at it but it hasn t had any work done for several years. I d assumed it has similar problems to JEP so I didn t pursue it. I ll have another
                    Message 9 of 15 , May 4, 2009
                    • 0 Attachment
                      2009/5/3 Tom Parker <thpr@...>:
                      > I think Andrew should take a look at this too, however, as I think
                      > he'll be leading up the formula work.

                      I glanced at it but it hasn't had any work done for several years.
                      I'd assumed it has similar problems to JEP so I didn't pursue
                      it. I'll have another look.

                      andrew
                    • Tom Parker
                      I share the time lag concern, but at least useful to look at the archictectural decisions since one of our options is starting fro scratch TP. -- Tom Parker
                      Message 10 of 15 , May 4, 2009
                      • 0 Attachment
                        I share the time lag concern, but at least useful to look at the archictectural decisions since one of our options is "starting fro scratch"

                        TP.
                        --
                        Tom Parker
                        thpr@... and tppublic@...

                        --- On Mon, 5/4/09, Andrew Wilson <affable@...> wrote:
                        From: Andrew Wilson <affable@...>
                        Subject: Re: [pcgen_developers] Re: [PCGD] Code Meeting April 2009 - Log
                        To: pcgen_developers@yahoogroups.com
                        Date: Monday, May 4, 2009, 6:31 AM

                        2009/5/3 Tom Parker <thpr@...>:
                        > I think Andrew should take a look at this too, however, as I think
                        > he'll be leading up the formula work.

                        I glanced at it but it hasn't had any work done for several years.
                        I'd assumed it has similar problems to JEP so I didn't pursue
                        it. I'll have another look.

                        andrew


                        ------------------------------------

                        Yahoo! Groups Links

                        <*> To visit your group on the web, go to:
                        http://groups.yahoo.com/group/pcgen_developers/

                        <*> Your email settings:
                        Individual Email | Traditional

                        <*> To change settings online go to:
                        http://groups.yahoo.com/group/pcgen_developers/join
                        (Yahoo! ID required)

                        <*> To change settings via email:
                        mailto:pcgen_developers-digest@yahoogroups.com
                        mailto:pcgen_developers-fullfeatured@yahoogroups.com

                        <*> To unsubscribe from this group, send an email to:
                        pcgen_developers-unsubscribe@yahoogroups.com

                        <*> Your use of Yahoo! Groups is subject to:
                        http://docs.yahoo.com/info/terms/


                      Your message has been successfully submitted and would be delivered to recipients shortly.