Loading ...
Sorry, an error occurred while loading the content.
 

Re: Reducing Sources (a feature suggestion)

Expand Messages
  • Keith Davies
    Date: Thu, 03 Oct 2002 16:29:56 +0000 From: Strom McLeod Subject: [pcgen] Re: Reducing Sources (a feature suggestion) In-reply-to:
    Message 1 of 5 , Oct 3, 2002
      Date: Thu, 03 Oct 2002 16:29:56 +0000
      From: Strom McLeod <pkmf@...>
      Subject: [pcgen] Re: Reducing Sources (a feature suggestion)
      In-reply-to: <20021002235747.GA29959@...>
      To: Keith Davies <keith.davies@...>
      User-Agent: eGroups-EW/0.82

      > Just found this group... so we're taking this up here...
      > "Me-Previously" = stuff I said before, "Keith" = Keith's response to
      > my "Me-Previously", and "Me-Now" = new stuff I'm saying now... :)
      >
      > ***
      > Me-Previously: At some point, object oriented programming dictates
      > that you're going to use domain-specific objects. However, to parse
      > through the XML file (and later, write one out) you'd probably still
      > want to, at some point use SAX (at least)
      >
      > Keith: My expected solution -- the one I monkey with when I have time
      > -- is to use SAX (Xerces-C++, myself) to parse the data. This is one
      > of the reasons I require that references only be made to items already
      > seen, so they can be resolved as they're found. They can be written
      > out to XML using whatever technology we want.
      >
      > Me-Now: I think we both agree then, yes? We've both stated that SAX
      > should be used to suck things into internal domain-specific objects.

      Definitely.

      > Keith: What I plan to have (and to propose to PCGen) is an internal
      > data model (probably close to the XML model, to a certain extent),
      > completely and utterly divorced from the loading mechanism. This will
      > make it possible to change to a different model -- load everything
      > from an Access (<shudder>) database, say -- without affecting the
      > internal model.
      >
      > Me-Now: I mentioned the MVC architecture previously, which might be
      > exactly the pattern you're looking for in this case. MVC involves
      > keeping how things are presented (View) sepearate from the data (Model)
      > I don't think we need to go full MVC, but if we do, it'll allow us to
      > run PCGen as a web-app, dump the clunkier AWT/Swing elements and go
      > with html. It would also mean that someone can 'host' pcgen on a
      > server with a stronger database in the back end and PCgen would be
      > accessed through a thin client. Sorry, that's my e-commerce showing
      > through. ;)

      What I was discussing was a little beyond MVC. Or a little before it --
      high cohesion, loose coupling. That is, *nothing* knows anything it
      doesn't need to. So, since the front end doesn't need to know about
      adding bonuses, it doesn't do it (well, with the exception of 'add
      arbitrary bonus' feature... which might still be data-based rather than
      front end). It knows 'add level of class', 'purchase weapon', etc. and
      otherwise leaves the data alone.

      The data engine has no knowledge of file formats or how things get
      loaded or saved (in C++ terms, we don't do anything like

      class Weapon {
      public:
      void write_lst( ostream &os);
      void write_xml( ostream &os);
      };

      because I/O is strictly external to the data engine). I think the data
      engine may be the base service of almost everything; the program creates
      a PCGenDataSource (loads LST or XML files, reads from database, randomly
      generates stuff, I don't care) and gives things to the data engine as
      needed. For instance, we might do

      class SomeSource : public PCGenDataSource {
      load_weapons( istream& is, DataEngine &de) {
      Weapon w;
      while (read_weapon( is, w)) {
      de.add_weapon( w);
      }
      }
      }

      The reader/writer know about the domain classes (Weapon, Spell) and just
      enough about the engine to add to it or fetch from it, as appropriate.
      No knowledge of lists/maps/Oracle tables/whatever at all.

      It's just good design; I hadn't gotten around to thinking of it
      (focusing on XML) so I figured I'd mention it so I don't forget.

      > Keith: replace the data engine completely with a relational database
      > (or whatever) if desired, possibly even dropping the input and output
      > engines
      >
      > Me-Now: Yes, check out J2EE specs... use JDBC for now, because it's
      > easier than trying to hack EJBs. The backend database doesn't matter
      > as much and becomes transparent (mostly) so that you can even use
      > freeware like MySQL, or... go the e-Tools route and shove it in
      > Access. even with a database though, you still need internal objects,
      > it just makes data transfer and storage a little tighter than flat files.
      > Oracle 8i can output/input directly from XML.... i doubt we can afford
      > the licensing costs though (around $10k per processor, IIRC) :)

      When we get around to coding, I intend to design something that hides
      that information from whatever's using it. I don't want client code to
      have to know (beyond the creation of the data engine) what type of
      engine it's dealing with (in memory, Oracle, whatever). The UML (sort
      of) would look someting like:

      Data Engine
      ^ ^
      | + In Memory Data Engine
      |
      + RDBMS Data Engine
      ^
      + Oracle
      + PostgreSQL
      + Access

      (although the last bit there may not be strictly true if you use a
      library (JDBC/ODBC/etc.) that abstracts to that point)

      > Keith: Should be. OTOH, there's a lot of parsing and reparsing and
      > parsing again of text data in PCGen, there are a huge number of
      > conversions from string to number, and so on. Early on in the XML
      > discussions there was talk of using JDOM internally, which would keep
      > things in XML(ish)technology all through runtime.
      >
      > Me-Now: I think we both agree that it sbould be parse-once, then
      > manipulate the internal representation there-after, unless sources are
      > reloaded.

      stipulation: as long as we are reading from a data file (XML, LST) I
      agree. If we're not reading from file (from RDBMS, say) we may want to
      ignore the input source and replace the data engine (as described
      above).

      As long as we're dealing with files, yes. Do the hard part once.

      > Keith: I'm not a speed freak, but I shouldn't be waiting for menus to
      > pop up when I click them, either.
      >
      > Me-Now: I tend to go for thin clients running off HTML/JSP/Javascript
      > which has potentially less functionality, but seems to load on screen
      > and refresh faster.

      As long as you're close to the server, and the server has the wheels for
      it. If I were to run something like that from my firewall (not a *bad*
      machine -- at one time it was pretty impressive) it would *suck* to use.
      I don't like depending on a decent pipe for my software to behave well.

      That said, yeah, it should work. It's not the route I'd use, but it
      seems viable.

      > Me-Previously: You can somewhat rely on Java exception handling to
      > avoid doing this.> This is probably not the right forum for debating
      > coding style.
      >
      > Keith: Hmm... I hadn't considered exceptions here. By nature I'm a
      > C++ programmer; exceptions are used for *unexpected* things, such as
      > running out of memory or a file suddenly being unwritable (these
      > things can be anticipated, but aren't expected). Parsing errors,
      > especially when dealing with human-keyed text, are to be expected and
      > handled more
      > gracefully. IMO, exceptions are not intended to be used for things
      > that are expected to go wrong. It's one of the things that drives me
      > nuts about Java, actually, is how exceptions are used for things like
      > parse errors.
      >
      > Me-Now: Yes, I come from a C++ (and C) camp also, and it took some
      > time to embrace the whole exception-handling issue. Coding
      > style-wise, here's my opinion... do the data validation at the point
      > of input/output, and assume valid information there after. Use Parse
      > exceptions to handle cases where you're expecting an 'int' but receive
      > something else... that's what I meant by exception handling. Java has
      > built-in input formating that will simplify parsing error handling.
      > In the Java mindset: Expect the right format/data, handle exceptions
      > separately.
      >
      Definitely. If you don't allow bad data into the system, you don't have
      to worry about bad data in the system (barring system failure, weird
      behavior, and general paranoia).

      However, in this case we don't have full control over the input data.
      Specifically, someone could modify a data file manually. This is on
      purpose -- we've always used text files simply because they are human
      readable. More or less. We're switching to XML because it makes the
      data *more* human readable (it's supposed to be a self-describing
      format), but more importantly, so even *more* tools can make use of the
      data without requiring custom code just to figure out what the files say
      (for instance, I could use a general XSLT processor to create a report
      containing all the spells I allow in my game... or to create a file for
      import into PCGen. In any case, because we are going with an
      easily-accessible format, one that will allow files to be read, written,
      and buggered up by software *and* humans, we can't take anything about
      the content on faith and must test everything as we go.

      As a result, IMO 'finding errors' is to be expected, not just
      anticipated, and as such I think would not usually be a good candidate
      for using exceptions. It depends a lot on what you want to do with the
      errors, though. For instance, if 'any error borks the load' (which
      could be a reasonable thing to do, because a failed load could leave the
      data engine in an... unusual state) exceptions could be a good way of
      handling it because you could just trap it at the top, give an error
      message ('file X could not load properly because thing T was invalid;
      see line N'), flush the bogus data. OTOH, if you want to recover from
      the error that's best done close to the source of the error ('value for
      F not found, using default'); in this case I'd find exceptions a
      cumbersome mechanism.

      Again, it's a style issue more than anything.


      > Keith: Style and language issue. I can do OO in Java and write Java
      > code, but I don't like the way some things are done in Java. I don't
      > like the way some things are done in C++ either, but I'm more
      > accustomed to it, I find it faster, and I have templates!
      >
      > Me-Now: You can write base classes and abstract classes (templates)
      > and then extend (inherit) from them in a similar way. Yes, it's a
      > slightly different way of thinking though. again, your mileage may
      > vary and you are definitely entitled to your view on this. :) I just
      > enjoy not having to deal with garbage collection and worrying about
      > whether the other coder is going to segfault me.

      If you mean what I think you do, it's a route I prefer not to follow
      because it's caused problems for me in the past (it's one of the first
      ways of handling the problem, I think, and it chews on delicate anatomy
      as soon as you get too far with it. That could be an issue with the
      implementation of the solution described, though.

      > Keith: I think I mentioned a system that was deliberately made hard to
      > use, to avoid borking the data.
      >
      > Me-Now: Yes, and we had similar constraints dealing with e-commerce...
      > wouldn't want the thousands of users to be able to screw up the system
      > and get free CDs, etc. Yet, provide enough flexibility for our
      > web-dev guys.

      I *like* flexibility. I want to make the system as flexible as is
      reasonable. I want them to be able to extend data anf functionality as
      much as desired, as long as they can't extend the flexibility.... <g>


      Keith
      --
      Keith Davies
      keith.davies@...

      PCGen: <reaper/>, smartass
      "You just can't argue with a moron. It's like handling Nuclear
      waste. It's not good, it's not evil, but for Christ's sake, don't
      get any on you!!" -- Chuck, PCGen mailing list
    • Strom McLeod
      ... That s what the MVC model emphasizes. ... Yes, the View side only knows to render whatever gets passed to it from the Model. ... Yes and No. At some point
      Message 2 of 5 , Oct 3, 2002
        --- In pcgen-xml@y..., Keith Davies <keith.davies@k...> wrote:
        > What I was discussing was a little beyond MVC. Or a little before it --
        > high cohesion, loose coupling. That is, *nothing* knows anything it
        > doesn't need to.

        That's what the MVC model emphasizes.

        > So, since the front end doesn't need to know about
        > adding bonuses, it doesn't do it (well, with the exception of 'add
        > arbitrary bonus' feature... which might still be data-based rather than
        > front end). It knows 'add level of class', 'purchase weapon', etc. and
        > otherwise leaves the data alone.

        Yes, the View side only knows to render whatever gets passed to it
        from the Model.

        > The data engine has no knowledge of file formats or how things get
        > loaded or saved

        Yes and No. At some point in the data access layer, you will have to
        specify "hello, I'm using Oracle". Think of it as a display driver.
        Above the display driver level, you only need to know which display
        driver to call. Within the display driver level, that driver needs to
        know specifically how to manipulated a specific video card series.

        > When we get around to coding, I intend to design something that hides
        > that information from whatever's using it. I don't want client code
        to have to know (beyond the creation of the data engine) what type of
        > engine it's dealing with (in memory, Oracle, whatever).

        yes, you're reiterating what's in J2EE and the MVC model.

        > Data Engine
        > ^ ^
        > | + In Memory Data Engine
        > |
        > + RDBMS Data Engine
        > ^
        > + Oracle
        > + PostgreSQL
        > + Access

        Try this instead:

        Data Engine
        ^ ^...... Objects representing data in memory
        |
        Data Access Layer
        ^
        |
        | (SQL Statements)
        |
        JDBC Driver -> any database that has a jdbc driver

        > OTOH, if you want to recover from the error that's best done close
        to the source of the error ('value for F not found, using default');
        in this case I'd find exceptions a cumbersome mechanism.

        What you said above is exactly what you do in the 'catch' clause of
        the exceptions mechanism, Keith.

        Here's the structure of a hypothetical program:
        You have a input handler called userInputHandler, which keeps track of
        which set of inputs it is expecting, perhaps with information about
        which screen the user is in.
        Let's say it's getting the 6 attributes. So it loops through a list,
        starting with Strength, which is listed as a pair, ["Strength",
        "Int"], followed by ["Dexterity, "Int"] and so on... so that the input
        parser knows to expect an int.

        userInputHandler calls textInput->parse(InputString, InputFormat)
        (where InputFormat = "Int" as given from the list)
        textInput.parse() reads in InputString (say it says "Bob") and throws
        an exception, invalid format.

        userInputHandler catches that exception and knows that this specific
        instance, when it called textInput.parse(), it was asking for the
        value of Strength, and can thus, in the catch statement, say "Your
        Strength value is invalid." AND, the catch statement can set this to
        a default value.
        IN ADDITION, this catch statement can choose to propogate the
        exception up it's call chain or stop it right here.
        Some choose to pass back a List of error messages for the View to show
        later.

        All this is done using the exception pattern.

        textInput.parse doesn't need to know that it's going for Strength
        verses Character Name, it only cares about the type you expect it to
        return. The caller can decide how to handle that exception depending
        on the case.
        You write one parser, you call it many times, the caller knows what to
        do if something bad/exceptional happens.

        again, your mileage and style may vary, but the exception handling
        paradigm goes pretty much as described above, using try-catch clauses.
      • Keith Davies
        ... I m more accustomed to seeing MVC in reference to GUIs, not just abstraction and encapsulation. ... Not necessarily. There re two ways to implement a
        Message 3 of 5 , Oct 4, 2002
          On Thu, Oct 03, 2002 at 07:13:29PM +0000, Strom McLeod wrote:
          > --- In pcgen-xml@y..., Keith Davies <keith.davies@k...> wrote:
          > > What I was discussing was a little beyond MVC. Or a little before it --
          > > high cohesion, loose coupling. That is, *nothing* knows anything it
          > > doesn't need to.
          >
          > That's what the MVC model emphasizes.

          I'm more accustomed to seeing MVC in reference to GUIs, not just
          abstraction and encapsulation.

          > > The data engine has no knowledge of file formats or how things get
          > > loaded or saved
          >
          > Yes and No. At some point in the data access layer, you will have to
          > specify "hello, I'm using Oracle". Think of it as a display driver.
          > Above the display driver level, you only need to know which display
          > driver to call. Within the display driver level, that driver needs to
          > know specifically how to manipulated a specific video card series.

          Not necessarily. There're two ways to implement a system that uses
          Oracle. The first is where it is the primary store of the data, where
          the program gets it while working (an OracleDataEngine, if you will).
          When the front end requests the description of, say, /fireball/, it
          calls the data engine through its defined interface, then behind that
          the data engine starts sending queries to Oracle, constructs the Spell
          that contains the description, then passes it back to the caller. In
          this case, the implementation of the OracleDataEngine of couse must know
          that it's using Oracle.

          The case I was talking about, though, was that the data engine uses the
          'normal' implementation of storing everything in RAM. A DataSource will
          be used to retrieve the data from somewhere and pass the created objects
          to the DataEngine. In the XML case, that means an object will exist
          that will open the specified XML file and parse it; as it completes
          construction of business objects and loading them with their data, it
          passes those objects to the DataEngine. An OracleSource would open the
          database, suck all the related data out of it, construct the objects,
          and pass them to the DataEngine. In this case, the DataEngine has no
          knowledge that its data came out of Oracle, or XML, or LST, or whatever,
          hence my initial statement.

          The output object is the same, a client of the DataEngine. It asks the
          DataEngine for all the objects and stores them however it wishes.
          Again, the DataEngine does not know that this is XML, LST, or Oracle.

          > > When we get around to coding, I intend to design something that hides
          > > that information from whatever's using it. I don't want client code
          > to have to know (beyond the creation of the data engine) what type of
          > > engine it's dealing with (in memory, Oracle, whatever).
          >
          > yes, you're reiterating what's in J2EE and the MVC model.
          >
          > > Data Engine
          > > ^ ^
          > > | + In Memory Data Engine
          > > |
          > > + RDBMS Data Engine
          > > ^
          > > + Oracle
          > > + PostgreSQL
          > > + Access
          >
          > Try this instead:
          >
          > Data Engine
          > ^ ^...... Objects representing data in memory
          > |
          > Data Access Layer
          > ^
          > |
          > | (SQL Statements)
          > |
          > JDBC Driver -> any database that has a jdbc driver

          Sure. The beauty of this is that it isn't necessary to do it only one
          way. It's an implementational issue well away from where we're trying
          to do here right now.

          > > OTOH, if you want to recover from the error that's best done close
          > to the source of the error ('value for F not found, using default');
          > in this case I'd find exceptions a cumbersome mechanism.
          >
          > What you said above is exactly what you do in the 'catch' clause of
          > the exceptions mechanism, Keith.

          Remember, I'm not *that* accustomed to Java exceptions. I found them
          intrusive, when I've got more experience with another mechanism. I am
          also aware that I wasn't expressing myself well in earlier posts.

          I'll stipulate that they can certainly work, that it might be the
          correct way to do it in Java, but that my preference is to not use them
          for expected errors. Anticipated errors, yes -- those that I think can
          happen and need to plan for -- but not necessarily those that I know
          will happen. I'm not saying that using exceptions is wrong, just that
          my experience has been *not* using exceptions.

          It's a style issue from another language.

          > Here's the structure of a hypothetical program:
          > You have a input handler called userInputHandler, which keeps track of
          > which set of inputs it is expecting, perhaps with information about
          > which screen the user is in.
          > Let's say it's getting the 6 attributes. So it loops through a list,
          > starting with Strength, which is listed as a pair, ["Strength",
          > "Int"], followed by ["Dexterity, "Int"] and so on... so that the input
          > parser knows to expect an int.
          >
          > All this is done using the exception pattern.
          >
          > textInput.parse doesn't need to know that it's going for Strength
          > verses Character Name, it only cares about the type you expect it to
          > return. The caller can decide how to handle that exception depending
          > on the case.
          > You write one parser, you call it many times, the caller knows what to
          > do if something bad/exceptional happens.
          >
          > again, your mileage and style may vary, but the exception handling
          > paradigm goes pretty much as described above, using try-catch clauses.

          It certainly seems a reasonable solution.


          Keith
          --
          Keith Davies
          keith.davies@...

          PCGen: <reaper/>, smartass
          "You just can't argue with a moron. It's like handling Nuclear
          waste. It's not good, it's not evil, but for Christ's sake, don't
          get any on you!!" -- Chuck, PCGen mailing list
        • Strom McLeod
          ... It s not, it s a pattern for developing the whole system, not just the GUI. ... The whole point is to not need to specify exactly what database will be
          Message 4 of 5 , Oct 4, 2002
            --- In pcgen-xml@y..., Keith Davies <keith.davies@k...> wrote:
            > I'm more accustomed to seeing MVC in reference to GUIs, not just
            > abstraction and encapsulation.

            It's not, it's a pattern for developing the whole system, not just the
            GUI.

            > Not necessarily. There're two ways to implement a system that uses
            > Oracle.

            The whole point is to not need to specify exactly what database will
            be used and allow that to be abstracted throughout most of the system.

            You specify the details of the database/datasource to be used in a
            configuration file, in addition to a nickname that will be used to
            reference that database/datasource. You can switch databases used to
            realize without changing the nickname.

            The rest of the application uses the nickname to run SQL statements
            etc. without worrying about what driver is actually being used.
            There's no reference to 'Oracle' within the code, only "the database"
            or datasource.

            Technically, you should further encapsulate these SQL statements into
            a data access object/package that the rest of the system calls, so
            that all your SQL calls are centralized into one manageable place.

            Now, if you've already got how you want to architect and design this
            in mind already, as the rest of your post suggest, and given that I'm
            not even a recognized primate, I'll leave it be.

            Good luck
          • Keith Davies
            ... Yep. I discussed Oracle only as an example, and that I expect that the general design of the software would be such that, outside what the implementations
            Message 5 of 5 , Oct 4, 2002
              On Fri, Oct 04, 2002 at 09:02:02PM +0000, Strom McLeod wrote:
              > --- In pcgen-xml@y..., Keith Davies <keith.davies@k...> wrote:
              > > I'm more accustomed to seeing MVC in reference to GUIs, not just
              > > abstraction and encapsulation.
              >
              > It's not, it's a pattern for developing the whole system, not just the
              > GUI.
              >
              > > Not necessarily. There're two ways to implement a system that uses
              > > Oracle.
              >
              > The whole point is to not need to specify exactly what database will
              > be used and allow that to be abstracted throughout most of the system.

              Yep. I discussed Oracle only as an example, and that I expect that the
              general design of the software would be such that, outside what the
              implementations need to know, they don't need to know anything else.
              The specific example I used was that the data engine, if constructed
              correctly, will not need to know about the particular source used
              (database, XML file, LST file, etc.).

              > You specify the details of the database/datasource to be used in a
              > configuration file, in addition to a nickname that will be used to
              > reference that database/datasource. You can switch databases used to
              > realize without changing the nickname.

              Of course.

              > The rest of the application uses the nickname to run SQL statements
              > etc. without worrying about what driver is actually being used.
              > There's no reference to 'Oracle' within the code, only "the database"
              > or datasource.
              >
              > Technically, you should further encapsulate these SQL statements into
              > a data access object/package that the rest of the system calls, so
              > that all your SQL calls are centralized into one manageable place.

              This is what I would do.

              > Now, if you've already got how you want to architect and design this
              > in mind already, as the rest of your post suggest, and given that I'm
              > not even a recognized primate, I'll leave it be.

              The conversation has been useful. It's brought things to mind that I
              hadn't considered, or only superficially. I hope to be able to present
              something significant for discussion and dissection soon... but I've
              been saying that for a while now :/

              You do realize that we've been arguing the same points? <g> The only
              thing we've actually disagreed on is how to handle certain types of
              error, and in both cases the stances were based on different design
              styles and different languages... as for the rest of it, I can't think
              of anything either of us said that actually excluded what the other did.

              I do have strong opinions how my code should be built. I'm the one
              building it <g>. PCGen uses different technology (Java), and the
              techniques used to handle error conditions naturally differ. I'm
              working on the XML schema, and while *technically* it doesn't have
              anything to do with the program source, I think I must be able to
              demonstrate that the XML schema can be used for the purpose it is
              designed for. If I build a schema that does describe the data but is
              unusable, or causes us to end up with the data model problems we're
              facing now, it isn't, IMO, a good schema.

              The software design I'm working on in conjunction is only a suggestion
              as to how the data can be handled by the software. However, I suspect
              that the design may in fact be used. Even if it isn't, I don't want to
              present something technologically weak and have it get caught up in
              redesign because I did something outright stupid.


              Keith
              --
              Keith Davies
              keith.davies@...

              PCGen: <reaper/>, smartass
              "You just can't argue with a moron. It's like handling Nuclear
              waste. It's not good, it's not evil, but for Christ's sake, don't
              get any on you!!" -- Chuck, PCGen mailing list
            Your message has been successfully submitted and would be delivered to recipients shortly.