Loading ...
Sorry, an error occurred while loading the content.
 

Encapsulation was Re: [agileDatabases] Re: Proactive assessment of impact of database refactoring

Expand Messages
  • narayanang
    ... As a database guy, I love the idea of hiding all the tables from applications and putting a stored procedure interface in front of them. However, there are
    Message 1 of 14 , Sep 21, 2006
      > This is why the suggestion earlier (pardon me, but I
      > don't remember by whom) of hiding all tables from
      > applications and forcing those applications to go
      > through a stored procedure layer makes so much sense.
      > Application code need only change if the stored proc
      > API changes; the stored proc won't compile if it's in
      > error with respect to the underlying DB tables.

      As a database guy, I love the idea of hiding all the tables from
      applications and putting a stored procedure interface in front of
      them. However, there are some practical challenges from an overall
      system design perspective that one has to take into consideration.
      As with most design decisions, it is a question of making the right
      trade off.

      1. The stored procedure approach doesnt fit easily with O-R mapping
      paradigm. I'm yet to come across a good ORM solution that lets you
      map a class or its method to stored procedure(s). How do you fit
      stored procedures in a model-driven approach?

      2. Most OO developers consider writing JDBC code 'grunt work' at
      best, and would rather have a tool generate their DAO code. This is
      also reasonable from a developer productivity standpoint. Developers
      need to focus on coding business logic rather than
      marshalling/unmarshalling data for every database interaction.

      3. Another common reason we DBA's love SPs is Performance. Consider
      the nature of the application - in a transactional CRUD application,
      what is the marginal performance gain by using SPs vs the TCO?

      4. Are SPs simply used as CRUD methods on a table by table basis? If
      so, what's the real benefit? And, how does this scale as the data
      model gets bigger and bigger? Most projects I've seen have 0..1
      database developer and 1..* application developers!

      5. In an enterprise, where is the biggest bang for the buck on data
      abstraction/encapsulation? Is it between an app/service and it's
      data store or between the app/service and it's consumers?

      6. How often do we change the physical data model without a
      corresponding application change, unless it is purely a performance
      optimization?

      I've worked on a range of projects using DB driven, OO and model
      driven approaches. My assessment is, it does not have to be an all
      or nothing design. We can use ORM to manage the routine CRUD work
      and resort to SPs for bulk data processing, complex queries etc.
      i.e, use a tool to do the mundane work and free ourselves to focus
      on the fun stuff!


      --- In agileDatabases@yahoogroups.com, "Scott W. Ambler" <swa@...>
      wrote:
      >
      >
      >
      > ----- Original Message -----
      > From: "Russ Milliken" <milliken99@...>
      > To: <agileDatabases@yahoogroups.com>
      > Sent: Thursday, September 14, 2006 8:38 AM
      > Subject: Re: [agileDatabases] Re: Proactive assessment of impact
      of database
      > refactoring
      >
      >
      > > This is why the suggestion earlier (pardon me, but I
      > > don't remember by whom) of hiding all tables from
      > > applications and forcing those applications to go
      > > through a stored procedure layer makes so much sense.
      > > Application code need only change if the stored proc
      > > API changes; the stored proc won't compile if it's in
      > > error with respect to the underlying DB tables.
      >
      > Although I'm a firm believer in encapsulating database access, and
      there are
      > other options than just writing stored procs (see
      >
      http://www.agiledata.org/essays/implementationStrategies.html#Impleme
      ntationStrategies )
      > , you'll find it difficult at best to force everyone to use it.
      My advice
      > is to encapsulate access but don't count on everyone using it
      effectively.
      >
      > - Scott
      > Scott W. Ambler
      > Practice Leader Agile Development, IBM
      > http://www-306.ibm.com/software/rational/bios/ambler.html
      >
      > Every organization gets the process that it deserves.
      >
    • Dave Sanders
      ... I agree, however, its something I m keen to work on soon as an add-on to the Gentle framework, but I haven t experimented with it yet. ... Couldn t agree
      Message 2 of 14 , Sep 22, 2006
        --- In agileDatabases@yahoogroups.com, "narayanang" <narayanang@...>
        wrote:
        >
        > 1. The stored procedure approach doesnt fit easily with O-R mapping
        > paradigm. I'm yet to come across a good ORM solution that lets you
        > map a class or its method to stored procedure(s). How do you fit
        > stored procedures in a model-driven approach?

        I agree, however, its something I'm keen to work on soon as an add-on
        to the Gentle framework, but I haven't experimented with it yet.

        > 2. Most OO developers consider writing JDBC code 'grunt work' at
        > best, and would rather have a tool generate their DAO code. This is
        > also reasonable from a developer productivity standpoint. Developers
        > need to focus on coding business logic rather than
        > marshalling/unmarshalling data for every database interaction.

        Couldn't agree more.

        > 3. Another common reason we DBA's love SPs is Performance. Consider
        > the nature of the application - in a transactional CRUD application,
        > what is the marginal performance gain by using SPs vs the TCO?

        Agreed again. The vast majority of a typical web app is small, easily
        digestible CRUD. Occasionally, there are certain processes that
        require performance help, but those are the exception - not the rule -
        and can always be dealt with separately on a case by case basis.

        > 4. Are SPs simply used as CRUD methods on a table by table basis? If
        > so, what's the real benefit? And, how does this scale as the data
        > model gets bigger and bigger? Most projects I've seen have 0..1
        > database developer and 1..* application developers!

        This is why I switched from NetTiers to Gentle for my .NET
        development. NetTiers generated the SP layer, but it was all small
        CRUD statements. Not only did I need to maintain all of those files,
        but the perceived benefit of being able to modify them was moot,
        because once I did that I could never regenerate that file again. No
        benefit was provided EXCEPT ability to lock down the tables away from
        the webuser account.

        > 5. In an enterprise, where is the biggest bang for the buck on data
        > abstraction/encapsulation? Is it between an app/service and it's
        > data store or between the app/service and it's consumers?
        >
        > 6. How often do we change the physical data model without a
        > corresponding application change, unless it is purely a performance
        > optimization?

        Apparently, some folks on this list do this all the time - though I
        personally have never done it, and its usually my primary argument. :)
        In every situation I've been in, we'll make the performance upgrades
        or refactoring but its ALWAYS in the context of broader changes to the
        application. Usually these performance or "developability"
        enhancements were little tasks that get put into the iteration along
        with the larger new functionality tasks.

        > I've worked on a range of projects using DB driven, OO and model
        > driven approaches. My assessment is, it does not have to be an all
        > or nothing design. We can use ORM to manage the routine CRUD work
        > and resort to SPs for bulk data processing, complex queries etc.
        > i.e, use a tool to do the mundane work and free ourselves to focus
        > on the fun stuff!

        Agree 100%. You can usually never solve all problems with a single
        approach. The 80/20 rule applies, and I'd rather not deliberately
        create developer policy that causes more tedious work. I'd rather
        focus on the exceptions.

        D
      • Willem Bogaerts
        ... I couldn t agree less. Generated code is the biggest programming smell ever. If code is so common it can be automatically generated, it can surely be
        Message 3 of 14 , Sep 25, 2006
          >> 2. Most OO developers consider writing JDBC code 'grunt work' at
          >> best, and would rather have a tool generate their DAO code. This is
          >> also reasonable from a developer productivity standpoint. Developers
          >> need to focus on coding business logic rather than
          >> marshalling/unmarshalling data for every database interaction.
          >
          > Couldn't agree more.

          I couldn't agree less. Generated code is the biggest programming smell
          ever. If code is so common it can be automatically generated, it can
          surely be pulled up into a superclass.

          Storage can be perfectly handled in a storage layer. It needs not to be
          more specific than that. If your application can store a row (may be a
          collection or array) in a specific location and retrieve a row from a
          location, then that is all that is necessary.

          Outside implementations of the storage library, there should not be any
          SQL or even a notion of a database. The "table wrappers" communicate
          with data lines, not with databases.
          For example, I can decide to change the data line from a
          DatabaseQueryLine to an IniFileDataLine at any moment for my
          ApplicationSettings collection. And ApplicationSettings just inherits
          from LazyTableWrapper and gets the data line passed in the constructor.
          So there is no vast code duplication caused by some code spamming
          generator, and there has never been any reason for it either.

          >> 4. Are SPs simply used as CRUD methods on a table by table basis? If
          >> so, what's the real benefit? And, how does this scale as the data
          >> model gets bigger and bigger? Most projects I've seen have 0..1
          >> database developer and 1..* application developers!

          Sometimes. They may even start that way, like dumb-data-objects in
          object-oriented programs evolve to functional entities. I expect the
          stored procedures to evolve the same way.

          >> 6. How often do we change the physical data model without a
          >> corresponding application change, unless it is purely a performance
          >> optimization?

          In refactoring steps, quite often.

          >> I've worked on a range of projects using DB driven, OO and model
          >> driven approaches. My assessment is, it does not have to be an all
          >> or nothing design. We can use ORM to manage the routine CRUD work
          >> and resort to SPs for bulk data processing, complex queries etc.
          >> i.e, use a tool to do the mundane work and free ourselves to focus
          >> on the fun stuff!

          Instead of having the code spammed to you by a generator, write a good
          object model for the data handling. Now THAT is fun stuff!


          Best regards,
          Willem Bogaerts
        • Steve Jorgensen
          ... I see your point, but I don t think it s universally true. ... All that s necessary, perhaps, but not always all that s desirable. For instance, should
          Message 4 of 14 , Sep 25, 2006
            Willem Bogaerts wrote:

            >> 2. Most OO developers consider writing JDBC code 'grunt work' at
            >> best, and would rather have a tool generate their DAO code. This is
            >> also reasonable from a developer productivity standpoint. Developers
            >> need to focus on coding business logic rather than
            >> marshalling/ unmarshalling data for every database interaction.
            >
            > Couldn't agree more.

            I couldn't agree less. Generated code is the biggest programming smell
            ever. If code is so common it can be automatically generated, it can
            surely be pulled up into a superclass.

            I see your point, but I don't think it's universally true.

            Storage can be perfectly handled in a storage layer. It needs not to be
            more specific than that. If your application can store a row (may be a
            collection or array) in a specific location and retrieve a row from a
            location, then that is all that is necessary.

            All that's necessary, perhaps, but not always all that's desirable.  For instance, should there be front-end validation of some database constraints for performance reasons?  If so, how should the layer know what those constraints are?  How about relationships?  Should the storage layer know about them, and provide has-many and belongs-to lookups?  Should the calling code have to know what the linking keys are to ask the storage layer to do that?  Should the keys even be visible to the calling code when the storage layer can handle the relationships (ok, probably yes, but...)?

            I would ague that, in some cases, it makes sense to generate configuration files or actual code for that at the same time the related aspects of the schema are generated.  Regarding the difference between generated code and generated configuration files, my feeling is that they're just 2 different colors of the same animal.

            Outside implementations of the storage library, there should not be any
            SQL or even a notion of a database. The "table wrappers" communicate
            with data lines, not with databases.

            I'm mostly with you on that one, but sometimes, it ends up being a better choice to lock in to the user of a relational database, and be free to take advantage of its unique capabilities without having to wrap every single one of those that you want to use in some wrapper that could be used to abstract it to work with another back-end.  What if, for instance, I want to join records in  tables on a linking criterion while limiting records on both sides to a date range, and my tables have 100,000 rows each, and I expect 20 rows in my result?  If I want to take advantage of the server's ability to optimize the query based on the available indexes, etc., I need to give the whole thing to the server as one query that it can parse as a whole.

            For example, I can decide to change the data line from a
            DatabaseQueryLine to an IniFileDataLine at any moment for my
            ApplicationSettings collection. And ApplicationSettings just inherits
            from LazyTableWrapper and gets the data line passed in the constructor.
            So there is no vast code duplication caused by some code spamming
            generator, and there has never been any reason for it either.

            One need not think of duplication created by a code generator as duplication.  It's really just a build product.  The source code is the part the programmer must modify to change the code's behavior, and there's no duplication there.  If you use a macro or an inline hint in C, do you call it duplication?

            [...]

            Instead of having the code spammed to you by a generator, write a good
            object model for the data handling. Now THAT is fun stuff!

            I do agree that's often the perfect approach.  I think it would be wrong, though, to say that's a universal best answer.

            - Steve Jorgensen
          • Dave Sanders
            ... Then you don t understand the Gentle framework because thats EXACTLY what it does. The Gentle framework encapsulates all of the data access and speaks to
            Message 5 of 14 , Sep 25, 2006
              --- In agileDatabases@yahoogroups.com, Willem Bogaerts <w-p@...> wrote:
              > I couldn't agree less. Generated code is the biggest programming smell
              > ever. If code is so common it can be automatically generated, it can
              > surely be pulled up into a superclass.
              >
              > Storage can be perfectly handled in a storage layer. It needs not to be
              > more specific than that. If your application can store a row (may be a
              > collection or array) in a specific location and retrieve a row from a
              > location, then that is all that is necessary.
              >
              > Outside implementations of the storage library, there should not be any
              > SQL or even a notion of a database. The "table wrappers" communicate
              > with data lines, not with databases.
              > For example, I can decide to change the data line from a
              > DatabaseQueryLine to an IniFileDataLine at any moment for my
              > ApplicationSettings collection. And ApplicationSettings just inherits
              > from LazyTableWrapper and gets the data line passed in the constructor.
              > So there is no vast code duplication caused by some code spamming
              > generator, and there has never been any reason for it either.


              Then you don't understand the Gentle framework because thats EXACTLY
              what it does. The Gentle framework encapsulates all of the data
              access and speaks to various data providers (MSSQL, MYSQL, etc, the
              framework defines a data provider and you can write whatever you need)
              and abstracts ALL of the code to speak to the database.

              Regardless of the underlying framework, you still need to have classes
              that describe your data objects. You will have a User class, a Car
              class, whatever. The code generation comes in by taking your schema
              model from your database and autogenerating the classes that represent
              your data model TO INTERACT WITH GENTLE. This doesn't mean that there
              are hundreds of methods that all write some embedded SQL to go back to
              the database. This means that it generates classes that:

              1. Inherit from the base Persistence object - the cornerstone of Gentle.

              2. Implement the major methods from the Persistence object, like
              "Persist" and "Remove".

              3. Implement object linking to relationships. So, if a User had
              multiple Addresses assigned, you would automatically get an
              AddressList property on the User, which calls on the Gentle framework
              to handle the relationship data lookups.

              While the code generated is not a lot per class, if you have 20 or
              more tables, its still pretty tedious stuff. Also, you don't HAVE to
              use the code generator, and in fact, I don't think they are even
              packaged with the base framework. Its purely a tool to help speed up
              implementation. Gentle provides a way, through .NET attributes, to
              let you assign a table / column (or whatever paradigm your datastore
              uses) to any class you want. If I wanted to create a Monkey class and
              link it to the User table, I can.

              Once all of the code generation is done, the class is ready to do all
              of my CRUD. This usually takes about 2-5 seconds to run my generator.
              Then if I want to save an object to a table I simply say:

              Object.Persist();

              And it does it all for me.

              > >> 4. Are SPs simply used as CRUD methods on a table by table basis? If
              > >> so, what's the real benefit? And, how does this scale as the data
              > >> model gets bigger and bigger? Most projects I've seen have 0..1
              > >> database developer and 1..* application developers!
              >
              > Sometimes. They may even start that way, like dumb-data-objects in
              > object-oriented programs evolve to functional entities. I expect the
              > stored procedures to evolve the same way.

              Then it sounds like you are imparting business logic at your data
              tier? Why else would you be modifying your simple CRUD stored procs?
              I could see adding OTHER procs for doing certain bad performance
              tasks like going through very large tables with complex logic, or
              where you don't want to pull an entire table over the wire to the app
              server...
              or where the architecture is poor and you have to go through hoops to
              do basic things. Otherwise I rarely have to modify the basic crud.
              (even at the object layer)


              > >> 6. How often do we change the physical data model without a
              > >> corresponding application change, unless it is purely a performance
              > >> optimization?
              >
              > In refactoring steps, quite often.

              Different experiences. In my professional life, we've rarely had the
              time to simply go in and refactor. If we did a refactor, it was
              usually linked to a performance problem and almost always required a
              business level change.

              For better or worse, I've usually worked at places that saw "More
              Features" as more important than "Perfect Architecture". "More
              Features" = "More Money". "Perfect Architecture" = "More Cost" (In
              many managers eyes - even if you scream yourself blue in the face that
              the less than perfect architecture is costing you more in maintenance.)

              > >> I've worked on a range of projects using DB driven, OO and model
              > >> driven approaches. My assessment is, it does not have to be an all
              > >> or nothing design. We can use ORM to manage the routine CRUD work
              > >> and resort to SPs for bulk data processing, complex queries etc.
              > >> i.e, use a tool to do the mundane work and free ourselves to focus
              > >> on the fun stuff!
              >
              > Instead of having the code spammed to you by a generator, write a good
              > object model for the data handling. Now THAT is fun stuff!

              Why? There are more than enough of them out there - join an OSS
              project (like Gentle, if you are a dotNet person) and help make those
              ones perfect...

              D
            • narayanang
              ... Model-driven devlopment is all about focusing on building the model and have the tools generate - quite a bit, if not all of the code. Consider a 500 table
              Message 6 of 14 , Sep 25, 2006
                > Generated code is the biggest programming smell. If code is so
                > common it can be automatically generated, it can surely be pulled
                > up into a superclass.

                Model-driven devlopment is all about focusing on building the model
                and have the tools generate - quite a bit, if not all of the code.
                Consider a 500 table ERP application - what will be the effort
                required to created CRUD stored procedures for each table and how do
                you sell this to your architect or project manager :) ?

                Again, in a CRUD scenario, howis sp_insert/sp_update/sp_delete
                better than normal insert/update/delete - when they are all dealing
                with one row?

                > Instead of having the code spammed to you by a generator, write a
                > good object model for the data handling. Now THAT is fun stuff!

                That's exactly what persistence frameworks like Hibernate (java) or
                NHibernate (.NET)do. They use meta-data on how a class maps to a
                table and use it to prepare sql statements. This idea has been around
                since pre client/server days. Anyone remember Oracle Forms 2.3/3.0 or
                Power Builder? You didn't have to write a line of code to take what's
                on the screen and save it to the database and do the reverse.


                - Narayanan

                --- In agileDatabases@yahoogroups.com, Willem Bogaerts <w-p@...>
                wrote:
                >
                > >> 2. Most OO developers consider writing JDBC code 'grunt work' at
                > >> best, and would rather have a tool generate their DAO code. This
                is
                > >> also reasonable from a developer productivity standpoint.
                Developers
                > >> need to focus on coding business logic rather than
                > >> marshalling/unmarshalling data for every database interaction.
                > >
                > > Couldn't agree more.
                >
                > I couldn't agree less. Generated code is the biggest programming
                smell
                > ever. If code is so common it can be automatically generated, it can
                > surely be pulled up into a superclass.
                >
                > Storage can be perfectly handled in a storage layer. It needs not
                to be
                > more specific than that. If your application can store a row (may
                be a
                > collection or array) in a specific location and retrieve a row from
                a
                > location, then that is all that is necessary.
                >
                > Outside implementations of the storage library, there should not be
                any
                > SQL or even a notion of a database. The "table wrappers" communicate
                > with data lines, not with databases.
                > For example, I can decide to change the data line from a
                > DatabaseQueryLine to an IniFileDataLine at any moment for my
                > ApplicationSettings collection. And ApplicationSettings just
                inherits
                > from LazyTableWrapper and gets the data line passed in the
                constructor.
                > So there is no vast code duplication caused by some code spamming
                > generator, and there has never been any reason for it either.
                >
                > >> 4. Are SPs simply used as CRUD methods on a table by table
                basis? If
                > >> so, what's the real benefit? And, how does this scale as the
                data
                > >> model gets bigger and bigger? Most projects I've seen have 0..1
                > >> database developer and 1..* application developers!
                >
                > Sometimes. They may even start that way, like dumb-data-objects in
                > object-oriented programs evolve to functional entities. I expect the
                > stored procedures to evolve the same way.
                >
                > >> 6. How often do we change the physical data model without a
                > >> corresponding application change, unless it is purely a
                performance
                > >> optimization?
                >
                > In refactoring steps, quite often.
                >
                > >> I've worked on a range of projects using DB driven, OO and model
                > >> driven approaches. My assessment is, it does not have to be an
                all
                > >> or nothing design. We can use ORM to manage the routine CRUD
                work
                > >> and resort to SPs for bulk data processing, complex queries etc.
                > >> i.e, use a tool to do the mundane work and free ourselves to
                focus
                > >> on the fun stuff!
                >
                > Instead of having the code spammed to you by a generator, write a
                good
                > object model for the data handling. Now THAT is fun stuff!
                >
                >
                > Best regards,
                > Willem Bogaerts
                >
              • alexweatherall
                ... I m in a SQL Server shop, where we don t have access to Hibernate and don t autogenerate CRUD procedures, and I find that the number of simple CRUD
                Message 7 of 14 , Sep 26, 2006
                  > Model-driven devlopment is all about focusing on building the model
                  > and have the tools generate - quite a bit, if not all of the code.
                  > Consider a 500 table ERP application - what will be the effort
                  > required to created CRUD stored procedures for each table and how do
                  > you sell this to your architect or project manager :) ?
                  >
                  > Again, in a CRUD scenario, howis sp_insert/sp_update/sp_delete
                  > better than normal insert/update/delete - when they are all dealing
                  > with one row?
                  >
                  > > Instead of having the code spammed to you by a generator, write a
                  > > good object model for the data handling. Now THAT is fun stuff!
                  >
                  > That's exactly what persistence frameworks like Hibernate (java) or
                  > NHibernate (.NET)do. They use meta-data on how a class maps to a
                  > table and use it to prepare sql statements. This idea has been around
                  > since pre client/server days. Anyone remember Oracle Forms 2.3/3.0 or
                  > Power Builder? You didn't have to write a line of code to take what's
                  > on the screen and save it to the database and do the reverse.
                  >
                  >
                  > - Narayanan
                  >

                  I'm in a SQL Server shop, where we don't have access to Hibernate and
                  don't autogenerate CRUD procedures, and I find that the number of
                  simple CRUD procedures (that could be generated automatically) we
                  write are few and far between - we nearly always add extra
                  functionality, or checks and data business logic that is either
                  required for data validity or just appropriate to be dealt with at the
                  database layer.

                  There can be such an impedance between a relational model at the data
                  layer and an oo model at the business/application layer, that we tend
                  to write procedures that do the translation between the two models.
                  While there are areas where I can see we could automate the
                  development, they are few and far between. We have in the order of 250
                  tables and 1500 procedures across 3 databases, and we also have around
                  10-15 different applications accessing these databases.

                  I think if you have a tight match between your object model and your
                  relational model then you could use procedure generation to good
                  effect, but the amount of metadata we would have to create to allow
                  our database code to be generated automatically would *probably*
                  defeat the purpose.

                  As some one else said (I think it was in this thread) the balance of
                  resources between database developers and middle and top tier
                  developers can require the auto generation of CRUD procedures. However
                  in a large user base application, performance gains (due to server
                  round tripsare to be made by handcoding the logic at the stored
                  procedure level and not simply relying on O-R mapping and generation.

                  Maybe my experience is coloured by having only really worked on legacy
                  systems, with many coupled applications. I can see that when designing
                  an architecture and application model from the ground up, encompassing
                  all layers, then generation of code based on the model becomes more of
                  a reality.
                • Dave Sanders
                  ... Well, we ve thrown a lot of names out of software like Gentle, Hibernate, NHibernate. There is also NetTiers which I m familiar with, and tons of others
                  Message 8 of 14 , Sep 26, 2006
                    --- In agileDatabases@yahoogroups.com, "alexweatherall"
                    <alex.weatherall@...> wrote:
                    > Maybe my experience is coloured by having only really worked on legacy
                    > systems, with many coupled applications. I can see that when designing
                    > an architecture and application model from the ground up, encompassing
                    > all layers, then generation of code based on the model becomes more of
                    > a reality.

                    Well, we've thrown a lot of names out of software like Gentle,
                    Hibernate, NHibernate. There is also NetTiers which I'm familiar
                    with, and tons of others over at the sites for MyGeneration and
                    CodeSmith, for various languages / dbs.

                    I also do 100% SQL server and .NET, and think that even at my previous
                    company with its legacy app, we could have applied Gentle to at least
                    part of the architecture.

                    Depending upon what your coders are developing in, I'd still take a
                    look and at least understand how these tools work and what they
                    produce. Who knows, you might have some smaller add-on project later
                    that might work beautifully as a test case. At worst, you spend a
                    couple hours figuring out and generating a test case based on your
                    current DB. At best, you find a way to get rid of literally 1000s of
                    hours of senseless development.

                    D
                  • Adrian Walker
                    Alex -- You wrote... Maybe my experience is coloured by having only really worked on legacy systems, with many coupled applications. I can see that when
                    Message 9 of 14 , Sep 26, 2006
                      Alex --

                      You wrote...

                      Maybe my experience is coloured by having only really worked on legacy
                      systems, with many coupled applications. I can see that when designing
                      an architecture and application model from the ground up, encompassing
                      all layers, then generation of code based on the model becomes more of
                      a reality.

                      There is actually an approach to model=application that lends itelf well to code generation.

                      The approach specifies the model as business rules, runs the rules as the application, and generates and runs SQL that is too complex to write reliably by hand.

                      There's a paper about this [1], and the system that does this is live, online [2].  Shared use is free.

                                                                     Cheers,   -- Adrian


                      [1]  www.reengineeringllc.com/Oil_Industry_Supply_Chain_by_Kowalski_and_Walker.pdf

                      [2] Internet Business Logic (R)
                      Executable open vocabulary English
                      Online at www.reengineeringllc.com
                      Shared use is free

                      Adrian Walker
                      Reengineering
                      Phone: USA 860 830 2085


                      On 9/26/06, alexweatherall <alex.weatherall@...> wrote:


                      > Model-driven devlopment is all about focusing on building the model
                      > and have the tools generate - quite a bit, if not all of the code.
                      > Consider a 500 table ERP application - what will be the effort
                      > required to created CRUD stored procedures for each table and how do
                      > you sell this to your architect or project manager :) ?
                      >
                      > Again, in a CRUD scenario, howis sp_insert/sp_update/sp_delete
                      > better than normal insert/update/delete - when they are all dealing
                      > with one row?
                      >
                      > > Instead of having the code spammed to you by a generator, write a
                      > > good object model for the data handling. Now THAT is fun stuff!
                      >
                      > That's exactly what persistence frameworks like Hibernate (java) or
                      > NHibernate (.NET)do. They use meta-data on how a class maps to a
                      > table and use it to prepare sql statements. This idea has been around
                      > since pre client/server days. Anyone remember Oracle Forms 2.3/3.0 or
                      > Power Builder? You didn't have to write a line of code to take what's
                      > on the screen and save it to the database and do the reverse.
                      >
                      >
                      > - Narayanan
                      >

                      I'm in a SQL Server shop, where we don't have access to Hibernate and
                      don't autogenerate CRUD procedures, and I find that the number of
                      simple CRUD procedures (that could be generated automatically) we
                      write are few and far between - we nearly always add extra
                      functionality, or checks and data business logic that is either
                      required for data validity or just appropriate to be dealt with at the
                      database layer.

                      There can be such an impedance between a relational model at the data
                      layer and an oo model at the business/application layer, that we tend
                      to write procedures that do the translation between the two models.
                      While there are areas where I can see we could automate the
                      development, they are few and far between. We have in the order of 250
                      tables and 1500 procedures across 3 databases, and we also have around
                      10-15 different applications accessing these databases.

                      I think if you have a tight match between your object model and your
                      relational model then you could use procedure generation to good
                      effect, but the amount of metadata we would have to create to allow
                      our database code to be generated automatically would *probably*
                      defeat the purpose.

                      As some one else said (I think it was in this thread) the balance of
                      resources between database developers and middle and top tier
                      developers can require the auto generation of CRUD procedures. However
                      in a large user base application, performance gains (due to server
                      round tripsare to be made by handcoding the logic at the stored
                      procedure level and not simply relying on O-R mapping and generation.

                      Maybe my experience is coloured by having only really worked on legacy
                      systems, with many coupled applications. I can see that when designing
                      an architecture and application model from the ground up, encompassing
                      all layers, then generation of code based on the model becomes more of
                      a reality.


                    • Curt Sampson
                      ... Well, I hate generated code too, but some languages, such as Java, are so limited in power that the facilities the language itself offers just won t let
                      Message 10 of 14 , Sep 27, 2006
                        On Mon, 25 Sep 2006, Willem Bogaerts wrote:

                        > I couldn't agree less. Generated code is the biggest programming smell
                        > ever. If code is so common it can be automatically generated, it can
                        > surely be pulled up into a superclass.

                        Well, I hate generated code too, but some languages, such as Java, are
                        so limited in power that the facilities the language itself offers just
                        won't let you do the sorts of things you can do when you generate that
                        code from a specification in a more powerful language.

                        > Outside implementations of the storage library, there should not be any
                        > SQL or even a notion of a database.

                        This, I disagree with. The relational model is certainly more powerful,
                        flexible and easy to use than the OO model of "linking" things with
                        references between objects, since the relational model can essentially
                        generate on the fly any possible reference graphs you'd need. If you can
                        bring that into the OO world, you can save yourself a lot of reference
                        chasing, deciding where to put hash tables, etc.

                        cjs
                        --
                        Curt Sampson <cjs@...> +81 90 7737 2974
                        The power of accurate observation is commonly called cynicism
                        by those who have not got it. --George Bernard Shaw
                      • Steve Jorgensen
                        ... Everyone seems to feel the need to agree that they hate generated code. Certainly, there s a lot to hate about a lot of what s out there, but I think
                        Message 11 of 14 , Sep 28, 2006
                          Curt Sampson wrote:

                          On Mon, 25 Sep 2006, Willem Bogaerts wrote:

                          > I couldn't agree less. Generated code is the biggest programming smell
                          > ever. If code is so common it can be automatically generated, it can
                          > surely be pulled up into a superclass.

                          Well, I hate generated code too, but some languages, such as Java, are
                          so limited in power that the facilities the language itself offers just
                          won't let you do the sorts of things you can do when you generate that
                          code from a specification in a more powerful language.

                          Everyone seems to feel the need to agree that they hate generated code.  Certainly, there's a lot to hate about a lot of what's out there, but I think coming from a place of "I hate generated code" is limiting and potentially harmful, because in many cases, a well-written code generation scheme is not merely a lesser of evils, but a powerful and positive approach to solving a problem.

                          While I agree each case where code generation is called for indicates a limitation of the language being used, I also recognize that every language necessarily -does- have limitations, and each application has a unique combination of needs for which no ideal language has been or should be created.  It is often a more straightforward approach to work around language limitations using code generation than to force the language to behave by using complex and trouble-prone code structures or integrate with another language.

                          Of course, one can construct a DSL optimized for a particular problem, but that's often best implemented using, uhh, code generation.

                          Just my 2c.

                          - Steve J.

                        • Donald McLean
                          ... Domain Specific Languages (DSL) are the whole focus of the Meta Programming System, a project under development by those wacky but brilliant guys of
                          Message 12 of 14 , Sep 28, 2006
                            Steve Jorgensen wrote:
                            >
                            > While I agree each case where code generation is called for indicates a
                            > limitation of the language being used, I also recognize that every
                            > language necessarily -does- have limitations, and each application has a
                            > unique combination of needs for which no ideal language has been or
                            > should be created.
                            >
                            > Of course, one can construct a DSL optimized for a particular problem,
                            > but that's often best implemented using, uhh, code generation.

                            Domain Specific Languages (DSL) are the whole focus of the Meta
                            Programming System, a project under development by those wacky but
                            brilliant guys of JetBrains / IntelliJ. I wish that I had the time to
                            try it - if it's anywhere near as good as their Java development tools
                            then it could be a real step forward for some types of software.

                            As I understand it, they are attempting to build a system that doesn't
                            just create a compiler but also language specific editors and other
                            tools - probably even refactoring tools!

                            Donald
                          Your message has been successfully submitted and would be delivered to recipients shortly.