Loading ...
Sorry, an error occurred while loading the content.
 

[Ann] Article on evolutionary database design

Expand Messages
  • psadalage <psadalage@yahoo.com>
    My colleague Martin Fowler and I have written an article on Evolutionary Database Design see: http://www.martinfowler.com/articles/evodb.html Over the last
    Message 1 of 8 , Jan 1, 2003
      My colleague Martin Fowler and I have written an article on
      Evolutionary Database Design see:
      http://www.martinfowler.com/articles/evodb.html

      "Over the last few years we've developed a number of techniques that
      allow a database design to evolve as an application develops. This is
      a
      very important capability for agile methodologies. The techniques rely
      on applying continuous integration and automated refactoring to
      database
      development, together with a close collaboration between DBAs and
      application developers. The techniques work in both pre-production and
      released systems."
    • yuvaraj_a_r <yuva@excite.com>
      Hello, I found this an interesting article as I was thinking on adaptable database schemas for a while now. And I see that your proposal on evolutionary
      Message 2 of 8 , Jan 3, 2003
        Hello,

        I found this an interesting article as I was thinking on adaptable
        database schemas for a while now. And I see that your proposal on
        evolutionary database design could work!

        However, I have a concern.

        EAI Problem :
        -------------
        Many large scale enterprise systems have asynchoronous operations
        which could result in two applications having own databases and
        communicating changes through a middleware. Question then is how
        would the evolutionary mechanism extend to multiple database
        applications?

        I donot see directly how the lineage idea can be used as, typically,
        the release cycles of the applications are also different. Maybe if
        the direction of propagation of database change is defined as
        unidirectional, it could work(along with automation of the middleware
        transports). However, for getting maximum flexibility of multi-
        database applications, we would like to have the applications evolve
        independently as well.

        To make the idea concrete, lets take an example : Let there be a
        central ERP Application which has a defined database. Let there also
        be a asynchronous CRM Application which uses a middleware to
        synchornize with the ERP Application. Now the ERP and the CRM operate
        on different domains in terms of data volumes and devices (and
        possibly many more). The challenge would then be how to allow for the
        ERP Application and the CRM Application evolve independently to
        maximize utilization of the domain resources without(or minimaly)
        affecting integration.

        As mentioned above, if we define direction of change propagation as
        ERP -> Middleware -> CRM then lineage ideas could work. But, if CRM
        has to evolve at a higher speed than the ERP, we get the direction as
        CRM -> Middleware ?-> ERP. Unfortunately ERP systems are simply too
        big to change at the speed of CRM.

        So, how to handle evolutionary changes in EAI? Does evolving
        Middlewares make sense? And how would one construct such systems?

        Regards,
        yuva







        --- In agileDatabases@yahoogroups.com, "psadalage <psadalage@y...>"
        <psadalage@y...> wrote:
        > My colleague Martin Fowler and I have written an article on
        > Evolutionary Database Design see:
        > http://www.martinfowler.com/articles/evodb.html
        >
        > "Over the last few years we've developed a number of techniques that
        > allow a database design to evolve as an application develops. This
        is
        > a
        > very important capability for agile methodologies. The techniques
        rely
        > on applying continuous integration and automated refactoring to
        > database
        > development, together with a close collaboration between DBAs and
        > application developers. The techniques work in both pre-production
        and
        > released systems."
      • Scott W. Ambler
        ... From: To: Sent: Friday, January 03, 2003 9:03 AM Subject: [agileDatabases] Re: [Ann] Article on
        Message 3 of 8 , Jan 3, 2003
          ----- Original Message -----
          From: <yuva@...>
          To: <agileDatabases@yahoogroups.com>
          Sent: Friday, January 03, 2003 9:03 AM
          Subject: [agileDatabases] Re: [Ann] Article on evolutionary database design


          > Hello,
          >
          > I found this an interesting article as I was thinking on adaptable
          > database schemas for a while now. And I see that your proposal on
          > evolutionary database design could work!
          >
          > However, I have a concern.
          >
          > EAI Problem :
          > -------------
          > Many large scale enterprise systems have asynchoronous operations
          > which could result in two applications having own databases and
          > communicating changes through a middleware. Question then is how
          > would the evolutionary mechanism extend to multiple database
          > applications?

          I've done some writing about the multiple database problem at
          www.agiledata.org/essays/databaseRefactoring.html if it helps.

          >
          > I donot see directly how the lineage idea can be used as, typically,
          > the release cycles of the applications are also different. Maybe if
          > the direction of propagation of database change is defined as
          > unidirectional, it could work(along with automation of the middleware
          > transports). However, for getting maximum flexibility of multi-
          > database applications, we would like to have the applications evolve
          > independently as well.

          You need to have a reasonably long deprecation period that supports both the
          old and the new schema. The other application teams need to update their
          apps to use the new schema, and then deploy those apps, before the
          deprecation period is over. This requires a lot of negotiation and
          teamwork. The multiple database scenario, which is unfortunately incredibly
          common, is also unfortunately incredibly more difficult.

          >
          > To make the idea concrete, lets take an example : Let there be a
          > central ERP Application which has a defined database. Let there also
          > be a asynchronous CRM Application which uses a middleware to
          > synchornize with the ERP Application. Now the ERP and the CRM operate
          > on different domains in terms of data volumes and devices (and
          > possibly many more). The challenge would then be how to allow for the
          > ERP Application and the CRM Application evolve independently to
          > maximize utilization of the domain resources without(or minimaly)
          > affecting integration.

          Because the database(s) are shared resources they need a protocol to manage
          these resources effectively. This is why I'm a firm believer of the concept
          of contract models
          (www.agilemodeling.com/practices.htm#FormalizeContractModels) because it
          makes it clear that anyone involved with a shared resource needs to work
          together to evolve it successfully.

          >
          > As mentioned above, if we define direction of change propagation as
          > ERP -> Middleware -> CRM then lineage ideas could work. But, if CRM
          > has to evolve at a higher speed than the ERP, we get the direction as
          > CRM -> Middleware ?-> ERP. Unfortunately ERP systems are simply too
          > big to change at the speed of CRM.

          One, or both, teams will need to find ways to change their velocity (or the
          faster team may simply need to learn to live with the constraints imposed on
          them by the slower one).

          >
          > So, how to handle evolutionary changes in EAI? Does evolving
          > Middlewares make sense? And how would one construct such systems?
          >
          > Regards,
          > yuva
          - Scott
          =====================================================
          Scott W. Ambler scott.ambler@...
          Senior Consultant, Ronin International, Inc.
          www.ronin-intl.com/company/scottAmbler.html

          The Elements of UML Style is out! www.ambysoft.com/elementsUMLStyle.html

          www.agiledata.org * www.agilemodeling.com * www.ambysoft.com *
          www.enterpriseunifiedprocess.info * www.modelingstyle.info *
          www.ronin-intl.com
        • jhrothjr <yahoogroups@jhrothjr.com>
          ... I m not certain what you mean by middleware in this case. If you re talking about a layer that simply assures synchronization (and yes, there s nothing
          Message 4 of 8 , Jan 4, 2003
            --- In agileDatabases@yahoogroups.com, "yuvaraj_a_r <yuva@e...>" <yuva@e...> wrote:
            > Hello,
            >
            > I found this an interesting article as I was thinking on adaptable
            > database schemas for a while now. And I see that your proposal on
            > evolutionary database design could work!
            >
            > However, I have a concern.
            >
            > EAI Problem :
            > -------------
            > Many large scale enterprise systems have asynchoronous operations
            > which could result in two applications having own databases and
            > communicating changes through a middleware. Question then is how
            > would the evolutionary mechanism extend to multiple database
            > applications?
            >
            > I donot see directly how the lineage idea can be used as, typically,
            > the release cycles of the applications are also different. Maybe if
            > the direction of propagation of database change is defined as
            > unidirectional, it could work(along with automation of the middleware
            > transports). However, for getting maximum flexibility of multi-
            > database applications, we would like to have the applications evolve
            > independently as well.
            >
            > To make the idea concrete, lets take an example : Let there be a
            > central ERP Application which has a defined database. Let there also
            > be a asynchronous CRM Application which uses a middleware to
            > synchornize with the ERP Application. Now the ERP and the CRM operate
            > on different domains in terms of data volumes and devices (and
            > possibly many more). The challenge would then be how to allow for the
            > ERP Application and the CRM Application evolve independently to
            > maximize utilization of the domain resources without(or minimaly)
            > affecting integration.
            >
            > As mentioned above, if we define direction of change propagation as
            > ERP -> Middleware -> CRM then lineage ideas could work. But, if CRM
            > has to evolve at a higher speed than the ERP, we get the direction as
            > CRM -> Middleware ?-> ERP. Unfortunately ERP systems are simply too
            > big to change at the speed of CRM.
            >
            > So, how to handle evolutionary changes in EAI? Does evolving
            > Middlewares make sense? And how would one construct such systems?

            I'm not certain what you mean by middleware in this case. If you're talking about a layer that simply assures synchronization (and yes, there's nothing simple about that) then the other responses to your question apply.

            If that's the case, you need to slide another architectural layer in there somewhere so that no application is allowed to deal with any other application's data directly - that is, via SQL queries or anything equivalent.

            If all the application is allowed to do is make business related requests of another application, then migration is much simpler. Do whatever you want to the data base as long as the old requests continue to work.

            I realize this isn't an easy answer either to sell to management or to implement, but it does cut the coupling that creates the problems.

            John Roth

            >
            > Regards,
            > yuva
            >
            >
            >
            >
            >
            >
            >
            > --- In agileDatabases@yahoogroups.com, "psadalage <psadalage@y...>"
            > <psadalage@y...> wrote:
            > > My colleague Martin Fowler and I have written an article on
            > > Evolutionary Database Design see:
            > > http://www.martinfowler.com/articles/evodb.html
            > >
            > > "Over the last few years we've developed a number of techniques that
            > > allow a database design to evolve as an application develops. This
            > is
            > > a
            > > very important capability for agile methodologies. The techniques
            > rely
            > > on applying continuous integration and automated refactoring to
            > > database
            > > development, together with a close collaboration between DBAs and
            > > application developers. The techniques work in both pre-production
            > and
            > > released systems."
          • Gregor the Great <list@hohpe.com>
            You are right in that propagating changes in an EAI environment is a difficult topic for a number of reasons: - Lack of control over applications - One-to-many
            Message 5 of 8 , Jan 7, 2003
              You are right in that propagating changes in an EAI environment is a
              difficult topic for a number of reasons:
              - Lack of control over applications
              - One-to-many or bi-directional relationships
              - The sheer size of the solution

              For these reasons EAI cannot couple applications as tightly as an
              application is tied to its database. Rather, we need to define an
              intermediate layer of abstraction, sometimes referred to as the
              common domain model (or something similar). Messages travelling from
              one application (or the application's database) are then translated
              twice. First from the source app into the commonformat and then from
              the common format into the target app(s). This allows us to isolate
              changes (at least somewhat) if applications to evolve.

              The other aspect is that integration applications via their databases
              is in most cases the less preferred way. If possible, we would try to
              integrate using published interfaces or APIs because the software
              vendors tend to respect the portability of those a bit more when they
              roll out new releases. At the end of the day, though, you have to
              work with what the software vendor gives you and in many cases that
              may be just direct database access.

              These approaches are obviously no silver bullets, but they do help
              manage some of the dependencies.

              - Gregor

              --- In agileDatabases@yahoogroups.com, "jhrothjr <yahoogroups@j...>"
              <yahoogroups@j...> wrote:
              > --- In agileDatabases@yahoogroups.com, "yuvaraj_a_r <yuva@e...>"
              <yuva@e...> wrote:
              > > Hello,
              > >
              > > I found this an interesting article as I was thinking on
              adaptable
              > > database schemas for a while now. And I see that your proposal on
              > > evolutionary database design could work!
              > >
              > > However, I have a concern.
              > >
              > > EAI Problem :
              > > -------------
              > > Many large scale enterprise systems have asynchoronous operations
              > > which could result in two applications having own databases and
              > > communicating changes through a middleware. Question then is how
              > > would the evolutionary mechanism extend to multiple database
              > > applications?
              > >
              > > I donot see directly how the lineage idea can be used as,
              typically,
              > > the release cycles of the applications are also different. Maybe
              if
              > > the direction of propagation of database change is defined as
              > > unidirectional, it could work(along with automation of the
              middleware
              > > transports). However, for getting maximum flexibility of multi-
              > > database applications, we would like to have the applications
              evolve
              > > independently as well.
              > >
              > > To make the idea concrete, lets take an example : Let there be a
              > > central ERP Application which has a defined database. Let there
              also
              > > be a asynchronous CRM Application which uses a middleware to
              > > synchornize with the ERP Application. Now the ERP and the CRM
              operate
              > > on different domains in terms of data volumes and devices (and
              > > possibly many more). The challenge would then be how to allow for
              the
              > > ERP Application and the CRM Application evolve independently to
              > > maximize utilization of the domain resources without(or minimaly)
              > > affecting integration.
              > >
              > > As mentioned above, if we define direction of change propagation
              as
              > > ERP -> Middleware -> CRM then lineage ideas could work. But, if
              CRM
              > > has to evolve at a higher speed than the ERP, we get the
              direction as
              > > CRM -> Middleware ?-> ERP. Unfortunately ERP systems are simply
              too
              > > big to change at the speed of CRM.
              > >
              > > So, how to handle evolutionary changes in EAI? Does evolving
              > > Middlewares make sense? And how would one construct such systems?
              >
              > I'm not certain what you mean by middleware in this case. If you're
              talking about a layer that simply assures synchronization (and yes,
              there's nothing simple about that) then the other responses to your
              question apply.
              >
              > If that's the case, you need to slide another architectural layer
              in there somewhere so that no application is allowed to deal with any
              other application's data directly - that is, via SQL queries or
              anything equivalent.
              >
              > If all the application is allowed to do is make business related
              requests of another application, then migration is much simpler. Do
              whatever you want to the data base as long as the old requests
              continue to work.
              >
              > I realize this isn't an easy answer either to sell to management or
              to implement, but it does cut the coupling that creates the problems.
              >
              > John Roth
              >
              > >
              > > Regards,
              > > yuva
              > >
              > >
              > >
              > >
              > >
              > >
              > >
              > > --- In agileDatabases@yahoogroups.com, "psadalage
              <psadalage@y...>"
              > > <psadalage@y...> wrote:
              > > > My colleague Martin Fowler and I have written an article on
              > > > Evolutionary Database Design see:
              > > > http://www.martinfowler.com/articles/evodb.html
              > > >
              > > > "Over the last few years we've developed a number of techniques
              that
              > > > allow a database design to evolve as an application develops.
              This
              > > is
              > > > a
              > > > very important capability for agile methodologies. The
              techniques
              > > rely
              > > > on applying continuous integration and automated refactoring to
              > > > database
              > > > development, together with a close collaboration between DBAs
              and
              > > > application developers. The techniques work in both pre-
              production
              > > and
              > > > released systems."
            • johnc650
              I read your article below.. But I have to take exception with the You don t need a DBA section. Every company doing database work needs someone who has more
              Message 6 of 8 , Nov 4, 2005
                I read your article below.. But I have to take exception with
                the "You don't need a DBA" section. Every company doing database
                work needs someone who has more than a "passing interest" in
                databases. I cannot believe the companies I go to that have tables
                with no structure, no indexes and no concept of normal form. You may
                think "I'm a startup we will just redo the database later" but once
                the snowball starts rolling down the hill it gets really big and is
                hard to stop. I believe that an hour of DB design up front saves
                100's of hours down the road!!

                -jfc-

                --- In agileDatabases@yahoogroups.com, "psadalage <psadalage@y...>"
                <psadalage@y...> wrote:
                >
                > My colleague Martin Fowler and I have written an article on
                > Evolutionary Database Design see:
                > http://www.martinfowler.com/articles/evodb.html
                >
                > "Over the last few years we've developed a number of techniques
                that
                > allow a database design to evolve as an application develops. This
                is
                > a
                > very important capability for agile methodologies. The techniques
                rely
                > on applying continuous integration and automated refactoring to
                > database
                > development, together with a close collaboration between DBAs and
                > application developers. The techniques work in both pre-production
                and
                > released systems."
                >
              • johnc650
                I read your article below.. But I have to take exception with the You don t need a DBA section. Every company doing database work needs someone who has more
                Message 7 of 8 , Nov 4, 2005
                  I read your article below.. But I have to take exception with
                  the "You don't need a DBA" section. Every company doing database
                  work needs someone who has more than a "passing interest" in
                  databases. I cannot believe the companies I go to that have tables
                  with no structure, no indexes and no concept of normal form. You may
                  think "I'm a startup we will just redo the database later" but once
                  the snowball starts rolling down the hill it gets really big and is
                  hard to stop. I believe that an hour of DB design up front saves
                  100's of hours down the road!!

                  -jfc-

                  --- In agileDatabases@yahoogroups.com, "psadalage <psadalage@y...>"
                  <psadalage@y...> wrote:
                  >
                  > My colleague Martin Fowler and I have written an article on
                  > Evolutionary Database Design see:
                  > http://www.martinfowler.com/articles/evodb.html
                  >
                  > "Over the last few years we've developed a number of techniques
                  that
                  > allow a database design to evolve as an application develops. This
                  is
                  > a
                  > very important capability for agile methodologies. The techniques
                  rely
                  > on applying continuous integration and automated refactoring to
                  > database
                  > development, together with a close collaboration between DBAs and
                  > application developers. The techniques work in both pre-production
                  and
                  > released systems."
                  >
                • Scott W. Ambler
                  A better way to think about it is that we don t need a traditional DBA, what we need is an agile DBA. See www.agiledata.org for some ideas, including a range
                  Message 8 of 8 , Nov 4, 2005
                    A better way to think about it is that we don't need a traditional
                    DBA, what we need is an agile DBA. See www.agiledata.org for some
                    ideas, including a range of techniques, for agile dbas.

                    The Cutter group will have a management issue on agile database
                    techniques coming out in December which I'm editing and which Pramod
                    and I have a joint article in. Our article overviews agile data
                    techniques such as refactoring, TDD, sandboxes, and so on. Three
                    articles describe case studies, one where an agile dba came into a
                    project about a year into it and fixed a few things up (but if I
                    remember correctly didn't need to institute any new modeling
                    activities), another where an agile dba had to focus on TDD, and
                    another where there was no DBA needed at all due to superior
                    tooling. All of these articles were written by people with actual
                    experience on agile projects. A fifth article, written by someone
                    without agile experience, argued for the need for a lot more modeling
                    than I would recommend (I'm the guy behind Agile Modeling,
                    www.agilemodeling.com).

                    The sorts of things that Pramod wrote about in the article below do
                    seem to work very well in practice. I highly recommend that data
                    professionals take them seriously.

                    - Scott

                    At 01:07 PM 11/4/2005, you wrote:
                    >I read your article below.. But I have to take exception with
                    >the "You don't need a DBA" section. Every company doing database
                    >work needs someone who has more than a "passing interest" in
                    >databases. I cannot believe the companies I go to that have tables
                    >with no structure, no indexes and no concept of normal form. You may
                    >think "I'm a startup we will just redo the database later" but once
                    >the snowball starts rolling down the hill it gets really big and is
                    >hard to stop. I believe that an hour of DB design up front saves
                    >100's of hours down the road!!
                    >
                    >-jfc-
                    >
                    >--- In agileDatabases@yahoogroups.com, "psadalage <psadalage@y...>"
                    ><psadalage@y...> wrote:
                    > >
                    > > My colleague Martin Fowler and I have written an article on
                    > > Evolutionary Database Design see:
                    > > http://www.martinfowler.com/articles/evodb.html
                    > >
                  Your message has been successfully submitted and would be delivered to recipients shortly.