Loading ...
Sorry, an error occurred while loading the content.

Migrating data and code from legacy system to a new one

Expand Messages
  • amandavarell
    Hi, I´m new to this group. Now I´m involved in a decision of a migration of a legacy system. Here is the scenarium: 1 - The system is written in VB6 2 - It
    Message 1 of 16 , Jul 18 3:11 PM
    • 0 Attachment
      Hi,

      I´m new to this group. Now I´m involved in a decision of a migration of a legacy system. Here is the scenarium:
      1 - The system is written in VB6
      2 - It uses an oracle database
      3 - It will be migrated from VB6 to .NET
      4 - The schema will be moved to another database, where naming conventions will be applied, tables can be a little bit different... in other words, the new schema will be a little bit different than the old one

      Talking to the team, they said that would be necessary 1 year to migrate the whole system. But, usually we tend to give very optimistics estimates, so I imagine that this time can be twice or three times higher, so, a big bang approach is not a option.

      I would like to know your opinions. Had anybody here had similar experience? I would like to test approachs like:
      - New system, new database. As I update the new database the old one is also updated. What should I use? Triggers? Views?

      I apprecciate any ideas.

      Thank you
    • David Poole
      Timescales for migrating from an old system to new depend to a large extent on how well the existing system is known and understood. There needs to be a common
      Message 2 of 16 , Jul 19 12:44 AM
      • 0 Attachment
        Timescales for migrating from an old system to new depend to a large
        extent on how well the existing system is known and understood.

        There needs to be a common understanding between business users and IT
        people of how the application and the way in which data it contains is
        recorded.



        If you have two systems that you have to keep in sync then you need to
        devote a lot of time to designing and testing the process that will do
        so. If the system is complex you will find yourself getting into ESB
        (Enterprise Service Bus) territory.



        Unless you use views on top of the legacy schema to present the data as
        you would like it in the new schema you are going to have to do a
        mapping exercise for every attribute describing how it gets transformed
        into the new schema. I've just spent two months doing precisely that
        for a relatively modest schema. It has been a real eye opener as to how
        long such a process takes and the sheer amount of work that is involved.



        Shifting technology platforms is also something to be wary of. If you
        have experienced .NET developers then you may be in luck. If your team
        are legacy VB programmers who are converting to .NET then you have to
        factor in the learning curve.







        David Poole
        Data Architect

        MoneySupermarket.com
        <http://www.linkedin.com/company/moneysupermarket.com>
        <http://www.facebook.com/moneysupermarket>
        <http://twitter.com/#%21/moneysupermkt>
        MoneySupermarket House | St. David's Park | Ewloe | Flintshire | CH5 3UZ
        | UK
        Tel: +44(0) 1244 665700 x(2691) | Mob: +44(0) 7743272397
        For details on jobs at MoneySupermarket.com click here
        <http://www.moneysupermarketjobs.com/>
        <http://www.moneysupermarket.com/>
        <http://www.bestcompaniesguide.co.uk/company_profile.aspx?CompanySurveyI
        D=42711> <http://www.walesairambulance.com/>




        The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information, or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Although this e-mail and any attachments are believed to be free of any virus, or other defect which might affect any computer or system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Moneysupermarket.com Financial Group Limited for any loss or damage from receipt or use thereof. The views expressed are of the individual, and do not necessarily reflect the views of Moneysupermarket.com Financial Group Limited.
        Moneysupermarket.com Limited is an appointed representative of Moneysupermarket.com Financial Group Limited, which is authorised and regulated by the Financial Services Authority (FSA FRN 303190).
        Moneysupermarket.com Financial Group Limited, registered in England No. 3157344. Registered Office: Moneysupermarket House, St. David’s Park, Ewloe, CH5 3UZ. Telephone 01244 665700.



        [Non-text portions of this message have been removed]
      • w-p
        ... You are not working on a live database are you? With enough database abstraction, you could switch databases, but a switch-while-running is a huge lot of
        Message 3 of 16 , Jul 19 12:45 AM
        • 0 Attachment
          > .... As I update the new database the old one is also updated. What should I use? Triggers? Views?

          You are not working on a live database are you?

          With enough database abstraction, you could switch databases, but a
          switch-while-running is a huge lot of effort.

          You could switch in pieces, so running two databases. If you log errors
          to a log table, it is usually a safe table to begin with.

          But I would stop the users, migrate the table, and then tell the users
          they can upgrade.

          Best regards,
          --
          Willem Bogaerts

          Applicatiesmid
          Kratz B.V.
          http://www.kratz.nl/
        • Vasco Figueira
          Hi, ... If: 1. the new system will be using the new schema; 2. you can tolerate a downtime of a few hours; 3. you have a test site; 4. the old system is not
          Message 4 of 16 , Jul 19 3:24 AM
          • 0 Attachment
            Hi,

            On 18 July 2011 23:11, amandavarell <amandavarella@...> wrote:
            > I´m new to this group. Now I´m involved in a decision of a migration of a legacy system. Here is the
            > scenarium:
            > 1 - The system is written in VB6
            > 2 - It uses an oracle database
            > 3 - It will be migrated from VB6 to .NET
            > 4 - The schema will be moved to another database, where naming conventions will be applied, tables
            > can be a little bit different... in other words, the new schema will be a little bit different than the old
            > one
            >
            > Talking to the team, they said that would be necessary 1 year to migrate the whole system. But,
            > usually we tend to give very optimistics estimates, so I imagine that this time can be twice or three
            > times higher, so, a big bang approach is not a option.

            If:
            1. the new system will be using the new schema;
            2. you can tolerate a downtime of a few hours;
            3. you have a test site;
            4. the old system is not going to be used once the new starts,

            I would suggest to simply develop a good set of migration scripts.
            Test them carefully and it's probably your best solution.

            If (I noticed it now), .4 doesn't hold and you have to have both
            systems in sync, then either:
            1. You deal with the mismatch of both schemas in runtime, or;
            2. You evolve the old schema to match the new one (in the common parts).

            .1 becomes easily unmanageable if your schema is complex. You will
            probably develop a lot of procedural code to maintain integrity across
            the two schemas. That effort may be better spent elsewhere.

            .2 is an incremental approach to approximate the two. If you get it
            done right, i.e. you achieve schema compatibility you can even have
            the two applications running on top of the same schema, the old and
            the new.

            This latter approach is well detailed in the book "Refactoring
            Databases : Evolutionary Database Design".

            http://databaserefactoring.com/

            > I would like to know your opinions. Had anybody here had similar experience? I would like to test
            > approachs like:
            > - New system, new database. As I update the new database the old one is also updated. What
            > should I use? Triggers? Views?

            Triggers and views are two things that indeed help you achieve schema
            compatibility. The mentioned book is your friend.

            Vasco Figueira
            Consultor Especialista Java
            KnowledgeWorks - Consultoria em Sistemas de Informação

            Av. Eng. Arantes e Oliveira, nº 11, 4º A
            1900-221 Olaias

            Lisboa - Portugal

            Tel : 21 247 31 34
            Fax: 21 010 23 09
            Tel: 91 428 28 34

            http://www.knowledgeworks.pt
          • Pramod Sadalage
            You don t mention what kind of system is this. Web facing 24*7 or fat clients locally installed etc. As Willem said, it will be lot of work to keep both the
            Message 5 of 16 , Jul 19 7:12 AM
            • 0 Attachment
              You don't mention what kind of system is this.

              Web facing 24*7 or fat clients locally installed etc.
              As Willem said, it will be lot of work to keep both the databases in-sync
              while they are live. One other track I would look at is why move from VB6 to
              .NET and apply database naming standards at the same time.

              You could move to .NET first with the original naming convention and then
              when thats stable.. move to apply the new naming convention. or some other
              combination of work streams.

              Cheers
              *Pramod Sadalage
              @pramodsadalage
              <http://www.twitter.com/pramodsadalage><http://www.sadalage.com/>
              *
              *www.sadalage.com
              www.databaserefactoring.com*



              On Tue, Jul 19, 2011 at 2:45 AM, w-p <w-p@...> wrote:

              > **
              >
              >
              > > .... As I update the new database the old one is also updated. What
              > should I use? Triggers? Views?
              >
              > You are not working on a live database are you?
              >
              > With enough database abstraction, you could switch databases, but a
              > switch-while-running is a huge lot of effort.
              >
              > You could switch in pieces, so running two databases. If you log errors
              > to a log table, it is usually a safe table to begin with.
              >
              > But I would stop the users, migrate the table, and then tell the users
              > they can upgrade.
              >
              > Best regards,
              > --
              > Willem Bogaerts
              >
              > Applicatiesmid
              > Kratz B.V.
              > http://www.kratz.nl/
              >
              >


              [Non-text portions of this message have been removed]
            • Amanda Varella
              Thank you for your answers. Completing my scenarium: 1 - It´s a information system, whose purpose is to have a strategic project portfolio and helps
              Message 6 of 16 , Jul 19 1:52 PM
              • 0 Attachment
                Thank you for your answers. Completing my scenarium:

                1 - It´s a information system, whose purpose is to have a strategic project
                portfolio and helps executives to make decisions based on simulations
                2 - It has been developed for 10 years
                3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                4 - The changes in the database will not be restricted to naming convetions,
                but also to new semantics and new organization of tables (these could
                profound changes, for a better database organization)
                5 - There will be a migration from a local database to a enterprise database
                (integration of some concepts)
                6 - We would like to have an improvement of the OO model (actually, creating
                one from scratch), to have a more maintenable application

                One of the strategies that we are thinking is to have is the new application
                pointing to the old database
                and build a transformation layer, that receives the data in a good design
                (to the new database and model) and transforms to the old tables. After we
                change from VB6 to .NET, we could point the application to the new database
                without the transformation step.

                Caveats: We are afraid of migrate to the new database only at the end of the
                proccess and have an integration problem, ex: The OO model could not fit
                with the new database (impedance mismatch)

                Thank you
                Amanda


                On Tue, Jul 19, 2011 at 11:12 AM, Pramod Sadalage
                <pramodsadalage@...>wrote:

                > You don't mention what kind of system is this.
                >
                > Web facing 24*7 or fat clients locally installed etc.
                > As Willem said, it will be lot of work to keep both the databases in-sync
                > while they are live. One other track I would look at is why move from VB6
                > to
                > .NET and apply database naming standards at the same time.
                >
                > You could move to .NET first with the original naming convention and then
                > when thats stable.. move to apply the new naming convention. or some other
                > combination of work streams.
                >



                >
                > Cheers
                > *Pramod Sadalage
                > @pramodsadalage
                > <http://www.twitter.com/pramodsadalage><http://www.sadalage.com/>
                > *
                > *www.sadalage.com
                > www.databaserefactoring.com*
                >
                >
                >
                > On Tue, Jul 19, 2011 at 2:45 AM, w-p <w-p@...> wrote:
                >
                > > **
                > >
                > >
                > > > .... As I update the new database the old one is also updated. What
                > > should I use? Triggers? Views?
                > >
                > > You are not working on a live database are you?
                > >
                > > With enough database abstraction, you could switch databases, but a
                > > switch-while-running is a huge lot of effort.
                > >
                > > You could switch in pieces, so running two databases. If you log errors
                > > to a log table, it is usually a safe table to begin with.
                > >
                > > But I would stop the users, migrate the table, and then tell the users
                > > they can upgrade.
                > >
                > > Best regards,
                > > --
                > > Willem Bogaerts
                > >
                > > Applicatiesmid
                > > Kratz B.V.
                > > http://www.kratz.nl/
                > >
                > >
                >
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >
                > ------------------------------------
                >
                > Yahoo! Groups Links
                >
                >
                >
                >


                --
                Amanda Varella


                [Non-text portions of this message have been removed]
              • Pramod Sadalage
                In that case its actually a re-write and you want to migrate data.. may be this will help http://martinfowler.com/bliki/IncrementalMigration.html Cheers
                Message 7 of 16 , Jul 19 2:01 PM
                • 0 Attachment
                  In that case its actually a re-write and you want to migrate data..
                  may be this will help
                  http://martinfowler.com/bliki/IncrementalMigration.html

                  Cheers
                  *Pramod Sadalage
                  @pramodsadalage
                  <http://www.twitter.com/pramodsadalage><http://www.sadalage.com/>
                  *
                  *www.sadalage.com
                  www.databaserefactoring.com*



                  On Tue, Jul 19, 2011 at 3:52 PM, Amanda Varella <amandavarella@...>wrote:

                  > **
                  >
                  >
                  > Thank you for your answers. Completing my scenarium:
                  >
                  > 1 - It�s a information system, whose purpose is to have a strategic project
                  > portfolio and helps executives to make decisions based on simulations
                  > 2 - It has been developed for 10 years
                  > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                  > 4 - The changes in the database will not be restricted to naming
                  > convetions,
                  > but also to new semantics and new organization of tables (these could
                  > profound changes, for a better database organization)
                  > 5 - There will be a migration from a local database to a enterprise
                  > database
                  > (integration of some concepts)
                  > 6 - We would like to have an improvement of the OO model (actually,
                  > creating
                  > one from scratch), to have a more maintenable application
                  >
                  > One of the strategies that we are thinking is to have is the new
                  > application
                  > pointing to the old database
                  > and build a transformation layer, that receives the data in a good design
                  > (to the new database and model) and transforms to the old tables. After we
                  > change from VB6 to .NET, we could point the application to the new database
                  > without the transformation step.
                  >
                  > Caveats: We are afraid of migrate to the new database only at the end of
                  > the
                  > proccess and have an integration problem, ex: The OO model could not fit
                  > with the new database (impedance mismatch)
                  >
                  > Thank you
                  > Amanda
                  >
                  > On Tue, Jul 19, 2011 at 11:12 AM, Pramod Sadalage
                  > <pramodsadalage@...>wrote:
                  >
                  >
                  > > You don't mention what kind of system is this.
                  > >
                  > > Web facing 24*7 or fat clients locally installed etc.
                  > > As Willem said, it will be lot of work to keep both the databases in-sync
                  > > while they are live. One other track I would look at is why move from VB6
                  > > to
                  > > .NET and apply database naming standards at the same time.
                  > >
                  > > You could move to .NET first with the original naming convention and then
                  > > when thats stable.. move to apply the new naming convention. or some
                  > other
                  > > combination of work streams.
                  > >
                  >
                  > >
                  > > Cheers
                  > > *Pramod Sadalage
                  > > @pramodsadalage
                  > > <http://www.twitter.com/pramodsadalage><http://www.sadalage.com/>
                  > > *
                  > > *www.sadalage.com
                  >
                  > > www.databaserefactoring.com*
                  > >
                  > >
                  > >
                  > > On Tue, Jul 19, 2011 at 2:45 AM, w-p <w-p@...> wrote:
                  > >
                  > > > **
                  > > >
                  > > >
                  > > > > .... As I update the new database the old one is also updated. What
                  > > > should I use? Triggers? Views?
                  > > >
                  > > > You are not working on a live database are you?
                  > > >
                  > > > With enough database abstraction, you could switch databases, but a
                  > > > switch-while-running is a huge lot of effort.
                  > > >
                  > > > You could switch in pieces, so running two databases. If you log errors
                  > > > to a log table, it is usually a safe table to begin with.
                  > > >
                  > > > But I would stop the users, migrate the table, and then tell the users
                  > > > they can upgrade.
                  > > >
                  > > > Best regards,
                  > > > --
                  > > > Willem Bogaerts
                  > > >
                  > > > Applicatiesmid
                  > > > Kratz B.V.
                  > > > http://www.kratz.nl/
                  > > >
                  > > >
                  > >
                  > >
                  > > [Non-text portions of this message have been removed]
                  > >
                  > >
                  > >
                  > > ------------------------------------
                  > >
                  > > Yahoo! Groups Links
                  > >
                  > >
                  > >
                  > >
                  >
                  > --
                  > Amanda Varella
                  >
                  > [Non-text portions of this message have been removed]
                  >
                  >
                  >


                  [Non-text portions of this message have been removed]
                • David Poole
                  There is a danger when rewriting legacy systems to assume that legacy means not very good . I ve experienced a major rewrite that addressed the weaknesses
                  Message 8 of 16 , Jul 20 12:38 AM
                  • 0 Attachment
                    There is a danger when rewriting legacy systems to assume that "legacy" means "not very good".



                    I've experienced a major rewrite that addressed the weaknesses in the old system but failed to capture its strengths. On top of that it introduced a whole new load of weaknesses.



                    I've found that legacy systems tend to gain something analogous to middle-aged spread. At the central core stuff is still pretty healthy but over time the systems has got a bit flabby, gained a few warts and bunions etc.



                    From: agileDatabases@yahoogroups.com [mailto:agileDatabases@yahoogroups.com] On Behalf Of Amanda Varella
                    Sent: 19 July 2011 21:52
                    To: agileDatabases@yahoogroups.com
                    Subject: Re: [agileDatabases] Migrating data and code from legacy system to a new one





                    Thank you for your answers. Completing my scenarium:

                    1 - It´s a information system, whose purpose is to have a strategic project
                    portfolio and helps executives to make decisions based on simulations
                    2 - It has been developed for 10 years
                    3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                    4 - The changes in the database will not be restricted to naming convetions,
                    but also to new semantics and new organization of tables (these could
                    profound changes, for a better database organization)
                    5 - There will be a migration from a local database to a enterprise database
                    (integration of some concepts)
                    6 - We would like to have an improvement of the OO model (actually, creating
                    one from scratch), to have a more maintenable application

                    One of the strategies that we are thinking is to have is the new application
                    pointing to the old database
                    and build a transformation layer, that receives the data in a good design
                    (to the new database and model) and transforms to the old tables. After we
                    change from VB6 to .NET, we could point the application to the new database
                    without the transformation step.

                    Caveats: We are afraid of migrate to the new database only at the end of the
                    proccess and have an integration problem, ex: The OO model could not fit
                    with the new database (impedance mismatch)

                    Thank you
                    Amanda

                    On Tue, Jul 19, 2011 at 11:12 AM, Pramod Sadalage
                    <pramodsadalage@... <mailto:pramodsadalage%40gmail.com> >wrote:

                    > You don't mention what kind of system is this.
                    >
                    > Web facing 24*7 or fat clients locally installed etc.
                    > As Willem said, it will be lot of work to keep both the databases in-sync
                    > while they are live. One other track I would look at is why move from VB6
                    > to
                    > .NET and apply database naming standards at the same time.
                    >
                    > You could move to .NET first with the original naming convention and then
                    > when thats stable.. move to apply the new naming convention. or some other
                    > combination of work streams.
                    >

                    >
                    > Cheers
                    > *Pramod Sadalage
                    > @pramodsadalage
                    > <http://www.twitter.com/pramodsadalage><http://www.sadalage.com/>
                    > *
                    > *www.sadalage.com
                    > www.databaserefactoring.com*
                    >
                    >
                    >
                    > On Tue, Jul 19, 2011 at 2:45 AM, w-p <w-p@... <mailto:w-p%40dds.nl> > wrote:
                    >
                    > > **
                    > >
                    > >
                    > > > .... As I update the new database the old one is also updated. What
                    > > should I use? Triggers? Views?
                    > >
                    > > You are not working on a live database are you?
                    > >
                    > > With enough database abstraction, you could switch databases, but a
                    > > switch-while-running is a huge lot of effort.
                    > >
                    > > You could switch in pieces, so running two databases. If you log errors
                    > > to a log table, it is usually a safe table to begin with.
                    > >
                    > > But I would stop the users, migrate the table, and then tell the users
                    > > they can upgrade.
                    > >
                    > > Best regards,
                    > > --
                    > > Willem Bogaerts
                    > >
                    > > Applicatiesmid
                    > > Kratz B.V.
                    > > http://www.kratz.nl/
                    > >
                    > >
                    >
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >
                    > ------------------------------------
                    >
                    > Yahoo! Groups Links
                    >
                    >
                    >
                    >

                    --
                    Amanda Varella

                    [Non-text portions of this message have been removed]




                    The information contained in this message may be CONFIDENTIAL and is intended for the addressee only. Any unauthorised use, dissemination of the information, or copying of this message is prohibited. If you are not the addressee, please notify the sender immediately by return e-mail and delete this message. Although this e-mail and any attachments are believed to be free of any virus, or other defect which might affect any computer or system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Moneysupermarket.com Financial Group Limited for any loss or damage from receipt or use thereof. The views expressed are of the individual, and do not necessarily reflect the views of Moneysupermarket.com Financial Group Limited.
                    Moneysupermarket.com Limited is an appointed representative of Moneysupermarket.com Financial Group Limited, which is authorised and regulated by the Financial Services Authority (FSA FRN 303190).
                    Moneysupermarket.com Financial Group Limited, registered in England No. 3157344. Registered Office: Moneysupermarket House, St. David’s Park, Ewloe, CH5 3UZ. Telephone 01244 665700.



                    [Non-text portions of this message have been removed]
                  • w-p
                    ... Often, you can have a transformation layer in the database as well as in the code. So you can define a set of stored routines that you develop against, or
                    Message 9 of 16 , Jul 20 1:43 AM
                    • 0 Attachment
                      > One of the strategies that we are thinking is to have is the new application
                      > pointing to the old database
                      > and build a transformation layer

                      Often, you can have a transformation layer in the database as well as in
                      the code. So you can define a set of stored routines that you develop
                      against, or two implementations in the client. I have worked for a
                      company where we were only allowed to communicate to the database
                      through stored routines. That was because of security considerations,
                      but you could use this for structural freedom as well.

                      Good luck,
                      --
                      Willem Bogaerts

                      Applicatiesmid
                      Kratz B.V.
                      http://www.kratz.nl/
                    • Amanda Varella
                      Pramod, very interesting the approach to incremental migration. Could be a very good strategy. In my context, I see I still will have the following challenges:
                      Message 10 of 16 , Jul 20 5:10 AM
                      • 0 Attachment
                        Pramod,

                        very interesting the approach to incremental migration. Could be a very good
                        strategy. In my context, I see I still will have the following challenges:
                        - The new application will access the new database and the old application
                        will access the old one database.
                        - There will be some tables that will be so used that both systems (new and
                        old) will have to use it. So, I still will have a synchronization effort (I
                        think I can�t get away from this)


                        David,

                        you have a good point. Of course not all the schema is so bad. But we are
                        sure that some refactorings must be made.
                        Other points:
                        - The old system is well known and understood
                        - You�ve warned me correctly. The team is formed by VB programmers and will
                        have the learning curve for .NET. But at this moment, the time to migrate
                        it�s not such a big thing. The current system is live and working
                        - One of the reasons I would like to maintain both systems live is that I
                        don�t want to wait, maybe three years to have the new system been used. We
                        never know how much time it takes, and if there is no real use, in
                        production enviroment, many bugs, integration problems can occur.
                        - Besides this, new features are required, and I want to minimize the
                        problem of one system always behind another, and have two efforts of
                        implementations (in the old one and the new one)

                        Vasco,

                        1. Yes, the new system will use the new schema
                        2. Yes, I think I can tolerate some downtime
                        3. We plan to have a test suite
                        4. Once the whole migration is completed, the old one will be turned off.

                        It is a complex schema.
                        The only way to have the two applications running on the same schema is to
                        have the new application runnning on the old database, because we don�t want
                        to alter the VB6 code. But we also want to change some structures of the
                        database for the new version.

                        Thank you all
                        I apprecciate new ideas based on that restrictions

                        PS.: Just bought Refactoring Databases

                        On Wed, Jul 20, 2011 at 5:43 AM, w-p <w-p@...> wrote:

                        > **
                        >
                        >
                        > > One of the strategies that we are thinking is to have is the new
                        > application
                        > > pointing to the old database
                        > > and build a transformation layer
                        >
                        > Often, you can have a transformation layer in the database as well as in
                        > the code. So you can define a set of stored routines that you develop
                        > against, or two implementations in the client. I have worked for a
                        > company where we were only allowed to communicate to the database
                        > through stored routines. That was because of security considerations,
                        > but you could use this for structural freedom as well.
                        >
                        > Good luck,
                        > --
                        > Willem Bogaerts
                        >
                        > Applicatiesmid
                        > Kratz B.V.
                        > http://www.kratz.nl/
                        >
                        >
                        >



                        --
                        Amanda Varella


                        [Non-text portions of this message have been removed]
                      • Don Albertson
                        ... My operational definition of legacy is using hardware or software that is no longer supported by the manufacturer/vendor . For example, about 9 years
                        Message 11 of 16 , Jul 20 5:28 AM
                        • 0 Attachment
                          On 7/20/2011 3:38 AM, David Poole wrote:
                          > There is a danger when rewriting legacy systems to assume that "legacy" means "not very good".
                          >
                          >
                          >
                          > I've experienced a major rewrite that addressed the weaknesses in the old system but failed to capture its strengths. On top of that it introduced a whole new load of weaknesses.
                          >
                          >
                          >
                          > I've found that legacy systems tend to gain something analogous to middle-aged spread. At the central core stuff is still pretty healthy but over time the systems has got a bit flabby, gained a few warts and bunions etc.
                          >
                          My operational definition of "legacy" is "using hardware or software that is no longer supported by the manufacturer/vendor". For example, about 9 years ago we purchased a software package version 4 which worked with Oracle 8. Various managers opted not to upgrade to versions 5 and 6. In the meantime, we created our own integration packages to exchange data between this package and some other packages from other vendors. Now another vendor owns the code and doesn't support version 4 which has issues running on Windows 7, won't connect with Oracle 10+. The code still does everything it was intended to do but it's a legacy.
                        • deddy205ar
                          Amanda - Will consideration be given to the existing data in the database? There s an activity called data profiling that looks at the actual data in every
                          Message 12 of 16 , Aug 30, 2011
                          • 0 Attachment
                            Amanda -

                            Will consideration be given to the existing data in the database? There's an activity called
                            "data profiling" that looks at the actual data in every single column & row. Typically
                            surprises will be found.

                            I remember being burned when doing a mutual fund share split... and I *** ASSUMED *** that shares could only be positive. Oh, wrongo!

                            So look deep, look close.

                            - David


                            --- In agileDatabases@yahoogroups.com, "amandavarell" <amandavarella@...> wrote:
                            >
                            > Hi,
                            >
                            > I´m new to this group. Now I´m involved in a decision of a migration of a legacy system. Here is the scenarium:
                            > 1 - The system is written in VB6
                            > 2 - It uses an oracle database
                            > 3 - It will be migrated from VB6 to .NET
                            > 4 - The schema will be moved to another database, where naming conventions will be applied, tables can be a little bit different... in other words, the new schema will be a little bit different than the old one
                            >
                            > Talking to the team, they said that would be necessary 1 year to migrate the whole system. But, usually we tend to give very optimistics estimates, so I imagine that this time can be twice or three times higher, so, a big bang approach is not a option.
                            >
                            > I would like to know your opinions. Had anybody here had similar experience? I would like to test approachs like:
                            > - New system, new database. As I update the new database the old one is also updated. What should I use? Triggers? Views?
                            >
                            > I apprecciate any ideas.
                            >
                            > Thank you
                            >
                          • Paul
                            Amanda, We ve been in the process of a legacy system conversion such as yours now for the last 3 years. Coordinating the rewrite of over a dozen interconnected
                            Message 13 of 16 , Sep 11, 2011
                            • 0 Attachment
                              Amanda,

                              We've been in the process of a legacy system conversion such as yours now for the last 3 years. Coordinating the rewrite of over a dozen interconnected systems. Like yourself, reorganizing schemas, tables, columns to bring the new system online in the most efficient way possible.

                              We've had to write both SSIS (MSSQL to MSSQL) and custom conversions (C# .Net) ETL's to migrate the data from the old to the new and custom SQL validations to prove that the data all made it over.

                              All I can say is make data your highest priority in the conversion.

                              New code can't be fully tested until you have the migrated data as well as any new data you add on top of it. We run a weekly ETL to prove that the existing data can come into the new system correctly.

                              After 2 years we had to spend the last month catching up on the data conversion part, as we had written code/data stubs to keep the code moving while the data conversion parts got further and further behind. Not Fun.

                              As the other poster said, look close look deep, make sure every bit is accounted for.

                              Paul


                              --- In agileDatabases@yahoogroups.com, "deddy205ar" <deddy@...> wrote:
                              >
                              > Amanda -
                              >
                              > Will consideration be given to the existing data in the database? There's an activity called
                              > "data profiling" that looks at the actual data in every single column & row. Typically
                              > surprises will be found.
                              >
                              > I remember being burned when doing a mutual fund share split... and I *** ASSUMED *** that shares could only be positive. Oh, wrongo!
                              >
                              > So look deep, look close.
                              >
                              > - David
                              >
                              >
                              > --- In agileDatabases@yahoogroups.com, "amandavarell" <amandavarella@> wrote:
                              > >
                              > > Hi,
                              > >
                              > > I´m new to this group. Now I´m involved in a decision of a migration of a legacy system. Here is the scenarium:
                              > > 1 - The system is written in VB6
                              > > 2 - It uses an oracle database
                              > > 3 - It will be migrated from VB6 to .NET
                              > > 4 - The schema will be moved to another database, where naming conventions will be applied, tables can be a little bit different... in other words, the new schema will be a little bit different than the old one
                              > >
                              > > Talking to the team, they said that would be necessary 1 year to migrate the whole system. But, usually we tend to give very optimistics estimates, so I imagine that this time can be twice or three times higher, so, a big bang approach is not a option.
                              > >
                              > > I would like to know your opinions. Had anybody here had similar experience? I would like to test approachs like:
                              > > - New system, new database. As I update the new database the old one is also updated. What should I use? Triggers? Views?
                              > >
                              > > I apprecciate any ideas.
                              > >
                              > > Thank you
                              > >
                              >
                            • wbrianwhite
                              Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the
                              Message 14 of 16 , Feb 23, 2012
                              • 0 Attachment
                                Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.

                                Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.

                                Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.

                                Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.

                                Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.

                                --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@...> wrote:
                                >
                                > Thank you for your answers. Completing my scenarium:
                                >
                                > 1 - It´s a information system, whose purpose is to have a strategic project
                                > portfolio and helps executives to make decisions based on simulations
                                > 2 - It has been developed for 10 years
                                > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                                > 4 - The changes in the database will not be restricted to naming convetions,
                                > but also to new semantics and new organization of tables (these could
                                > profound changes, for a better database organization)
                                > 5 - There will be a migration from a local database to a enterprise database
                                > (integration of some concepts)
                                > 6 - We would like to have an improvement of the OO model (actually, creating
                                > one from scratch), to have a more maintenable application
                                >
                                > One of the strategies that we are thinking is to have is the new application
                                > pointing to the old database
                                > and build a transformation layer, that receives the data in a good design
                                > (to the new database and model) and transforms to the old tables. After we
                                > change from VB6 to .NET, we could point the application to the new database
                                > without the transformation step.
                                >
                                > Caveats: We are afraid of migrate to the new database only at the end of the
                                > proccess and have an integration problem, ex: The OO model could not fit
                                > with the new database (impedance mismatch)
                                >
                                > Thank you
                                > Amanda
                                >
                                >
                              • ROBERT
                                This is a great post. I hope Amanda is still watching! Bob Corrick.
                                Message 15 of 16 , Feb 24, 2012
                                • 0 Attachment
                                  This is a great post.
                                  I hope Amanda is still watching!
                                  Bob Corrick.

                                  --- In agileDatabases@yahoogroups.com, "wbrianwhite" <wbrianwhite@...> wrote:
                                  >
                                  >
                                  >
                                  > Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.
                                  >
                                  > Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.
                                  >
                                  > Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.
                                  >
                                  > Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.
                                  >
                                  > Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.
                                  >
                                  > --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@> wrote:
                                  > >
                                  > > Thank you for your answers. Completing my scenarium:
                                  > >
                                  > > 1 - It´s a information system, whose purpose is to have a strategic project
                                  > > portfolio and helps executives to make decisions based on simulations
                                  > > 2 - It has been developed for 10 years
                                  > > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                                  > > 4 - The changes in the database will not be restricted to naming convetions,
                                  > > but also to new semantics and new organization of tables (these could
                                  > > profound changes, for a better database organization)
                                  > > 5 - There will be a migration from a local database to a enterprise database
                                  > > (integration of some concepts)
                                  > > 6 - We would like to have an improvement of the OO model (actually, creating
                                  > > one from scratch), to have a more maintenable application
                                  > >
                                  > > One of the strategies that we are thinking is to have is the new application
                                  > > pointing to the old database
                                  > > and build a transformation layer, that receives the data in a good design
                                  > > (to the new database and model) and transforms to the old tables. After we
                                  > > change from VB6 to .NET, we could point the application to the new database
                                  > > without the transformation step.
                                  > >
                                  > > Caveats: We are afraid of migrate to the new database only at the end of the
                                  > > proccess and have an integration problem, ex: The OO model could not fit
                                  > > with the new database (impedance mismatch)
                                  > >
                                  > > Thank you
                                  > > Amanda
                                  > >
                                  > >
                                  >
                                • Amanda Varella
                                  Yes I am, Thank you for the ideas! Amanda Varella Enviado via iPad ... [Non-text portions of this message have been removed]
                                  Message 16 of 16 , Feb 24, 2012
                                  • 0 Attachment
                                    Yes I am,

                                    Thank you for the ideas!

                                    Amanda Varella

                                    Enviado via iPad

                                    Em 24/02/2012, às 07:15, "ROBERT" <bobcorrick@...> escreveu:

                                    > This is a great post.
                                    > I hope Amanda is still watching!
                                    > Bob Corrick.
                                    >
                                    > --- In agileDatabases@yahoogroups.com, "wbrianwhite" <wbrianwhite@...> wrote:
                                    > >
                                    > >
                                    > >
                                    > > Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.
                                    > >
                                    > > Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.
                                    > >
                                    > > Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.
                                    > >
                                    > > Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.
                                    > >
                                    > > Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.
                                    > >
                                    > > --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@> wrote:
                                    > > >
                                    > > > Thank you for your answers. Completing my scenarium:
                                    > > >
                                    > > > 1 - It´s a information system, whose purpose is to have a strategic project
                                    > > > portfolio and helps executives to make decisions based on simulations
                                    > > > 2 - It has been developed for 10 years
                                    > > > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
                                    > > > 4 - The changes in the database will not be restricted to naming convetions,
                                    > > > but also to new semantics and new organization of tables (these could
                                    > > > profound changes, for a better database organization)
                                    > > > 5 - There will be a migration from a local database to a enterprise database
                                    > > > (integration of some concepts)
                                    > > > 6 - We would like to have an improvement of the OO model (actually, creating
                                    > > > one from scratch), to have a more maintenable application
                                    > > >
                                    > > > One of the strategies that we are thinking is to have is the new application
                                    > > > pointing to the old database
                                    > > > and build a transformation layer, that receives the data in a good design
                                    > > > (to the new database and model) and transforms to the old tables. After we
                                    > > > change from VB6 to .NET, we could point the application to the new database
                                    > > > without the transformation step.
                                    > > >
                                    > > > Caveats: We are afraid of migrate to the new database only at the end of the
                                    > > > proccess and have an integration problem, ex: The OO model could not fit
                                    > > > with the new database (impedance mismatch)
                                    > > >
                                    > > > Thank you
                                    > > > Amanda
                                    > > >
                                    > > >
                                    > >
                                    >
                                    >


                                    [Non-text portions of this message have been removed]
                                  Your message has been successfully submitted and would be delivered to recipients shortly.