Loading ...
Sorry, an error occurred while loading the content.

Re: Migrating data and code from legacy system to a new one

Expand Messages
  • wbrianwhite
    Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the
    Message 1 of 16 , Feb 23, 2012
    • 0 Attachment
      Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.

      Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.

      Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.

      Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.

      Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.

      --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@...> wrote:
      >
      > Thank you for your answers. Completing my scenarium:
      >
      > 1 - It´s a information system, whose purpose is to have a strategic project
      > portfolio and helps executives to make decisions based on simulations
      > 2 - It has been developed for 10 years
      > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
      > 4 - The changes in the database will not be restricted to naming convetions,
      > but also to new semantics and new organization of tables (these could
      > profound changes, for a better database organization)
      > 5 - There will be a migration from a local database to a enterprise database
      > (integration of some concepts)
      > 6 - We would like to have an improvement of the OO model (actually, creating
      > one from scratch), to have a more maintenable application
      >
      > One of the strategies that we are thinking is to have is the new application
      > pointing to the old database
      > and build a transformation layer, that receives the data in a good design
      > (to the new database and model) and transforms to the old tables. After we
      > change from VB6 to .NET, we could point the application to the new database
      > without the transformation step.
      >
      > Caveats: We are afraid of migrate to the new database only at the end of the
      > proccess and have an integration problem, ex: The OO model could not fit
      > with the new database (impedance mismatch)
      >
      > Thank you
      > Amanda
      >
      >
    • ROBERT
      This is a great post. I hope Amanda is still watching! Bob Corrick.
      Message 2 of 16 , Feb 24, 2012
      • 0 Attachment
        This is a great post.
        I hope Amanda is still watching!
        Bob Corrick.

        --- In agileDatabases@yahoogroups.com, "wbrianwhite" <wbrianwhite@...> wrote:
        >
        >
        >
        > Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.
        >
        > Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.
        >
        > Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.
        >
        > Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.
        >
        > Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.
        >
        > --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@> wrote:
        > >
        > > Thank you for your answers. Completing my scenarium:
        > >
        > > 1 - It´s a information system, whose purpose is to have a strategic project
        > > portfolio and helps executives to make decisions based on simulations
        > > 2 - It has been developed for 10 years
        > > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
        > > 4 - The changes in the database will not be restricted to naming convetions,
        > > but also to new semantics and new organization of tables (these could
        > > profound changes, for a better database organization)
        > > 5 - There will be a migration from a local database to a enterprise database
        > > (integration of some concepts)
        > > 6 - We would like to have an improvement of the OO model (actually, creating
        > > one from scratch), to have a more maintenable application
        > >
        > > One of the strategies that we are thinking is to have is the new application
        > > pointing to the old database
        > > and build a transformation layer, that receives the data in a good design
        > > (to the new database and model) and transforms to the old tables. After we
        > > change from VB6 to .NET, we could point the application to the new database
        > > without the transformation step.
        > >
        > > Caveats: We are afraid of migrate to the new database only at the end of the
        > > proccess and have an integration problem, ex: The OO model could not fit
        > > with the new database (impedance mismatch)
        > >
        > > Thank you
        > > Amanda
        > >
        > >
        >
      • Amanda Varella
        Yes I am, Thank you for the ideas! Amanda Varella Enviado via iPad ... [Non-text portions of this message have been removed]
        Message 3 of 16 , Feb 24, 2012
        • 0 Attachment
          Yes I am,

          Thank you for the ideas!

          Amanda Varella

          Enviado via iPad

          Em 24/02/2012, às 07:15, "ROBERT" <bobcorrick@...> escreveu:

          > This is a great post.
          > I hope Amanda is still watching!
          > Bob Corrick.
          >
          > --- In agileDatabases@yahoogroups.com, "wbrianwhite" <wbrianwhite@...> wrote:
          > >
          > >
          > >
          > > Here is how we solved a similar project at my company. We changed from a single database store to a sharded system, while also changing the data model and the way we saved the data to the database.
          > >
          > > Based on a thorough analysis of the domain, a new database system was created. You do have to get this pretty close to right up front. Not perfect, but spend the time thinking it out. In our main application we added a company by company setting to say whether they use the old system or new system. It was actually quad-state: "read/write old", "write both/read old", "write both/read new", "read/write new". Got the system basically working so you could do the same actions in the new system as you could in the old. You have to add a bunch of plumbing for this to work.
          > >
          > > Next is the migration of data part. In our case, the new system was web service based with no direct database access, so when our app was talking to the new system, it was doing it all through web service posts/gets. There is a layer of XSL transforms here too, to smooth over the data model mismatches. So we created a batch job that could look at our old data, and format it so that the web services for the new system could accept that document post. Had the normal batch system type stuff: keep track of where you were, do one company at a time, do all companies in mode "write both", etc. Don't expect to get this right up front. Expect to have to wipe out all the exported data a few time, and be able to re-export it. It's an iterative process. Particularly we hit problems with our oldest data, because our product just wasn't mature way back when.
          > >
          > > Next is the transfer from one system to the other. We would do things like flip all companies to "write both/read old" to get performance metrics on the new database system. We would also set up a system where we could gather up the thousands and thousands of posts happening in prod while this was going on, and get them into dev and test them out there and find anything that was unexpected. So to start transfering, you pick a client (pick yourselves, or a test company) and set to write both/read old for a while. Then kick off the batch export and see how it goes. Then flip them to write both/read new and see how it goes. Start moving different types of companies and monitoring, knowing that you can always fail them back to the old system if they have a critical problem. We finally got everyone read/write new system only. It was a very complicated process. But based on the types of issues we encountered, if we had tried to just shut down for a weekend, and move everyone to the new system, we would have only realized we had unsolved problems when we were irretrievably committed to the new system with no rollback system, and we would have lost major face and major customers.
          > >
          > > Having regression suites in place, which we didn't when we started, are a big help for something like this. This project was analogous to redesigning and rebuilding an airplane while in flight. Dangerous, complicated, and if you do everything exactly right, nobody ever knows you did anything at all. It took us about 18 months from start to finish.
          > >
          > > --- In agileDatabases@yahoogroups.com, Amanda Varella <amandavarella@> wrote:
          > > >
          > > > Thank you for your answers. Completing my scenarium:
          > > >
          > > > 1 - It´s a information system, whose purpose is to have a strategic project
          > > > portfolio and helps executives to make decisions based on simulations
          > > > 2 - It has been developed for 10 years
          > > > 3 - Its written in VB6 and the new version will use .NET WPF (desktop)
          > > > 4 - The changes in the database will not be restricted to naming convetions,
          > > > but also to new semantics and new organization of tables (these could
          > > > profound changes, for a better database organization)
          > > > 5 - There will be a migration from a local database to a enterprise database
          > > > (integration of some concepts)
          > > > 6 - We would like to have an improvement of the OO model (actually, creating
          > > > one from scratch), to have a more maintenable application
          > > >
          > > > One of the strategies that we are thinking is to have is the new application
          > > > pointing to the old database
          > > > and build a transformation layer, that receives the data in a good design
          > > > (to the new database and model) and transforms to the old tables. After we
          > > > change from VB6 to .NET, we could point the application to the new database
          > > > without the transformation step.
          > > >
          > > > Caveats: We are afraid of migrate to the new database only at the end of the
          > > > proccess and have an integration problem, ex: The OO model could not fit
          > > > with the new database (impedance mismatch)
          > > >
          > > > Thank you
          > > > Amanda
          > > >
          > > >
          > >
          >
          >


          [Non-text portions of this message have been removed]
        Your message has been successfully submitted and would be delivered to recipients shortly.