Loading ...
Sorry, an error occurred while loading the content.

Continuous Integraton and the database

Expand Messages
  • banshee858
    I was confronted with a mistake I did the other day at work - I created an automated database script which dropped and rebuilt the database with every build.
    Message 1 of 18 , Mar 30, 2005
    • 0 Attachment
      I was confronted with a mistake I did the other day at work - I
      created an automated database script which dropped and rebuilt the
      database with every build. Unfortunately, one of my co-workers did
      not include their work in the build script for the database and
      subsequently lost all their work they had done on the integration
      server. Irregardless of who did or did not do what, I was at fault
      for not informing everyone about what I was doing with the database
      script. Fortunately, we are not too far along in this project for
      this loss to be too painful. I also learned a lesson I need to
      communicate more with my teammates.

      We talked about this issue today and the main contention with dropping
      the DB and recreating from a script on every build was you manually
      had to update the script each time you made a change to the DB schema.
      If you did not update the script, then your change was not included
      in the build and you get a failure for reasons of the script and not
      the code. Or worse yet, you do not update from the repository before
      your commit, so in addition to your changes not being in the DB, all
      other changes from previous commits are lost. Has anyone figured a
      way to automate the creation of the DB script so it is transparent in
      the build process? Or what strategies have you applied for database
      stuff with CI?

      FYI, for some reason the developers on this team want to use the
      database on the CI server for their playground and not their sandbox.
      I would not mind some ideas to help convince them to switch, but I
      don't think it will happen. Also, chucking the database entirely or
      holding off until the last possible moment is a non-starter.

      Carlton
    • Rex Madden
      http://www.martinfowler.com/articles/evodb.html Also check out the agile-db archives. There s some discussion on handling this situation. Rex
      Message 2 of 18 , Mar 30, 2005
      • 0 Attachment
        http://www.martinfowler.com/articles/evodb.html

        Also check out the agile-db archives. There's some discussion on
        handling this situation.

        Rex

        On Wed, 30 Mar 2005 18:39:21 -0000, banshee858 <cnett858@...> wrote:
        >
        >
        > I was confronted with a mistake I did the other day at work - I
        > created an automated database script which dropped and rebuilt the
        > database with every build. Unfortunately, one of my co-workers did
        > not include their work in the build script for the database and
        > subsequently lost all their work they had done on the integration
        > server. Irregardless of who did or did not do what, I was at fault
        > for not informing everyone about what I was doing with the database
        > script. Fortunately, we are not too far along in this project for
        > this loss to be too painful. I also learned a lesson I need to
        > communicate more with my teammates.
        >
        > We talked about this issue today and the main contention with dropping
        > the DB and recreating from a script on every build was you manually
        > had to update the script each time you made a change to the DB schema.
        > If you did not update the script, then your change was not included
        > in the build and you get a failure for reasons of the script and not
        > the code. Or worse yet, you do not update from the repository before
        > your commit, so in addition to your changes not being in the DB, all
        > other changes from previous commits are lost. Has anyone figured a
        > way to automate the creation of the DB script so it is transparent in
        > the build process? Or what strategies have you applied for database
        > stuff with CI?
        >
        > FYI, for some reason the developers on this team want to use the
        > database on the CI server for their playground and not their sandbox.
        > I would not mind some ideas to help convince them to switch, but I
        > don't think it will happen. Also, chucking the database entirely or
        > holding off until the last possible moment is a non-starter.
        >
        > Carlton
        >
        > To Post a message, send it to: extremeprogramming@...
        >
        > To Unsubscribe, send a blank message to: extremeprogramming-unsubscribe@...
        >
        > ad-free courtesy of objectmentor.com
        > Yahoo! Groups Links
        >
        >
        >
        >
        >
      • Jeff Grigg
        ... You need to be able to build your system from its sources. That is, if you threw away everything except the source code, you should still be able to build
        Message 3 of 18 , Mar 30, 2005
        • 0 Attachment
          --- "banshee858" <cnett858@h...> wrote:
          > [...] the main contention with dropping the DB and recreating
          > from a script on every build was you manually had to update
          > the script each time you made a change to the DB schema. [...]

          You need to be able to build your system from its sources. That is, if
          you threw away everything except the source code, you should still be
          able to build and run the system. If you can't, then you have a
          problem.

          To address their issue of losing data when they fail to update the
          (DDL) source code, try backing up the data before doing an
          upgrade/build.

          (In some cases, I've been forced to consider the contents of some
          database to be part of the source code. In that case, you'll want to
          be *very careful* about access to that database, and control of its
          versioning.)
        • banshee858
          ... I agree with this, but not all on my team agree with me. Hence the friction over the database script caused by my clobbering someone else s work. ... I
          Message 4 of 18 , Mar 30, 2005
          • 0 Attachment
            >
            > You need to be able to build your system from its sources. That
            > is, if you threw away everything except the source code, you should
            > still be able to build and run the system. If you can't, then you
            > have a problem.
            >
            I agree with this, but not all on my team agree with me. Hence the
            friction over the database script caused by my clobbering someone
            else's work.

            >
            > To address their issue of losing data when they fail to update the
            > (DDL) source code, try backing up the data before doing an
            > upgrade/build.
            >
            > (In some cases, I've been forced to consider the contents of some
            > database to be part of the source code. In that case, you'll want
            > tobe *very careful* about access to that database, and control of
            > its versioning.)
            >
            I don't think losing data is the issue (however that is important),
            the objection comes from how do we maintain the SQL script used to
            create the database automatically? The objection comes from copying
            and pasting the items into the central script. That process is too
            manual and prone for error.

            The friction comes from two differing perspectives - mine is your work
            in your sandbox and integrate your changes to the CI machine. Another
            perspective is work in your sandbox for code stuff, but use the CI
            machine for database stuff.

            Carlton
          • Tim Moore
            ... Does that mean that people are committing code without testing it against a database? And how does the schema from the CI machine get installed in
            Message 5 of 18 , Mar 30, 2005
            • 0 Attachment
              On Mar 30, 2005, at 2:44 PM, banshee858 wrote:

              > The friction comes from two differing perspectives - mine is your work
              > in your sandbox and integrate your changes to the CI machine. Another
              > perspective is work in your sandbox for code stuff, but use the CI
              > machine for database stuff.
              >

              Does that mean that people are committing code without testing it
              against a database? And how does the schema from the CI machine get
              installed in production?
              --
              Tim Moore
            • Ken Scott
              ... Does this central script that you mention exist separate from the DDL sql files that are in your source repository? If so, could you change your process to
              Message 6 of 18 , Mar 30, 2005
              • 0 Attachment
                On Mar 30, 2005, at 12:44 PM, banshee858 wrote:

                >
                >
                >>
                >> You need to be able to build your system from its sources. That
                >> is, if you threw away everything except the source code, you should
                >> still be able to build and run the system. If you can't, then you
                >> have a problem.
                >>
                > I agree with this, but not all on my team agree with me. Hence the
                > friction over the database script caused by my clobbering someone
                > else's work.
                >
                >>
                >> To address their issue of losing data when they fail to update the
                >> (DDL) source code, try backing up the data before doing an
                >> upgrade/build.
                >>
                >> (In some cases, I've been forced to consider the contents of some
                >> database to be part of the source code. In that case, you'll want
                >> tobe *very careful* about access to that database, and control of
                >> its versioning.)
                >>
                > I don't think losing data is the issue (however that is important),
                > the objection comes from how do we maintain the SQL script used to
                > create the database automatically? The objection comes from copying
                > and pasting the items into the central script. That process is too
                > manual and prone for error.
                >

                Does this central script that you mention exist separate from the DDL
                sql files that are in your source repository?

                If so, could you change your process to pull the DDL sql files from
                your repository, and apply those 1-by-1 (or generate your all-in-one
                script) and then run that?

                To me, this would eliminate the duplication that the central script is
                causing. Depending on your database engine, you need to be careful in
                the order that you create indexes, foreign keys, constraints, etc. to
                make sure that all the tables that are involved exist first. Again, I
                see this as the job of your script that builds your database.

                > The friction comes from two differing perspectives - mine is your work
                > in your sandbox and integrate your changes to the CI machine. Another
                > perspective is work in your sandbox for code stuff, but use the CI
                > machine for database stuff.
                >

                Is this purely a difference of opinion about where to work, or is there
                an underlying issue that is being covered up? How much work is it for
                the developers to keep their local db up-to-date with both schema and
                test data? Do you have one script that a developer can run to get an
                up-to-date sandbox database?

                Ken

                --
                <>< Ken Scott ken@... http://hobbes.optikos.net/~ken

                This is the day that the Lord has made;
                Let us rejoice and be glad in it -- Psalm 118:24
              • banshee858
                ... This is a new project, so there is no production database. This is only an issue on a CI server. As for people committing code without testing it against
                Message 7 of 18 , Mar 30, 2005
                • 0 Attachment
                  >
                  > Does that mean that people are committing code without testing it
                  > against a database? And how does the schema from the CI machine get
                  > installed in production?
                  >
                  This is a new project, so there is no production database. This is
                  only an issue on a CI server.

                  As for people committing code without testing it against a DB, I would
                  have to say yes and no. No, because we are not coding; that is we are
                  in the "design phase". Yes, because part of the "design phase" is
                  tossing in c# files to the repository with skeleton methods - methods
                  which have names, signatures, but not working insides.

                  Carlton
                • banshee858
                  ... Yes, this script exists independent of seperate DDL sql files. In fact, there are no seperate DDL sql files since the database is being figured out.
                  Message 8 of 18 , Mar 30, 2005
                  • 0 Attachment
                    >
                    > Does this central script that you mention exist separate from the
                    > DDL sql files that are in your source repository?
                    >
                    > If so, could you change your process to pull the DDL sql files from
                    > your repository, and apply those 1-by-1 (or generate your
                    > all-in-one script) and then run that?
                    >
                    Yes, this script exists independent of seperate DDL sql files. In
                    fact, there are no seperate DDL sql files since the database is being
                    figured out. Personally, I think seperate DDL files cause confusion
                    and I like the idea of one single script which holds everything. It
                    tends to be easier to invoke and produces less "I forgot to update
                    that file" or "I forgot to add that file to the build" events.

                    >
                    > Is this purely a difference of opinion about where to work, or is
                    > there an underlying issue that is being covered up? How much work
                    > is it for the developers to keep their local db up-to-date with
                    > both schema and test data? Do you have one script that a developer
                    > can run to get an up-to-date sandbox database?
                    >
                    Let's assume a difference of opinion. I think this is an instance of
                    "I can imagine lots of theoretical problems with this, so we should
                    not try it". The goal of the single script was to make it easy for
                    developers to have a way to update their sandbox with the version on
                    the CI server. To me, having multiple sql DDL files makes it hard to
                    update the DB in my local environment.

                    Carlton

                    Carlton
                  • Donald Roby
                    ... You will eventually have to create a production database. If you continue to maintain a script for creating the CI database, creating the production
                    Message 9 of 18 , Mar 31, 2005
                    • 0 Attachment
                      --- In extremeprogramming@yahoogroups.com, "banshee858"
                      <cnett858@h...> wrote:
                      >
                      > >
                      > > Does that mean that people are committing code without testing it
                      > > against a database? And how does the schema from the CI machine get
                      > > installed in production?
                      > >
                      > This is a new project, so there is no production database. This is
                      > only an issue on a CI server.
                      >

                      You will eventually have to create a production database. If you
                      continue to maintain a script for creating the CI database, creating
                      the production database will be easy. If not, it may be painful.

                      > As for people committing code without testing it against a DB, I would
                      > have to say yes and no. No, because we are not coding; that is we are
                      > in the "design phase". Yes, because part of the "design phase" is
                      > tossing in c# files to the repository with skeleton methods - methods
                      > which have names, signatures, but not working insides.
                      >

                      DDL is code. If people are modifying schemas without creating and
                      maintaining scripts to reproduce the schemas, they are coding without
                      code control.

                      My take from this thread is that you've been trying to do it right by
                      scripting the DB creation. This is great, but you need everyone's
                      collaboration. This script should be regarded as part of your
                      codebase, owned by the whole team, and kept in code control.

                      > Carlton
                    • Jeff Grigg
                      ... Sounds to me like your teammates are effectively trying to treat the Continuous Integration database schema, in the database, as part of the source
                      Message 10 of 18 , Mar 31, 2005
                      • 0 Attachment
                        >> You need to be able to build your system from its sources. [...]

                        --- "banshee858" <cnett858@h...> wrote:
                        > I agree with this, but not all on my team agree with me. [...]


                        >> (In some cases, I've been forced to consider the contents
                        >> of some database to be part of the source code. [...])

                        --- "banshee858" <cnett858@h...> wrote:
                        > I don't think losing data is the issue (however that is
                        > important), the objection comes from how do we maintain
                        > the SQL script used to create the database automatically?
                        > [...]

                        Sounds to me like your teammates are effectively trying to treat the
                        Continuous Integration database schema, in the database, as part of
                        the "source configuration" (or source code) for the system. That is,
                        it's impossible to build and deploy the system without it.

                        I don't like to do it that way, but it can be made to work.

                        Consider generating the DDL script automatically from the database
                        schema. Check in the script, at each build, whenever it changes.
                        And test, from time to time, that you can build a database with this
                        script, and still successfully run all the tests.

                        Otherwise, you'll have to copy the database, whenever you want to
                        deploy. And worse, you won't have version control over an important
                        component's source code.
                      • Colin Putney
                        ... And actually, thinking about it that way has an interesting effect. On that part of the source code, it gives you *truly* continuous integration. You re
                        Message 11 of 18 , Mar 31, 2005
                        • 0 Attachment
                          Jeff Grigg wrote:

                          > --- "banshee858" <cnett858@h...> wrote:
                          >
                          >>I don't think losing data is the issue (however that is
                          >>important), the objection comes from how do we maintain
                          >>the SQL script used to create the database automatically?
                          >>[...]
                          >
                          > Sounds to me like your teammates are effectively trying to treat the
                          > Continuous Integration database schema, in the database, as part of
                          > the "source configuration" (or source code) for the system. That is,
                          > it's impossible to build and deploy the system without it.

                          And actually, thinking about it that way has an interesting effect. On
                          that part of the source code, it gives you *truly* continuous
                          integration. You're updating the entire team's view of the source, in
                          real time, as you make changes.

                          Colin
                        • Jeff Grigg
                          ... ...until you learn that the non-DB source code is changed in developer sandboxes -- so there s no way to coordinate check-ins of related database and
                          Message 12 of 18 , Mar 31, 2005
                          • 0 Attachment
                            > --- Jeff Grigg wrote:
                            >> [...] your teammates are effectively trying to treat the
                            >> Continuous Integration database schema, in the database,
                            >> as part of the "source configuration" (or source code)
                            >> for the system. That is, it's impossible to build and
                            >> deploy the system without it.

                            --- Colin Putney <cputney@w...> wrote:
                            > And actually, thinking about it that way has an interesting
                            > effect. On that part of the source code, it gives you *truly*
                            > continuous integration. You're updating the entire team's
                            > view of the source, in real time, as you make changes.

                            ...until you learn that the non-DB source code is changed in
                            developer "sandboxes" -- so there's no way to coordinate "check-ins" of
                            related database and corresponding non-DB source changes. ... "Ouch!"
                            when it hits you -- IE: broken builds. ;->
                          • Colin Putney
                            ... Good point. The database schema should be checked into source control along with the rest of the source code. That is, generating the DDL script should be
                            Message 13 of 18 , Apr 1, 2005
                            • 0 Attachment
                              Jeff Grigg wrote:

                              > --- Colin Putney <cputney@w...> wrote:
                              >
                              >>And actually, thinking about it that way has an interesting
                              >>effect. On that part of the source code, it gives you *truly*
                              >>continuous integration. You're updating the entire team's
                              >>view of the source, in real time, as you make changes.
                              >
                              >
                              > ...until you learn that the non-DB source code is changed in
                              > developer "sandboxes" -- so there's no way to coordinate "check-ins" of
                              > related database and corresponding non-DB source changes. ... "Ouch!"
                              > when it hits you -- IE: broken builds. ;->

                              Good point. The database schema should be checked into source control
                              along with the rest of the source code. That is, generating the DDL
                              script should be part of the commit procedure, not the build.

                              More generally, the database schema should be under source control along
                              with the rest of the code. If you can do that directly, great. If not,
                              you'll have to boil it down to a DDL script. That's not too bad, because
                              you'll need to create a DDL script as part of the build, so that the
                              database can be created at install time.

                              In the long term, the team might run into problems because of the
                              different integration rate between different parts of the source code.

                              Colin
                            • Olson, Curtis B
                              Hi Carlton. ... We ve experienced this too with DDL, and I think the best approach is Once And Only Once: don t copy and paste from multiple small files into
                              Message 14 of 18 , Apr 1, 2005
                              • 0 Attachment
                                Hi Carlton.

                                > > the objection comes from how do we maintain the SQL script used to
                                > > create the database automatically? The objection comes
                                > from copying
                                > > and pasting the items into the central script. That process is too
                                > > manual and prone for error.

                                > Personally, I think
                                > seperate DDL files cause confusion and I like the idea of one
                                > single script which holds everything. It tends to be easier
                                > to invoke and produces less "I forgot to update
                                > that file" or "I forgot to add that file to the build" events.

                                > The goal of the single
                                > script was to make it easy for developers to have a way to
                                > update their sandbox with the version on the CI server. To
                                > me, having multiple sql DDL files makes it hard to update the
                                > DB in my local environment.

                                We've experienced this too with DDL, and I think the best approach is
                                Once And Only Once: don't copy and paste from multiple small files into
                                one big file, but just have the master script do nothing more than call
                                the individual small files.

                                But if you are going to generate one huge DDL script from the database
                                rather than have the database generated from the DDL, you have no
                                choice. I think this is a smell, but some of my colleagues find it to
                                be a more pragmatic way to produce DDL. I see it as a smell because if
                                changes to the database are applied by someone working interactively,
                                then the script may not get regenerated, and will therefore be out of
                                date. Also, it's one huge friggin' file. Furthermore, the temptation
                                exists to neglect placing said huge friggin' file under source control -
                                I've heard arguments stating that there's no reason to put it under
                                source control because the most current version can be regenerated
                                straight from the database at any time (at any time other than when the
                                system crashes and is unrecoverable, I reply, but the response is that
                                with proper backups this cannot happen).

                                I like the OAOO thing - having folks create a single small file when
                                needed is of sufficiently small pain that most folks will do it, and
                                it's not far from their normal flow of working.

                                Cheers,
                                Curtis
                              • banshee858
                                I have no strong objections about the small files. I agree that the big file is a hassle. However, do you still need another seperate file for creating roles,
                                Message 15 of 18 , Apr 1, 2005
                                • 0 Attachment
                                  I have no strong objections about the small files. I agree that the
                                  big file is a hassle.

                                  However, do you still need another seperate file for creating roles,
                                  execute permissions and creating constraints? Or are they added to
                                  the small file that makes most sense? Also, do the small files do a
                                  drop of the object before the object is created?

                                  Carlton
                                • Olson, Curtis B
                                  ... We generally tend to create a file for each object. This matches our flow: we need a new object and are about to type the DDL into (in our case) Oracle
                                  Message 16 of 18 , Apr 1, 2005
                                  • 0 Attachment
                                    > However, do you still need another seperate file for creating
                                    > roles, execute permissions and creating constraints? Or are
                                    > they added to the small file that makes most sense?

                                    We generally tend to create a file for each object. This matches our
                                    flow: we need a new object and are about to type the DDL into (in our
                                    case) Oracle SQL*Plus, but usually it's less error-prone to write it in
                                    Notepad first and them copy and paste the DDL into SQL*Plus. Hey, might
                                    as well save that Notepad file while we're at it - something like
                                    MYOBJECT.SQL. Related information, such as permissions and constraints,
                                    would probably go into that file right then, because now is when we need
                                    them.

                                    We tried keeping a TRIGGERS.SQL file and a VIEWS.SQL file, but honestly
                                    this approaches the one huge friggin' file approach (instead about a
                                    dozen sort-big files). However makes it easy to track where specific
                                    scripts are. When using lots of small, roughly per-object files, one
                                    needs to use good file naming.


                                    > Also, do
                                    > the small files do a drop of the object before the object is created?

                                    No, we just have the creation statements. I'm guessing your next
                                    question is "how do you change the scripts without dropping the
                                    objects". We change the creation script file, but in SQL*Plus we type
                                    in ALTER statements. Not ideal, but it's usually still within normal
                                    flow to work this way.

                                    Some objects (triggers, views, synonyms) can use a CREATE OR REPLACE
                                    statement which is nifty, but with tables it's either a CREATE statement
                                    or an ALTER statement. Wish all objects could use CREATE OR REPLACE.

                                    Cheers,
                                    Curtis
                                  • Brad Appleton
                                    Interesting timing ... just after the Continuous Integration and the Database Thread settled down, I saw someone on a completely unrelated forum mention they
                                    Message 17 of 18 , Apr 3, 2005
                                    • 0 Attachment
                                      Interesting timing ... just after the "Continuous Integration and the
                                      Database Thread" settled down, I saw someone on a completely unrelated
                                      forum mention they were going to start developing an OpenSource tool
                                      to address the lack of software version-control tool functionality for
                                      databases and the repeated frustration of having to use DDL/SQL
                                      scripts to (recreate) the database/schema.

                                      The post was on CMCrossroads.com at
                                      <http://www.cmcrossroads.com/ubbthreads/showflat.php?Number=42162>

                                      They started a releated Webpage on the CMWiki at
                                      <http://www.cmcrossroads.com/cgi-bin/cmwiki/bin/view.cgi/CM/DaversySpecs>
                                    • eligolovinsky
                                      ... bin/cmwiki/bin/view.cgi/CM/DaversySpecs Thanks Brad for mentioning my efforts here. As I ve seen in a few discussions and often felt myself, the way we
                                      Message 18 of 18 , Apr 8, 2005
                                      • 0 Attachment
                                        --- In extremeprogramming@yahoogroups.com, "Brad Appleton"
                                        > Interesting timing ... just after the "Continuous Integration
                                        > and the Database Thread" settled down, I saw someone on a
                                        > completely unrelated forum mention they were going to
                                        > start developing an OpenSource tool to address the lack of
                                        > software version-control tool functionality for databases and the
                                        > repeated frustration of having to use DDL/SQL scripts to
                                        > (recreate) the database/schema.
                                        >
                                        > The post was on CMCrossroads.com at
                                        > <http://www.cmcrossroads.com/ubbthreads/showflat.php?Number=42162>
                                        >
                                        > They started a releated Webpage on the CMWiki at
                                        > <http://www.cmcrossroads.com/cgi-
                                        bin/cmwiki/bin/view.cgi/CM/DaversySpecs>

                                        Thanks Brad for mentioning my efforts here.

                                        As I've seen in a few discussions and often felt myself, the way we
                                        usually manage database versions is just not good enough. We've got
                                        most of the issues with managing and versioning source pinned down,
                                        but somehow the aspect of managing the versions of databases remains
                                        mostly untreated. But most business applications need a database,
                                        sometimes several databases at different locations and of different
                                        versions and vendors. Martin Fowler's Evolutionary Database Design
                                        has some very good points on database refactoring, but reading it I
                                        felt that without the proper tools, his method might be problematic.
                                        Good DBA's are a scarce and expensive resource and having a DBA
                                        close by for instant code review and integration is just not always
                                        possible.

                                        The basic idea that can allow us to move further on the path to
                                        database SCM is devise a neutral language (probablye an XML of some
                                        sort) that describes database structures. Files written in such a
                                        language could be held in source control and creation scripts could
                                        be created from those files. Furthermore, by comparing two such
                                        files, we could generate a script that moves a database from one
                                        version to another and by creating several such diffs we could move
                                        several versions forwards or backwards. But that's not all there is
                                        to it. Databases are built to hold data and in that they are very
                                        different from source code. A piece of source code can be replaced,
                                        recompiled and deployed, but such technique cannot be applied to
                                        databases. Suppose we are refactoring two tables of a 1-to-1
                                        relationship and want to move a column from TABLE1 to TABLE2. Using
                                        current techniques, we would create a script that would first add
                                        the new column to TABLE2, the run un UPDATE command that would copy
                                        the data into the new column and the drop the column in TABLE1. What
                                        we need is a way to automate that process, a way to build upgrade
                                        scenarios that can use comparison between db-structure files and
                                        custom written update commands to move a database from one version
                                        to another.

                                        It's a open source project because it has to support many databases
                                        and many SCM tools and it will require some community effort to make
                                        that happen.

                                        I would really like to hear your comments on my ideas, both in terms
                                        of requirements of such a tool and in terms of implementation
                                        recommendations. This tool (called Daversy - Database Vesioning
                                        System) could make all of our lives much easier, but if I'll do it
                                        on my own, I'm bound to miss a few critical spots.

                                        My up-to-date thoughts on this are on a wiki here:
                                        <http://www.cmcrossroads.com/cgi-
                                        bin/cmwiki/bin/view.cgi/CM/DaversySpecs>

                                        Hope to hear from you.

                                        Eli Golovinsky.
                                      Your message has been successfully submitted and would be delivered to recipients shortly.