Loading ...
Sorry, an error occurred while loading the content.

Continuous Staging (was Re: Rationale behind one step build process.)

Expand Messages
  • Brad Appleton
    ... You are both correct :-) ... Highly desirable. Not always feasible. It depends upon relationships between subteams, and their respective rhythms , as well
    Message 1 of 15 , Mar 1, 2004
    • 0 Attachment
      On Mon, Mar 01, 2004 at 01:56:13PM +1300, John Carter wrote:
      > I think having a one step build process is very important,
      > and he thinks having versioned libraries are important.

      You are both correct :-)

      > I believe you should be able to build the entire product
      > directly from the source code in one step, ensuring no
      > breakage. (Bitrot)

      Highly desirable. Not always feasible. It depends upon
      relationships between subteams, and their respective "rhythms",
      as well as geographical dispersement (and timezone), repository
      size and performance, system-build performance and available
      network resources, etc.

      > He believes that each subteam should deliver "tested, closed,
      > sealed and signed, versioned binary libraries" to the rest of the team.

      If the team is large or dispersed enough to actually warrant
      "subteams", he may be right - to a point. I believe each team
      should be responsible for ensuring it delivers working code
      *and* executables to other internal stakeholders. In XP, I do
      this with Continuous Integration by ensuring every commit to
      the codebase builds and passes all the tests PLUS explicitly
      identifying any version of the codebase that needs to be
      accessed by internal stakeholders (e.g., using a label/tag so
      that if anyone needs to look at it even after its no longer
      "latest and greatest" they can easily do so - as is often the
      case when a "build" must be handed-off to a QA/V&V group who
      require more time to do "their job" with it than it takes to
      complete and commit the next "task")

      > I believe that when you check into CVS mainline you are versioning,
      > signing and sealing that you have compiled, tested and it works.

      Yes (assuming continuous integration) - for the set of things that
      are in that repository and which are built and tested in that repository.
      If the repository contains several independently buildable components,
      and people only build their components rather than the "whole shmeer",
      you are doing this only for that component.

      Ideally - you have the storage capacity, and network bandwidth,
      and processing power, and load distribution, to build the whole
      system (or at least incrementally build the whole system) everytime
      before you commit.

      Sometimes, for reasons of build-cycle-time, or network resource load,
      or schedule coordination (e.g, multiple timezones, or interdependent
      delivery schedules of components), or other reasons, this is not
      always feasible.

      A common best-practice used here that tries to balance the
      best of both is what is called a "Staging Area" or "Staging
      Environment". A Staging Environment is like a "sandbox" or
      "workspace" reserved for sharing build+test dependent artifacts
      (headers, libraries, executables, etc.). It works something
      like this (note, this is NOT specific to XP/Agile development):
      * Each subteam "does its thing" and builds and compiles its
      code as they should and commits it to their repository

      * At agreed-upon points in time, the subteam "delivers" any
      artifacts that other teams require in order to build (headers,
      APIs, libraries, configs, etc.) into the staging area
      (and ideally some additional level of build+test is done)

      * When building in ones private developer workspace or even
      a subteams integration workspace/machine, the compilers and
      linkers (etc.) point to the "official" staging area for the
      non-subteam owned artifacts needed for the subteam to build

      The issue of versioning comes into play if it is necessary to
      know which version of a component is the "current" one in the
      staging area. If necessary, then:
      * When a subteam build has its executables delivered to the
      staging area, it also creates a corresponding tag/label,
      and perhaps writes it to a file (e.g., README or whatever)
      for that component in the staging area (this is a "simple
      persons" implementation of what might be more formally called
      a "version description document" or VDD at the component-level

      If versioning is required, then it is typically handled one of
      two ways:
      * SAME REPOSITORY: a separate directory tree in the repository
      is used to house any "installed" staging artifacts (staged
      artifacts). Developers will typically use the staging area
      plus the top-level directory for their own components in their
      sandbox (and don't extract/checkout anything else unless and
      until they need to view the source for something outside their
      owned component)

      * STAGING REPOSITORY: a separate repository is used to house
      all staged artifacts. It can therefore accommodate separate
      sets of versions and tags/labels (which has good points and
      bad points)

      Sometimes granularity of access-control, administration,
      mirroring/synchronizing will determine which of the above
      two approaches is best. If each component is large enough
      to already warrant its own separate repository from the
      others, then a separate staging repository is typically used.

      Sometimes, for local performance, a staging area might be
      mirrored or replicated to local sites/storage to cut-down on
      their network bandwidth for their build-cycle time.

      So that is the general (non-agile-specific) best practice. How
      might we apply an "agile" adaptation of it? Well, the
      "simple" case is when no separate staging area is needed because
      the whole "one team" can peacefully co-exit in "one repository"
      and each work at their own "sustainable pace" without unduly
      impacting the others. So there is little need to think so
      much about subparts and subteams and instead more easily focus
      on "the whole"

      Other times, factors of scale rear their ugly head! They may
      be issues of system/build scale, organization and organizational
      process, issues of ownership over computing resources, etc.
      (or maybe not all the subteams are using "agile" and some of them
      can't tolerate such high-frequency of changes/deliveries from the
      agile-teams into their own part of the codebase :-)

      The "Staging Environment" is a specific technique for separating
      (sub)team build-dependency interface from implementation for
      the benefit of the rest of the team(s). The "Staging Environment"
      is the component-version "mediator" (coordinator really) that
      houses the common interface and necessary artifacts to satisfy
      build+test dependencies across subteams.

      One of the key problems to solve is when and how-often a subteam
      should do a "signed+sealed" deliver to the staging area. If every
      commit to the codebase is too frequent for a staging delivery, then
      something needs to be negotiated between the other subteams. This
      is where some agile methods try to "scale" by using a team of teams
      (e.g. a "Scrum of Scrums") to manage the staging frequency and
      coordination.

      If it is necessary to "scale-up" my build process & resources to
      use a Staging Environment, how might I "scale-up" a practice like
      "Continuous Integration" to approximate "Continuous Staging" into
      the staging area? This would avoid, or at least minimize the
      need to tag/label every delivery into the staging area, and hence
      minimize the need to manage build-version-dependencies between
      components and the subteams that work on them. Even if it were
      no longer practical to use a single repository or full system build
      for every commit across the whole team, it might be feasible to:
      - Have every commit (or even just once or twice daily) "trigger"
      a delivery to the staging area.

      - Another trigger (or perhaps the same one) detects an update
      to the staging area and does the next-level of build+link+test
      (automated of course) using the current set of stuff in the
      staging area.
      - If it breaks, you take the appropriate course of action and
      notifications (just as one does for CI at smaller scale)

      Even in those cases where you might still need to "version" the
      codebase (and component) for what you delivered to the staging
      area, the staging area itself can be used to manage the current
      latest and greatest set of "system buildworthy" components and
      their versions (both source and binaries).

      Hope that helps! (sorry about the length. wonder if I should
      try to publish this somewhere, like that new "fundamentals"
      section of Agile Times? Any feedback would be appreciated)

      --
      Brad Appleton <brad@...> www.bradapp.net
      Software CM Patterns (www.scmpatterns.com)
      Effective Teamwork, Practical Integration
      "And miles to go before I sleep." -- Robert Frost
    • Paul Campbell
      ... and ... thinks ... from the ... sealed and ... at ... literature / ... Never keep product/intermediate/generated files under version control. If you ever
      Message 2 of 15 , Mar 1, 2004
      • 0 Attachment
        --- In extremeprogramming@yahoogroups.com, John Carter
        <john.carter@t...> wrote:
        > I have just realized that a major source of conflict between myself
        and
        > another developer on the team is the following.
        >
        > I think having a one step build process is very important, and he
        thinks
        > having versioned libraries are important.
        >
        > I believe you should be able to build the entire product directly
        from the
        > source code in one step, ensuring no breakage. (Bitrot)
        >
        > He believes that each subteam should deliver "tested, closed,
        sealed and
        > signed, versioned binary libraries" to the rest of the team.
        >
        > I believe that when you check into CVS mainline you are versioning,
        > signing and sealing that you have compiled, tested and it works.
        >
        > I believe the only thing binary libraries gives you these days is a
        > screwed up dependency tree at "make" time and maybe a second or two
        at
        > link time.
        >
        > Does anyone have any opinions, pointers to "Best Practice"
        literature /
        > web sites to support either view.

        Never keep product/intermediate/generated files under version
        control. If you ever think to yourself "maybe we wont be able to
        reproduce this in future" then you have a fundamental build
        reproduceablity problem that you must solve.

        Dont give in to irrational fears (his presumably) about lack of
        software tool determinism.

        "if you cant rebuild it - it doesnt exists" - me.

        Paul C.
      • John Brewer
        ... You re both right, depending on the design forces involved. See chapter 14 of Eric Evans Domain-Driven Design for when to use these and other
        Message 3 of 15 , Mar 1, 2004
        • 0 Attachment
          At 1:56 PM +1300 3/1/04, John Carter wrote:
          >I think having a one step build process is very important, and he thinks
          >having versioned libraries are important.

          You're both right, depending on the design forces involved. See
          chapter 14 of Eric Evans' "Domain-Driven Design" for when to use
          these and other integration patterns.
          --

          John Brewer
          Jera Design

          Extreme Programming FAQ: http://www.jera.com/techinfo/xpfaq.html
        • Jim Standley
          We have a fantasy, er, plan yet to be executed, that we can have subsytems in our system on different release schedules. The stable core should change as
          Message 4 of 15 , Mar 1, 2004
          • 0 Attachment
            We have a fantasy, er, plan yet to be executed, that we can have subsytems
            in our system on different release schedules. The "stable core" should
            change as infrequently as possible and we can add or update different
            product lines on business group schedules. One thought is we'll pretend the
            core is a shrink-wrap product delivered to the rest of the team. They only
            see pretty solid beta's and production cores. Product line components can
            be developed and shipped on a particular version of core.

            Are we completely nuts?

            At 09:29 PM 3/1/04 +0000, you wrote:

            >--- In extremeprogramming@yahoogroups.com, John Carter
            ><john.carter@t...> wrote:
            > > I have just realized that a major source of conflict between myself and
            > > another developer on the team is the following.
            > >
            > > I think having a one step build process is very important, and he thinks
            > > having versioned libraries are important.
            > >
            > > I believe you should be able to build the entire product directly from the
            > > source code in one step, ensuring no breakage. (Bitrot)
            > >
            > > He believes that each subteam should deliver "tested, closed, sealed and
            > > signed, versioned binary libraries" to the rest of the team.
            > >
            > > I believe that when you check into CVS mainline you are versioning,
            > > signing and sealing that you have compiled, tested and it works.
            > >
            > > I believe the only thing binary libraries gives you these days is a
            > > screwed up dependency tree at "make" time and maybe a second or two at
            > > link time.
            > >
            > > Does anyone have any opinions, pointers to "Best Practice"
            >literature /
            > > web sites to support either view.
          • Ron Jeffries
            ... No one is completely nuts. But you re on your way. ;- Why not just stay integrated, make whatever changes are needed by the business, and peel off a
            Message 5 of 15 , Mar 1, 2004
            • 0 Attachment
              On Monday, March 1, 2004, at 9:45:12 PM, Jim Standley wrote:

              > We have a fantasy, er, plan yet to be executed, that we can have subsytems
              > in our system on different release schedules. The "stable core" should
              > change as infrequently as possible and we can add or update different
              > product lines on business group schedules. One thought is we'll pretend the
              > core is a shrink-wrap product delivered to the rest of the team. They only
              > see pretty solid beta's and production cores. Product line components can
              > be developed and shipped on a particular version of core.

              > Are we completely nuts?

              No one is completely nuts. But you're on your way. ;->

              Why not just stay integrated, make whatever changes are needed by the
              business, and peel off a version when someone wants one?

              Ron Jeffries
              www.XProgramming.com
              Wisdom begins when we discover the difference between
              "That makes no sense" and "I don't understand". --Mary Doria Russell
            • Jeff Grigg
              (Very nice post, Brad.) I think a good question to ask at this point is... Are you building one application, with a number of subsystems? Or are you building
              Message 6 of 15 , Mar 1, 2004
              • 0 Attachment
                (Very nice post, Brad.)


                I think a good question to ask at this point is...

                Are you building one application, with a number of subsystems?

                Or are you building independent libraries, with different release
                schedules, that are assembled by application teams (with their own
                release schedules)?


                If the end result of your work is one coordinated release of a
                single application, then it would be best to version the source
                files, not the compiled libraries.

                On the other hand, if your shop is managing multiple projects that
                feed each other, with library reuse and independent release
                schedules, then you should probably version control the library
                releases. Think of it like this: Your operating system, IDE,
                compile tools, and third party component libraries are versioned
                independently of your releases. If some of your internal projects
                deliver "components" or libraries that are used "as if" they were
                third party libraries, then separate version control and an
                independent release schedule may be appropriate.


                At a previous employer, I ran the "distributed database" team,
                releasing our library to several application development teams that
                ran concurrently. While our requirements were driven by the
                application teams, our release schedules were only loosely
                coordinated. I think this was an appropriate situation for "signed
                and sealed" versioned library releases. It worked well for us.


                P.S. Something to be wary of:
                Some people will say that libraries with formal release controls
                improve reuse, ...as they improve "confidence" or something. They
                do not. The independent release schedules make it difficult to
                refactor useful reusable code into the libraries. Things that are
                difficult are, in effect, discouraged.


                --- Brad Appleton <brad@b...> wrote:
                > --- John Carter wrote:
                > > I think having a one step build process is very important,
                > > and he thinks having versioned libraries are important.
                >
                > You are both correct :-)
                >
                > > I believe you should be able to build the entire product
                > > directly from the source code in one step, ensuring no
                > > breakage. (Bitrot)
                >
                > Highly desirable. Not always feasible. It depends upon
                > relationships between subteams, and their respective "rhythms",
                > as well as geographical dispersement (and timezone), repository
                > size and performance, system-build performance and available
                > network resources, etc.
                >
                > > He believes that each subteam should deliver "tested, closed,
                > > sealed and signed, versioned binary libraries" to the rest of
                the team.
                >
                > [...]
              • Brad Appleton
                ... Excellent question! I think I was probably assuming the first of the two cases above. ... Agree. Even when versions are associated with what gets
                Message 7 of 15 , Mar 1, 2004
                • 0 Attachment
                  On Tue, Mar 02, 2004 at 03:30:58AM -0000, Jeff Grigg wrote:
                  > I think a good question to ask at this point is...
                  >
                  > Are you building one application, with a number of subsystems?
                  >
                  > Or are you building independent libraries, with different release
                  > schedules, that are assembled by application teams (with their own
                  > release schedules)?

                  Excellent question! I think I was probably assuming the first of
                  the two cases above.

                  > If the end result of your work is one coordinated release of a
                  > single application, then it would be best to version the source
                  > files, not the compiled libraries.

                  Agree. Even when "versions" are associated with what gets
                  delivered to the staging area, they typically refer to versions
                  of the source that produced the "staged" deliverables.

                  > On the other hand, if your shop is managing multiple projects that
                  > feed each other, with library reuse and independent release
                  > schedules, then you should probably version control the library
                  > releases. Think of it like this: Your operating system, IDE,
                  > compile tools, and third party component libraries are versioned
                  > independently of your releases. If some of your internal projects
                  > deliver "components" or libraries that are used "as if" they were
                  > third party libraries, then separate version control and an
                  > independent release schedule may be appropriate.

                  Another excellent point, and one I also agree with. There is
                  vendor and third-party stuff, and then there is stuff that may
                  be internal to your organization, but should be regarded as
                  internally vendor/3rd-party supplied to your particular
                  product and team. And in those cases versioning the delivered
                  binaries is recommended.

                  Some folks use a separate ThirdPartyRepository for such
                  purposes. One of the reasons is because its supplier and
                  release schedule is independent of the rest of the application.
                  Another reason is that if most of the elements are binary
                  in nature, it is often desirable to have a distinct storage
                  area with more efficient storage parameters/capacity (and
                  sometimes the repository can be configured so it is "tuned"
                  for performance based on knowledge of the kinds of elements
                  it will predominantly store).

                  And of course, if you get code delivered from any of those
                  third parties, you probably want to version it along with the
                  delivered binaries unless the binaries can be reproduced from
                  the code you were given (sometimes the source delivered is
                  insufficient for that).

                  If you have to modify any of that code from the third-party
                  for your own custom, value-added purpose, you will probably
                  want to use the ThirdPartyCodeline pattern (plus, get them
                  to incorporate your changes, unless the organization deems
                  them proprietary and is unwilling to submit them back to
                  the vendor).

                  > At a previous employer, I ran the "distributed database" team,
                  > releasing our library to several application development teams that
                  > ran concurrently. While our requirements were driven by the
                  > application teams, our release schedules were only loosely
                  > coordinated. I think this was an appropriate situation for "signed
                  > and sealed" versioned library releases. It worked well for us.

                  Definitely! (Good story too!)

                  > P.S. Something to be wary of:
                  > Some people will say that libraries with formal release controls
                  > improve reuse, ...as they improve "confidence" or something. They
                  > do not. The independent release schedules make it difficult to
                  > refactor useful reusable code into the libraries. Things that are
                  > difficult are, in effect, discouraged.

                  --
                  Brad Appleton <brad@...> www.bradapp.net
                  Software CM Patterns (www.scmpatterns.com)
                  Effective Teamwork, Practical Integration
                  "And miles to go before I sleep." -- Robert Frost
                Your message has been successfully submitted and would be delivered to recipients shortly.