Loading ...
Sorry, an error occurred while loading the content.
 

RE: [scrumdevelopment] Scrum and Architecture

Expand Messages
  • Ken Schwaber
    The problem with architecting and designing everything upfront is that that architecture and design is in response to requirements. If you take industry
    Message 1 of 10 , Jun 12, 2008

      The problem with architecting and designing everything upfront is that that architecture and design is in response to requirements. If you take industry statistics that 35% of those requirements change, and that over 50% of them are rarely or never used, much of that work is a waste. In Scrum we use emergent architecture and design. The non-functional requirements that drive the architecture and design are usually high value, so high in the product backlog. We have to build at least one piece of business facing functionality every Sprint, so we only built enough architecture and design to support that piece of business functionality with adequate architecture and design to meet the non-functional requirements (stability, security, response time, etc.). Then, every subsequent Sprint, more and more of the architecture and design emerge, in response to more and more business functionality. The purpose of this is only to build architecture and design in response to actual, high priority business needs. As we do so, we meet three needs: we prove that the architecture and design actually work, we have something to get feedback from the user, and we start improving our estimating. As the project or release proceeds, architectural and design work declines and business functionality increases.

      In order to employ emergent architecture, every Sprint must leave behind clean, commented, refactored work. Otherwise emergence hits the wall within six Sprints (or sooner, depending on how bad the morass is).

      Ken

       

       

      From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Wolfgang Schulze Zachau
      Sent: Thursday, June 12, 2008 6:32 AM
      To: scrumdevelopment@yahoogroups.com
      Subject: RE: [scrumdevelopment] Scrum and Architecture

       

      Hi Sue,

       

      seeing that nobody else has responded to this yet, here's my 2 cents worth:

       

      The whole idea of doing your architectural/design work as you go along comes from the experience of projects that are so big and complex that any attempt to do all the design as BDUF (big design up front) will stall the project completely. In those situations it is much better to just do a little basic architectural design and then start implementing features in an incremental fashion. You should then develop your architecture as you go along, i.e. as the additional features require it. And this works well under the following two conditions:

      a) you use TDD, refactoring and automated regression testing RUTHLESSLY FOR EVERYTHING.

      b) you have experienced people on the team (i.e. people with experience in all of the items in a) above, plus experience in architecture and design)

       

      If you don't have these two things in the team, then you will need to either have to get them from the outside or you'll have to accept some deficiencies when it comes to architecture.

       

      So why is this a good idea? Well, for a number of reasons:

      1) It avoids stalling the project when the BDUF would be extremely complex or large. Often in large projects there is a tendency to go around in circles at the design stage, because the requirements keep on changing and/or the stakeholders can't make up their minds and/or new requirements/acceptance criteria/etc are discovered.

      2) It avoids waste. You only build as much architecture as you need, at any point in time. When you need more, you refactor.

       

      Having said all of this, nobody says you should't do any planning at all. That would be just plain stupid.

       

      Regards,

      Wolfgang

       

       


      From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of sue.bramhall
      Sent: 11 June 2008 15:37
      To: scrumdevelopment@yahoogroups.com
      Subject: [scrumdevelopment] Scrum and Architecture

      Yep we are now working Just in Time i tell my new converts to Scum,
      but they cannot grasp this concept initially having had previous
      experience of planning, analysis, solution design docs etc. So does
      anyone out there have a good simple example on how to explain why this
      is now a good idea, bar actually having to go through the experience?

    • Cory Foy
      ... Is it a good idea? Did the planning, analysis and solution design docs work well for them? Was the customer happy? I ask that because somewhere there is
      Message 2 of 10 , Jun 12, 2008
        --- In scrumdevelopment@yahoogroups.com, "sue.bramhall"
        <sbramhall@...> wrote:
        > Yep we are now working Just in Time i tell my new converts to Scum,
        > but they cannot grasp this concept initially having had previous
        > experience of planning, analysis, solution design docs etc. So does
        > anyone out there have a good simple example on how to explain why this
        > is now a good idea, bar actually having to go through the experience?

        Is it a good idea? Did the planning, analysis and solution design docs
        work well for them? Was the customer happy?

        I ask that because somewhere there is some team where it does work.
        But otherwise I'll steal a bit from the Lean stuff - the time from
        requirements to having working software in the hands of the one
        defining the requirements could be construed as waste. So we want to
        work in increments, delivering running, tested software as rapidly as
        possible to get feedback from the customer.

        Why this is challenging to most teams relates to another Lean example
        - Toyota. Most car companies did all of the design upfront so they
        could send the die specs for the metal to be stamped. This meant that
        if the design changed, they had to redo the specs, redo the dies, and
        throw away everything.

        What Toyota did was work /with/ the die makers. The die cutters knew
        what would be likely to change, and what wouldn't, and could cut out a
        majority of the die, and from experience, leave parts of it till later
        in the process when they knew more.

        This is scary for many groups, but when you have experienced folks,
        they usually have a good idea of what is going to change and what
        isn't. Trust them to do the right thing.

        --
        Cory Foy
        http://www.cornetdesign.com
        http://www.agileflorida.com
      • atmchuck
        One of the benefits I ve found with good architecture is easily extensible code. Put another way, a good framework allows me to easily add new Forms (or new
        Message 3 of 10 , Jun 12, 2008
          One of the benefits I've found with good architecture is easily
          extensible code. Put another way, a good framework allows me to
          easily add new Forms (or new modules) to an application. It is
          somewhat straightforward to measure stability, security, response
          time (mentioned below), and therefore, easy to "sell" to the product
          owner.

          Has anyone included some kind of "ease of future enhancements" line
          item as a way to sell some of the design work?

          What are some ways that this line item could be measured?

          Am I just hanging on to BDUF?<g>

          I hope I'm not hijacking your post.

          Thanks,
          Chuck

          --- In scrumdevelopment@yahoogroups.com, "Ken Schwaber"
          <ken.schwaber@...> wrote:
          >
          > The problem with architecting and designing everything upfront is
          that that
          > architecture and design is in response to requirements. If you take
          industry
          > statistics that 35% of those requirements change, and that over 50%
          of them
          > are rarely or never used, much of that work is a waste. In Scrum we
          use
          > emergent architecture and design. The non-functional requirements
          that drive
          > the architecture and design are usually high value, so high in the
          product
          > backlog. We have to build at least one piece of business facing
          > functionality every Sprint, so we only built enough architecture
          and design
          > to support that piece of business functionality with adequate
          architecture
          > and design to meet the non-functional requirements (stability,
          security,
          > response time, etc.). Then, every subsequent Sprint, more and more
          of the
          > architecture and design emerge, in response to more and more
          business
          > functionality. The purpose of this is only to build architecture
          and design
          > in response to actual, high priority business needs. As we do so,
          we meet
          > three needs: we prove that the architecture and design actually
          work, we
          > have something to get feedback from the user, and we start
          improving our
          > estimating. As the project or release proceeds, architectural and
          design
          > work declines and business functionality increases.
          >
          > In order to employ emergent architecture, every Sprint must leave
          behind
          > clean, commented, refactored work. Otherwise emergence hits the
          wall within
          > six Sprints (or sooner, depending on how bad the morass is).
          >
          > Ken
          >
          >
          >
          >
          >
          > From: scrumdevelopment@yahoogroups.com
          > [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of Wolfgang
          Schulze
          > Zachau
          > Sent: Thursday, June 12, 2008 6:32 AM
          > To: scrumdevelopment@yahoogroups.com
          > Subject: RE: [scrumdevelopment] Scrum and Architecture
          >
          >
          >
          > Hi Sue,
          >
          >
          >
          > seeing that nobody else has responded to this yet, here's my 2
          cents worth:
          >
          >
          >
          > The whole idea of doing your architectural/design work as you go
          along comes
          > from the experience of projects that are so big and complex that
          any attempt
          > to do all the design as BDUF (big design up front) will stall the
          project
          > completely. In those situations it is much better to just do a
          little basic
          > architectural design and then start implementing features in an
          incremental
          > fashion. You should then develop your architecture as you go along,
          i.e. as
          > the additional features require it. And this works well under the
          following
          > two conditions:
          >
          > a) you use TDD, refactoring and automated regression testing
          RUTHLESSLY FOR
          > EVERYTHING.
          >
          > b) you have experienced people on the team (i.e. people with
          experience in
          > all of the items in a) above, plus experience in architecture and
          design)
          >
          >
          >
          > If you don't have these two things in the team, then you will need
          to either
          > have to get them from the outside or you'll have to accept some
          deficiencies
          > when it comes to architecture.
          >
          >
          >
          > So why is this a good idea? Well, for a number of reasons:
          >
          > 1) It avoids stalling the project when the BDUF would be extremely
          complex
          > or large. Often in large projects there is a tendency to go around
          in
          > circles at the design stage, because the requirements keep on
          changing
          > and/or the stakeholders can't make up their minds and/or new
          > requirements/acceptance criteria/etc are discovered.
          >
          > 2) It avoids waste. You only build as much architecture as you
          need, at any
          > point in time. When you need more, you refactor.
          >
          >
          >
          > Having said all of this, nobody says you should't do any planning
          at all.
          > That would be just plain stupid.
          >
          >
          >
          > Regards,
          >
          > Wolfgang
          >
          >
          >
          >
          >
          > _____
          >
          > From: scrumdevelopment@yahoogroups.com
          > [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of sue.bramhall
          > Sent: 11 June 2008 15:37
          > To: scrumdevelopment@yahoogroups.com
          > Subject: [scrumdevelopment] Scrum and Architecture
          >
          > Yep we are now working Just in Time i tell my new converts to Scum,
          > but they cannot grasp this concept initially having had previous
          > experience of planning, analysis, solution design docs etc. So does
          > anyone out there have a good simple example on how to explain why
          this
          > is now a good idea, bar actually having to go through the
          experience?
          >
        • Mark Baenziger
          Sue, The only thing I d like to add to is a brief comment on systems architecture (things like what operating system am I running the database on? how many
          Message 4 of 10 , Jun 12, 2008
            Sue,

            The only thing I'd like to add to is a brief comment on systems' architecture (things like "what operating system am I running the database on?  how many processors and memory does my tomcat server have?  What model of F5 BigIP do I use for my load-balancing?  What single-sign-on framework should I use?").  There are a few issues with systems and networking devices that are slightly different than software architecture decisions:
            • Unit-testing analogs exist for systems and network architectures:  Though there is no formal unit testing framework that I'm aware of that applies to systems/networking devices, most IT organizations have something similar in the form of a monitoring subsystem (Nagios, IPMonitor, HP Openview, etc.,) that can be used to measure the ability of the systems/network architecture to meet non-functional requirements (i.e., monitors that confirm that CPU < 10% utilization, no paging, <10ms latency for authentication, etc.,).  If you ensure that systems/network monitoring is part of your definition of done, this helps ensure that you can modify your underlying systems/network architecture in a fashion similar to the way you modify software architecture in the emergent fashion described below.  At the very least, it allows you to catch problems early enough to have a chance of meeting your goal of delivering a working product at the end of each sprint.
            • Hardware/Network purchase decisions often can't be completed in a sprint:  When in Sprint 5 you realize that one of your stories requires you rethink your Sprint 1 decision to go with a simplified network or systems architecture, the cost may be rather high, and the redesign decision (i.e., "we're upgrading our Sun 8G T1000's to 64G Sun T2000's and replacing our software load-balancing with F5 BigIP's") may not be something you can implement in your sprint time.  Sometimes that's just the way things are, but you can often speed some of these decisions by abstracting your physical/network hardware by relying on virtualization (i.e., "upgrading" your servers can be simply flipping a switch in VMWare that grants your server 4x as many processors and 16x as much memory).
            • It's hard to refactor/redesign a $2 million dollar SAN purchase:  A decision to refactor or design something in a software architecture is often an expensive decision, but it doesn't have quite the "oomph" to your CFO as a decision to throw away a piece of expensive hardware before it's fully depreciated because you're using emergent architecture and just discovered that you need to upgrade your SAN to a more expensive version.  But it is a risk you take.  I think this challenge is a straw man that actually comes up fairly infrequently, but it's one you still may have to face.  The best way to address the CFO fallout that I've found is to point out that the cost of your upgraded equipment purchased at Sprint 10 is cheaper than it would have been at Sprint 1 (i.e., use Moore's law in your favor).  That is - a 10TB SAN purchased in Sprint 1 is more expensive than the combined cost of a 1TB SAN purchased in Sprint 1 and a 10TB SAN purchased in Sprint 10.

            Hope that helps.
            Mark

            Mark Baenziger
            mmbaenz@... or mark@...

            ----- Original Message ----
            From: Ken Schwaber <ken.schwaber@...>
            To: scrumdevelopment@yahoogroups.com
            Sent: Thursday, June 12, 2008 5:34:46 AM
            Subject: RE: [scrumdevelopment] Scrum and Architecture

            The problem with architecting and designing everything upfront is that that architecture and design is in response to requirements. If you take industry statistics that 35% of those requirements change, and that over 50% of them are rarely or never used, much of that work is a waste. In Scrum we use emergent architecture and design. The non-functional requirements that drive the architecture and design are usually high value, so high in the product backlog. We have to build at least one piece of business facing functionality every Sprint, so we only built enough architecture and design to support that piece of business functionality with adequate architecture and design to meet the non-functional requirements (stability, security, response time, etc.). Then, every subsequent Sprint, more and more of the architecture and design emerge, in response to more and more business functionality. The purpose of this is only to build architecture and design in response to actual, high priority business needs. As we do so, we meet three needs: we prove that the architecture and design actually work, we have something to get feedback from the user, and we start improving our estimating. As the project or release proceeds, architectural and design work declines and business functionality increases.

            In order to employ emergent architecture, every Sprint must leave behind clean, commented, refactored work. Otherwise emergence hits the wall within six Sprints (or sooner, depending on how bad the morass is).

            Ken

             



            __________________________________________________
            Do You Yahoo!?
            Tired of spam? Yahoo! Mail has the best spam protection around
            http://mail.yahoo.com
          • Cees de Groot
            Just a very short comment on TDD and No-Architecture-Upfront etcetera: I find it useful, at the beginning of a project, to make an outline of the architecture
            Message 5 of 10 , Jun 12, 2008
              Just a very short comment on TDD and No-Architecture-Upfront etcetera:

              I find it useful, at the beginning of a project, to make an outline of
              the architecture with the team as we would like it to be at the end of
              a project. I usually call it the "Taj-Mahal at the Horizon". It is
              always in view, and if our code would ever be that way it'd be Damn
              Great Code[tm], but you will never get there.

              With such an outline in mind, I find it easier to make the daily micro
              decisions and it also helps guiding the discussion spikes that pop up
              now and then. In my current project, we've made quite a number of
              adjustments to the initial charcoal sketch and just because it was
              only a sketch, that was not a big issue (we refactored the whole lower
              layer - the data access etcetera - last week in what should be one of
              the last sprints). However, if you squint a bit while looking at our
              code it looks quite a bit like what we set out to built (only more
              adapted to the problem at hand ;-)).

              Some (a lot of?) craftsman also seem to work that way: they make a
              pencil sketch, evaluate it, maybe adjust it a bit, then throw it away
              and set out to work.
            • George Dinwiddie
              Chuck, ... Yes, the appropriate frame work makes a bit difference. In my experience, though, designing and building the framework ahead of time rarely results
              Message 6 of 10 , Jun 12, 2008
                Chuck,

                atmchuck wrote:
                > One of the benefits I've found with good architecture is easily
                > extensible code. Put another way, a good framework allows me to
                > easily add new Forms (or new modules) to an application. It is
                > somewhat straightforward to measure stability, security, response
                > time (mentioned below), and therefore, easy to "sell" to the product
                > owner.

                Yes, the appropriate frame work makes a bit difference. In my
                experience, though, designing and building the framework ahead of time
                rarely results in the one I need for the particular application.

                What has worked a lot better for me is to work through the application,
                user story by user story, being sensitive to the design aspects as I go.
                By refactoring to remove duplication and add expression of intent, I
                effectively extract the framework from the business application as I
                build it. This has worked extraordinarily well for me.

                - George

                --
                ----------------------------------------------------------------------
                * George Dinwiddie * http://blog.gdinwiddie.com
                Software Development http://www.idiacomputing.com
                Consultant and Coach http://www.agilemaryland.org
                ----------------------------------------------------------------------
              • Kiran Thakkar
                Scrum and Architecture; I have seen failed projects when organizations attempted to define heavy up-front architecture and design. How many of you have delayed
                Message 7 of 10 , Jun 12, 2008

                  Scrum and Architecture;

                  I have seen failed projects when organizations attempted to define heavy up-front architecture and design. How many of you have delayed your product delivery just because it does not perform as expected or does not scale? In order to improve performance in the 11th month; you simply have to redesign the whole thing. Again, such delay and rework are simply results into waste of resources, unhappy customers and so on.

                  All of applications are under constant pressure to change and adapt. As Ken and others mentioned that, you want to start with enough architecture and design to build few priority feature functions. Refactoring will allow you to change and adjust architectural framework to changing requirements.

                  You could define and decide on how much architecture and design effort is necessary for your project. Few factors you need to consider such as:

                  1.      Pure new product development or major addition to existing product line would require extra upfront effort on architecture and design work.

                  2.      For large scale and distributed Scrum, large-scale applications with complex architecture, you need to ensure that multiple Scrum teams adhere to standard architecture principles and maintain consistencies. For large scale and distributed Scrum, you could have a dedicated architecture team in a pre-release planning (Sprint Zero) to work out the architectural unknowns and provide guidelines to all Scrum teams.  

                  3.      Migration from legacy code base to new technology would require dedicated team and extra architecture effort.

                   

                  Regards,

                  Kiran Thakkar, MBA, MS,PMP CSM/CSP




                • Roy Morien
                  I remember a most alarming scene at one place that I worked, as a contractor. This was on Friday afternoon, immediately preceding the Monday that a system was
                  Message 8 of 10 , Jun 12, 2008
                    I remember a most alarming scene at one place that I worked, as a contractor. This was on Friday afternoon, immediately preceding the Monday that a system was to 'go live', after about a year of development activity. The project manager was wandering around the office wringing her hands, looking awfully worried, almost distraught, and repeating over and over aghain "Oh, I hope it works  ... Oh, Boy, I sure hope it works ... My God, what will happen if it doesn't work".
                     
                    I was fascinated watching this extraordinary sight. My thoughts were basically "Well, it's a bit late to be worrying about whether or not it will work. Why hasn't this been proven and validated a long time ago".
                     
                    Another rather bizarre scene that I observed, at an IS conference (ACIS) was an academic from the host university presenting a paper, in which he said "we have a problem. the students do their requirements analysis in first term, including an ER Model, and they get good marks for that as excellent documentation/. But then, when they come to implement it in second term, they find it can't be implemented without a lot of changes to the ER Model and the requirements. What can we do about this, I don't know".
                     
                    On this occasion my thoughts were 'Why don't you analyse, design and implement all at the same time, which allows you to prove and validate the requirements and the ER Model at every step of the way?" This conference was in 1993, as I recall. Unfortunately, I don't think universities and colleges in general have changed this approach to system development in their teaching.  

                    Recently, an academic colleague in a Thai university where I am teaching said "I don't understand this agile development that you talk about. It seems too simple".  Another quotable quote was when we were discussing estimating, and he said estimating in an agile approach didn't seem very accurate or successful. When I asked him how the traditional developer overcame this problem (of exceeding estimates) he said "Oh, that is why there are penalties and fines built into the contract". He seemed a little bemused when I suggested that the client would prefer working software rather than financial gain from penalties and fines.
                     
                    I find that so many practitioners and IS academics and others just cannot get their head around the idea that it is not necessary to know everything before you can do anything in a project. There is always this naive belief that estimates and plans and up-front requirements determination are essential, and will be correct and basically unchangeable. The tradition of Waterfall Development seems so entrenched in their thinking. There also seems to be a view that diligently following the methodology and specified development process is enough to claim success ... the idea that if you follow every step, fill out every form, lodge every report and have all required meetings, then the project must be deemed successful. Failure of the software to meet contemporaneous requirements is cause by other factors, not their diligent adherence to the development process.
                     
                    It's a strange world out there in Development Land.
                     
                    Regards,
                    Roy Morien


                    To: scrumdevelopment@yahoogroups.com
                    From: kthakkar@...
                    Date: Thu, 12 Jun 2008 20:05:44 -0700
                    Subject: [scrumdevelopment] Re: Scrum and Architecture

                    Scrum and Architecture;
                    I have seen failed projects when organizations attempted to define heavy up-front architecture and design. How many of you have delayed your product delivery just because it does not perform as expected or does not scale? In order to improve performance in the 11th month; you simply have to redesign the whole thing. Again, such delay and rework are simply results into waste of resources, unhappy customers and so on.
                    All of applications are under constant pressure to change and adapt. As Ken and others mentioned that, you want to start with enough architecture and design to build few priority feature functions. Refactoring will allow you to change and adjust architectural framework to changing requirements.
                    You could define and decide on how much architecture and design effort is necessary for your project. Few factors you need to consider such as:
                    1.      Pure new product development or major addition to existing product line would require extra upfront effort on architecture and design work.
                    2.      For large scale and distributed Scrum, large-scale applications with complex architecture, you need to ensure that multiple Scrum teams adhere to standard architecture principles and maintain consistencies. For large scale and distributed Scrum, you could have a dedicated architecture team in a pre-release planning (Sprint Zero) to work out the architectural unknowns and provide guidelines to all Scrum teams.  
                    3.      Migration from legacy code base to new technology would require dedicated team and extra architecture effort.
                     
                    Regards,
                    Kiran Thakkar, MBA,MS,PMP CSM/CSP







                    Hotmail on your mobile. Never miss another e-mail with
                  Your message has been successfully submitted and would be delivered to recipients shortly.