Loading ...
Sorry, an error occurred while loading the content.
 

Re: [XP] Microsoft Research on TDD

Expand Messages
  • pooya shahbazian
    hi jeff thanks for introducing the paper.   Pooya Shahbazian http://www.pooyablog.blogfa.com ________________________________ From: JeffGrigg
    Message 1 of 10 , Dec 29, 2012
      hi jeff
      thanks for introducing the paper.

       
      Pooya Shahbazian
      http://www.pooyablog.blogfa.com



      ________________________________
      From: JeffGrigg <jeffreytoddgrigg@...>
      To: extremeprogramming@yahoogroups.com
      Sent: Sunday, December 30, 2012 5:01 AM
      Subject: [XP] Microsoft Research on TDD


       
      An important paper by Microsoft research was referenced indirectly in on e of the current threads.

      It is "Realizing quality improvement through test driven
      development: results and experiences of four industrial
      teams" by Nachiappan Nagappan, E. Michael Maximilien,
      Thirumalesh Bhat, and Laurie Williams.

      reference:
      http://link.springer.com/article/10.1007%2Fs10664-008-9062-z

      full paper:
      http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf

      It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

      HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

      The paper says...
      "Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM."

      I'm emphasizing the "as subjectively estimated by management" part.




      [Non-text portions of this message have been removed]
    • Adam Sroka
      Nice! And closely matches my own observations in the field. ... [Non-text portions of this message have been removed]
      Message 2 of 10 , Dec 29, 2012
        Nice! And closely matches my own observations in the field.


        On Sat, Dec 29, 2012 at 11:18 PM, Dave Smith <davewsmith@...> wrote:

        > **
        >
        >
        > Even if they were shooting from the hip, it still reads better to turn
        > their numbers around.
        >
        > By using normal development practices instead of TDD, teams can complete
        > projects 14 to 26 percent faster at the expense of defect densities that
        > are 150 to 900 percent higher.
        >
        > Dave
        >
        > On Sat, Dec 29, 2012 at 5:31 PM, JeffGrigg <jeffreytoddgrigg@...
        > >wrote:
        >
        >
        > > An important paper by Microsoft research was referenced indirectly in on
        > e
        > > of the current threads.
        > >
        > > It is "Realizing quality improvement through test driven
        > > development: results and experiences of four industrial
        > > teams" by Nachiappan Nagappan, E. Michael Maximilien,
        > > Thirumalesh Bhat, and Laurie Williams.
        > >
        > > reference:
        > > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
        > >
        > > full paper:
        > > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
        > >
        > > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
        > >
        > > HOWEVER, before everyone runs off and plans projects based on those
        > > numbers, I'd like to point out that while the defect reduction numbers
        > were
        > > measurements, the "cost increases" were "shoot from the hip estimates" by
        > > the bosses.
        > >
        > > The paper says...
        > > "Another interesting observation from the outcome measures in Table 3 is
        > > the increase in time to develop the features attributed to the usage of
        > the
        > > TDD practice, as subjectively estimated by management. The increase in
        > > development time ranges from 15% to 35%. From an efficacy perspective
        > this
        > > increase in development time is offset by the by the reduced maintenance
        > > costs due to the improvement in quality (Erdogmus and Williams 2003), an
        > > observation that was backed up the product teams at Microsoft and IBM."
        > >
        > > I'm emphasizing the "as subjectively estimated by management" part.
        > >
        > >
        > >
        > > ------------------------------------
        >
        > >
        > > To Post a message, send it to: extremeprogramming@...
        > >
        > > To Unsubscribe, send a blank message to:
        > > extremeprogramming-unsubscribe@...
        > >
        > > ad-free courtesy of objectmentor.comYahoo! Groups Links
        > >
        > >
        > >
        > >
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >


        [Non-text portions of this message have been removed]
      • George Dinwiddie
        Pooya, ... If you re interested in such papers, you might find http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of use. There are a few other
        Message 3 of 10 , Dec 30, 2012
          Pooya,

          On 12/29/12 11:48 PM, pooya shahbazian wrote:
          > hi jeff
          > thanks for introducing the paper.

          If you're interested in such papers, you might find
          http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
          use. There are a few other papers along this line.

          - George

          --
          ----------------------------------------------------------------------
          * George Dinwiddie * http://blog.gdinwiddie.com
          Software Development http://www.idiacomputing.com
          Consultant and Coach http://www.agilemaryland.org
          ----------------------------------------------------------------------
        • Ron Jeffries
          Hi Jeff, Good stuff. ... What is unfortunate in these studies, and in all the studies I ve seen, is that they seem to be estimating the time to produce
          Message 4 of 10 , Dec 30, 2012
            Hi Jeff,

            Good stuff.
            On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:

            > An important paper by Microsoft research was referenced indirectly in on e of the current threads.
            >
            > It is "Realizing quality improvement through test driven
            > development: results and experiences of four industrial
            > teams" by Nachiappan Nagappan, E. Michael Maximilien,
            > Thirumalesh Bhat, and Laurie Williams.
            >
            > reference:
            > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
            >
            > full paper:
            > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
            >
            > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

            > HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

            What is unfortunate in these studies, and in all the studies I've seen, is that they seem to be estimating the time to produce bug-free code on the TDD side, and buggy code on the non-TDD side, and do not add in the time to fix the bugs that need to be fixed. So not only are those numbers pulled from the hip or lower, they're probably not comparable even if they were accurate.

            But still good stuff!

            Ron Jeffries
            www.XProgramming.com
            Wisdom begins when we learn the difference between "that makes no sense" and "I don't understand". -- Mary Doria Russell



            [Non-text portions of this message have been removed]
          • M. Manca
            Il 30/12/2012 02:31, JeffGrigg ha scritto: Hi Jeff, I have similar results about quality but about cost I have a reduction between 25% to 45% compared with
            Message 5 of 10 , Dec 30, 2012
              Il 30/12/2012 02:31, JeffGrigg ha scritto:
              Hi Jeff,
              I have similar results about quality but about cost I have a reduction
              between 25% to 45% compared with similar embedded projects.
              >
              >
              > An important paper by Microsoft research was referenced indirectly in
              > on e of the current threads.
              >
              > It is "Realizing quality improvement through test driven
              > development: results and experiences of four industrial
              > teams" by Nachiappan Nagappan, E. Michael Maximilien,
              > Thirumalesh Bhat, and Laurie Williams.
              >
              > reference:
              > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
              >
              > full paper:
              > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
              >
              > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
              >
              > HOWEVER, before everyone runs off and plans projects based on those
              > numbers, I'd like to point out that while the defect reduction numbers
              > were measurements, the "cost increases" were "shoot from the hip
              > estimates" by the bosses.
              >
              > The paper says...
              > "Another interesting observation from the outcome measures in Table 3
              > is the increase in time to develop the features attributed to the
              > usage of the TDD practice, as subjectively estimated by management.
              > The increase in development time ranges from 15% to 35%. From an
              > efficacy perspective this increase in development time is offset by
              > the by the reduced maintenance costs due to the improvement in quality
              > (Erdogmus and Williams 2003), an observation that was backed up the
              > product teams at Microsoft and IBM."
              >
              > I'm emphasizing the "as subjectively estimated by management" part.
              >
              >



              [Non-text portions of this message have been removed]
            • George Dinwiddie
              Ron, ... Yes, I can write code really fast if it doesn t have to work. - George -- ... * George Dinwiddie * http://blog.gdinwiddie.com
              Message 6 of 10 , Dec 30, 2012
                Ron,

                On 12/30/12 6:10 AM, Ron Jeffries wrote:
                > Hi Jeff,
                >
                > Good stuff.
                > On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:
                >
                >> An important paper by Microsoft research was referenced indirectly in on e of the current threads.
                >>
                >> It is "Realizing quality improvement through test driven
                >> development: results and experiences of four industrial
                >> teams" by Nachiappan Nagappan, E. Michael Maximilien,
                >> Thirumalesh Bhat, and Laurie Williams.
                >>
                >> reference:
                >> http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                >>
                >> full paper:
                >> http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                >>
                >> It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
                >
                >> HOWEVER, before everyone runs off and plans projects based on those
                >> numbers, I'd like to point out that while the defect reduction
                >> numbers were measurements, the "cost increases" were "shoot from
                >> the hip estimates" by the bosses.
                >
                > What is unfortunate in these studies, and in all the studies I've
                > seen, is that they seem to be estimating the time to produce bug-free
                > code on the TDD side, and buggy code on the non-TDD side, and do not
                > add in the time to fix the bugs that need to be fixed. So not only
                > are those numbers pulled from the hip or lower, they're probably not
                > comparable even if they were accurate.

                Yes, I can write code really fast if it doesn't have to work.

                - George

                --
                ----------------------------------------------------------------------
                * George Dinwiddie * http://blog.gdinwiddie.com
                Software Development http://www.idiacomputing.com
                Consultant and Coach http://www.agilemaryland.org
                ----------------------------------------------------------------------
              • pooya shahbazian
                hi thanks, it s so useful  Pooya Shahbazian http://www.pooyablog.blogfa.com/ ________________________________ From: George Dinwiddie
                Message 7 of 10 , Dec 30, 2012
                  hi
                  thanks, it's so useful 

                  Pooya Shahbazian
                  http://www.pooyablog.blogfa.com/



                  ________________________________
                  From: George Dinwiddie <lists@...>
                  To: extremeprogramming@yahoogroups.com
                  Sent: Sunday, December 30, 2012 2:03 PM
                  Subject: Re: [XP] Microsoft Research on TDD


                   

                  Pooya,

                  On 12/29/12 11:48 PM, pooya shahbazian wrote:
                  > hi jeff
                  > thanks for introducing the paper.

                  If you're interested in such papers, you might find
                  http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
                  use. There are a few other papers along this line.

                  - George

                  --
                  ----------------------------------------------------------
                  * George Dinwiddie * http://blog.gdinwiddie.com
                  Software Development http://www.idiacomputing.com
                  Consultant and Coach http://www.agilemaryland.org
                  ----------------------------------------------------------




                  [Non-text portions of this message have been removed]
                • JeffGrigg
                  ... Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the
                  Message 8 of 10 , Dec 30, 2012
                    --- "M. Manca" <m.manca@...> wrote:
                    > Hi Jeff,
                    > I have similar results about quality but about cost I
                    > have a reduction between 25% to 45% compared with
                    > similar embedded projects.

                    Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the decreased maintenance costs of having fewer bugs, and more readable and maintainable code, and in-place test suites to prevent regression, are often overlooked.

                    Even during the initial development phase, I've seen numerous benefits to the TDD approach. A friend of mine in the early 2000's studied a number of projects done in the same company with similar teams, similar management, the same tools, etc. -- giving him a good comparison of XP to their more conventional approaches. His numbers showed that not using XP cost the company two to four times as much -- just during the initial development project.
                  Your message has been successfully submitted and would be delivered to recipients shortly.