Loading ...
Sorry, an error occurred while loading the content.
 

Re: [XP] Microsoft Research on TDD

Expand Messages
  • Dave Smith
    Even if they were shooting from the hip, it still reads better to turn their numbers around. By using normal development practices instead of TDD, teams can
    Message 1 of 10 , Dec 29, 2012
      Even if they were shooting from the hip, it still reads better to turn
      their numbers around.

      By using normal development practices instead of TDD, teams can complete
      projects 14 to 26 percent faster at the expense of defect densities that
      are 150 to 900 percent higher.

      Dave

      On Sat, Dec 29, 2012 at 5:31 PM, JeffGrigg <jeffreytoddgrigg@...>wrote:

      > An important paper by Microsoft research was referenced indirectly in on e
      > of the current threads.
      >
      > It is "Realizing quality improvement through test driven
      > development: results and experiences of four industrial
      > teams" by Nachiappan Nagappan, E. Michael Maximilien,
      > Thirumalesh Bhat, and Laurie Williams.
      >
      > reference:
      > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
      >
      > full paper:
      > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
      >
      > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
      >
      > HOWEVER, before everyone runs off and plans projects based on those
      > numbers, I'd like to point out that while the defect reduction numbers were
      > measurements, the "cost increases" were "shoot from the hip estimates" by
      > the bosses.
      >
      > The paper says...
      > "Another interesting observation from the outcome measures in Table 3 is
      > the increase in time to develop the features attributed to the usage of the
      > TDD practice, as subjectively estimated by management. The increase in
      > development time ranges from 15% to 35%. From an efficacy perspective this
      > increase in development time is offset by the by the reduced maintenance
      > costs due to the improvement in quality (Erdogmus and Williams 2003), an
      > observation that was backed up the product teams at Microsoft and IBM."
      >
      > I'm emphasizing the "as subjectively estimated by management" part.
      >
      >
      >
      > ------------------------------------
      >
      > To Post a message, send it to: extremeprogramming@...
      >
      > To Unsubscribe, send a blank message to:
      > extremeprogramming-unsubscribe@...
      >
      > ad-free courtesy of objectmentor.comYahoo! Groups Links
      >
      >
      >
      >


      [Non-text portions of this message have been removed]
    • pooya shahbazian
      hi jeff thanks for introducing the paper.   Pooya Shahbazian http://www.pooyablog.blogfa.com ________________________________ From: JeffGrigg
      Message 2 of 10 , Dec 29, 2012
        hi jeff
        thanks for introducing the paper.

         
        Pooya Shahbazian
        http://www.pooyablog.blogfa.com



        ________________________________
        From: JeffGrigg <jeffreytoddgrigg@...>
        To: extremeprogramming@yahoogroups.com
        Sent: Sunday, December 30, 2012 5:01 AM
        Subject: [XP] Microsoft Research on TDD


         
        An important paper by Microsoft research was referenced indirectly in on e of the current threads.

        It is "Realizing quality improvement through test driven
        development: results and experiences of four industrial
        teams" by Nachiappan Nagappan, E. Michael Maximilien,
        Thirumalesh Bhat, and Laurie Williams.

        reference:
        http://link.springer.com/article/10.1007%2Fs10664-008-9062-z

        full paper:
        http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf

        It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

        HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

        The paper says...
        "Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM."

        I'm emphasizing the "as subjectively estimated by management" part.




        [Non-text portions of this message have been removed]
      • Adam Sroka
        Nice! And closely matches my own observations in the field. ... [Non-text portions of this message have been removed]
        Message 3 of 10 , Dec 29, 2012
          Nice! And closely matches my own observations in the field.


          On Sat, Dec 29, 2012 at 11:18 PM, Dave Smith <davewsmith@...> wrote:

          > **
          >
          >
          > Even if they were shooting from the hip, it still reads better to turn
          > their numbers around.
          >
          > By using normal development practices instead of TDD, teams can complete
          > projects 14 to 26 percent faster at the expense of defect densities that
          > are 150 to 900 percent higher.
          >
          > Dave
          >
          > On Sat, Dec 29, 2012 at 5:31 PM, JeffGrigg <jeffreytoddgrigg@...
          > >wrote:
          >
          >
          > > An important paper by Microsoft research was referenced indirectly in on
          > e
          > > of the current threads.
          > >
          > > It is "Realizing quality improvement through test driven
          > > development: results and experiences of four industrial
          > > teams" by Nachiappan Nagappan, E. Michael Maximilien,
          > > Thirumalesh Bhat, and Laurie Williams.
          > >
          > > reference:
          > > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
          > >
          > > full paper:
          > > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
          > >
          > > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
          > >
          > > HOWEVER, before everyone runs off and plans projects based on those
          > > numbers, I'd like to point out that while the defect reduction numbers
          > were
          > > measurements, the "cost increases" were "shoot from the hip estimates" by
          > > the bosses.
          > >
          > > The paper says...
          > > "Another interesting observation from the outcome measures in Table 3 is
          > > the increase in time to develop the features attributed to the usage of
          > the
          > > TDD practice, as subjectively estimated by management. The increase in
          > > development time ranges from 15% to 35%. From an efficacy perspective
          > this
          > > increase in development time is offset by the by the reduced maintenance
          > > costs due to the improvement in quality (Erdogmus and Williams 2003), an
          > > observation that was backed up the product teams at Microsoft and IBM."
          > >
          > > I'm emphasizing the "as subjectively estimated by management" part.
          > >
          > >
          > >
          > > ------------------------------------
          >
          > >
          > > To Post a message, send it to: extremeprogramming@...
          > >
          > > To Unsubscribe, send a blank message to:
          > > extremeprogramming-unsubscribe@...
          > >
          > > ad-free courtesy of objectmentor.comYahoo! Groups Links
          > >
          > >
          > >
          > >
          >
          > [Non-text portions of this message have been removed]
          >
          >
          >


          [Non-text portions of this message have been removed]
        • George Dinwiddie
          Pooya, ... If you re interested in such papers, you might find http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of use. There are a few other
          Message 4 of 10 , Dec 30, 2012
            Pooya,

            On 12/29/12 11:48 PM, pooya shahbazian wrote:
            > hi jeff
            > thanks for introducing the paper.

            If you're interested in such papers, you might find
            http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
            use. There are a few other papers along this line.

            - George

            --
            ----------------------------------------------------------------------
            * George Dinwiddie * http://blog.gdinwiddie.com
            Software Development http://www.idiacomputing.com
            Consultant and Coach http://www.agilemaryland.org
            ----------------------------------------------------------------------
          • Ron Jeffries
            Hi Jeff, Good stuff. ... What is unfortunate in these studies, and in all the studies I ve seen, is that they seem to be estimating the time to produce
            Message 5 of 10 , Dec 30, 2012
              Hi Jeff,

              Good stuff.
              On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:

              > An important paper by Microsoft research was referenced indirectly in on e of the current threads.
              >
              > It is "Realizing quality improvement through test driven
              > development: results and experiences of four industrial
              > teams" by Nachiappan Nagappan, E. Michael Maximilien,
              > Thirumalesh Bhat, and Laurie Williams.
              >
              > reference:
              > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
              >
              > full paper:
              > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
              >
              > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

              > HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

              What is unfortunate in these studies, and in all the studies I've seen, is that they seem to be estimating the time to produce bug-free code on the TDD side, and buggy code on the non-TDD side, and do not add in the time to fix the bugs that need to be fixed. So not only are those numbers pulled from the hip or lower, they're probably not comparable even if they were accurate.

              But still good stuff!

              Ron Jeffries
              www.XProgramming.com
              Wisdom begins when we learn the difference between "that makes no sense" and "I don't understand". -- Mary Doria Russell



              [Non-text portions of this message have been removed]
            • M. Manca
              Il 30/12/2012 02:31, JeffGrigg ha scritto: Hi Jeff, I have similar results about quality but about cost I have a reduction between 25% to 45% compared with
              Message 6 of 10 , Dec 30, 2012
                Il 30/12/2012 02:31, JeffGrigg ha scritto:
                Hi Jeff,
                I have similar results about quality but about cost I have a reduction
                between 25% to 45% compared with similar embedded projects.
                >
                >
                > An important paper by Microsoft research was referenced indirectly in
                > on e of the current threads.
                >
                > It is "Realizing quality improvement through test driven
                > development: results and experiences of four industrial
                > teams" by Nachiappan Nagappan, E. Michael Maximilien,
                > Thirumalesh Bhat, and Laurie Williams.
                >
                > reference:
                > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                >
                > full paper:
                > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                >
                > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
                >
                > HOWEVER, before everyone runs off and plans projects based on those
                > numbers, I'd like to point out that while the defect reduction numbers
                > were measurements, the "cost increases" were "shoot from the hip
                > estimates" by the bosses.
                >
                > The paper says...
                > "Another interesting observation from the outcome measures in Table 3
                > is the increase in time to develop the features attributed to the
                > usage of the TDD practice, as subjectively estimated by management.
                > The increase in development time ranges from 15% to 35%. From an
                > efficacy perspective this increase in development time is offset by
                > the by the reduced maintenance costs due to the improvement in quality
                > (Erdogmus and Williams 2003), an observation that was backed up the
                > product teams at Microsoft and IBM."
                >
                > I'm emphasizing the "as subjectively estimated by management" part.
                >
                >



                [Non-text portions of this message have been removed]
              • George Dinwiddie
                Ron, ... Yes, I can write code really fast if it doesn t have to work. - George -- ... * George Dinwiddie * http://blog.gdinwiddie.com
                Message 7 of 10 , Dec 30, 2012
                  Ron,

                  On 12/30/12 6:10 AM, Ron Jeffries wrote:
                  > Hi Jeff,
                  >
                  > Good stuff.
                  > On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:
                  >
                  >> An important paper by Microsoft research was referenced indirectly in on e of the current threads.
                  >>
                  >> It is "Realizing quality improvement through test driven
                  >> development: results and experiences of four industrial
                  >> teams" by Nachiappan Nagappan, E. Michael Maximilien,
                  >> Thirumalesh Bhat, and Laurie Williams.
                  >>
                  >> reference:
                  >> http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                  >>
                  >> full paper:
                  >> http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                  >>
                  >> It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
                  >
                  >> HOWEVER, before everyone runs off and plans projects based on those
                  >> numbers, I'd like to point out that while the defect reduction
                  >> numbers were measurements, the "cost increases" were "shoot from
                  >> the hip estimates" by the bosses.
                  >
                  > What is unfortunate in these studies, and in all the studies I've
                  > seen, is that they seem to be estimating the time to produce bug-free
                  > code on the TDD side, and buggy code on the non-TDD side, and do not
                  > add in the time to fix the bugs that need to be fixed. So not only
                  > are those numbers pulled from the hip or lower, they're probably not
                  > comparable even if they were accurate.

                  Yes, I can write code really fast if it doesn't have to work.

                  - George

                  --
                  ----------------------------------------------------------------------
                  * George Dinwiddie * http://blog.gdinwiddie.com
                  Software Development http://www.idiacomputing.com
                  Consultant and Coach http://www.agilemaryland.org
                  ----------------------------------------------------------------------
                • pooya shahbazian
                  hi thanks, it s so useful  Pooya Shahbazian http://www.pooyablog.blogfa.com/ ________________________________ From: George Dinwiddie
                  Message 8 of 10 , Dec 30, 2012
                    hi
                    thanks, it's so useful 

                    Pooya Shahbazian
                    http://www.pooyablog.blogfa.com/



                    ________________________________
                    From: George Dinwiddie <lists@...>
                    To: extremeprogramming@yahoogroups.com
                    Sent: Sunday, December 30, 2012 2:03 PM
                    Subject: Re: [XP] Microsoft Research on TDD


                     

                    Pooya,

                    On 12/29/12 11:48 PM, pooya shahbazian wrote:
                    > hi jeff
                    > thanks for introducing the paper.

                    If you're interested in such papers, you might find
                    http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
                    use. There are a few other papers along this line.

                    - George

                    --
                    ----------------------------------------------------------
                    * George Dinwiddie * http://blog.gdinwiddie.com
                    Software Development http://www.idiacomputing.com
                    Consultant and Coach http://www.agilemaryland.org
                    ----------------------------------------------------------




                    [Non-text portions of this message have been removed]
                  • JeffGrigg
                    ... Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the
                    Message 9 of 10 , Dec 30, 2012
                      --- "M. Manca" <m.manca@...> wrote:
                      > Hi Jeff,
                      > I have similar results about quality but about cost I
                      > have a reduction between 25% to 45% compared with
                      > similar embedded projects.

                      Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the decreased maintenance costs of having fewer bugs, and more readable and maintainable code, and in-place test suites to prevent regression, are often overlooked.

                      Even during the initial development phase, I've seen numerous benefits to the TDD approach. A friend of mine in the early 2000's studied a number of projects done in the same company with similar teams, similar management, the same tools, etc. -- giving him a good comparison of XP to their more conventional approaches. His numbers showed that not using XP cost the company two to four times as much -- just during the initial development project.
                    Your message has been successfully submitted and would be delivered to recipients shortly.