Loading ...
Sorry, an error occurred while loading the content.

Microsoft Research on TDD

Expand Messages
  • JeffGrigg
    An important paper by Microsoft research was referenced indirectly in on e of the current threads. It is Realizing quality improvement through test driven
    Message 1 of 10 , Dec 29, 2012
    • 0 Attachment
      An important paper by Microsoft research was referenced indirectly in on e of the current threads.

      It is "Realizing quality improvement through test driven
      development: results and experiences of four industrial
      teams" by Nachiappan Nagappan, E. Michael Maximilien,
      Thirumalesh Bhat, and Laurie Williams.

      reference:
      http://link.springer.com/article/10.1007%2Fs10664-008-9062-z

      full paper:
      http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf

      It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

      HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

      The paper says...
      "Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM."

      I'm emphasizing the "as subjectively estimated by management" part.
    • Dave Smith
      Even if they were shooting from the hip, it still reads better to turn their numbers around. By using normal development practices instead of TDD, teams can
      Message 2 of 10 , Dec 29, 2012
      • 0 Attachment
        Even if they were shooting from the hip, it still reads better to turn
        their numbers around.

        By using normal development practices instead of TDD, teams can complete
        projects 14 to 26 percent faster at the expense of defect densities that
        are 150 to 900 percent higher.

        Dave

        On Sat, Dec 29, 2012 at 5:31 PM, JeffGrigg <jeffreytoddgrigg@...>wrote:

        > An important paper by Microsoft research was referenced indirectly in on e
        > of the current threads.
        >
        > It is "Realizing quality improvement through test driven
        > development: results and experiences of four industrial
        > teams" by Nachiappan Nagappan, E. Michael Maximilien,
        > Thirumalesh Bhat, and Laurie Williams.
        >
        > reference:
        > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
        >
        > full paper:
        > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
        >
        > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
        >
        > HOWEVER, before everyone runs off and plans projects based on those
        > numbers, I'd like to point out that while the defect reduction numbers were
        > measurements, the "cost increases" were "shoot from the hip estimates" by
        > the bosses.
        >
        > The paper says...
        > "Another interesting observation from the outcome measures in Table 3 is
        > the increase in time to develop the features attributed to the usage of the
        > TDD practice, as subjectively estimated by management. The increase in
        > development time ranges from 15% to 35%. From an efficacy perspective this
        > increase in development time is offset by the by the reduced maintenance
        > costs due to the improvement in quality (Erdogmus and Williams 2003), an
        > observation that was backed up the product teams at Microsoft and IBM."
        >
        > I'm emphasizing the "as subjectively estimated by management" part.
        >
        >
        >
        > ------------------------------------
        >
        > To Post a message, send it to: extremeprogramming@...
        >
        > To Unsubscribe, send a blank message to:
        > extremeprogramming-unsubscribe@...
        >
        > ad-free courtesy of objectmentor.comYahoo! Groups Links
        >
        >
        >
        >


        [Non-text portions of this message have been removed]
      • pooya shahbazian
        hi jeff thanks for introducing the paper.   Pooya Shahbazian http://www.pooyablog.blogfa.com ________________________________ From: JeffGrigg
        Message 3 of 10 , Dec 29, 2012
        • 0 Attachment
          hi jeff
          thanks for introducing the paper.

           
          Pooya Shahbazian
          http://www.pooyablog.blogfa.com



          ________________________________
          From: JeffGrigg <jeffreytoddgrigg@...>
          To: extremeprogramming@yahoogroups.com
          Sent: Sunday, December 30, 2012 5:01 AM
          Subject: [XP] Microsoft Research on TDD


           
          An important paper by Microsoft research was referenced indirectly in on e of the current threads.

          It is "Realizing quality improvement through test driven
          development: results and experiences of four industrial
          teams" by Nachiappan Nagappan, E. Michael Maximilien,
          Thirumalesh Bhat, and Laurie Williams.

          reference:
          http://link.springer.com/article/10.1007%2Fs10664-008-9062-z

          full paper:
          http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf

          It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

          HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

          The paper says...
          "Another interesting observation from the outcome measures in Table 3 is the increase in time to develop the features attributed to the usage of the TDD practice, as subjectively estimated by management. The increase in development time ranges from 15% to 35%. From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM."

          I'm emphasizing the "as subjectively estimated by management" part.




          [Non-text portions of this message have been removed]
        • Adam Sroka
          Nice! And closely matches my own observations in the field. ... [Non-text portions of this message have been removed]
          Message 4 of 10 , Dec 29, 2012
          • 0 Attachment
            Nice! And closely matches my own observations in the field.


            On Sat, Dec 29, 2012 at 11:18 PM, Dave Smith <davewsmith@...> wrote:

            > **
            >
            >
            > Even if they were shooting from the hip, it still reads better to turn
            > their numbers around.
            >
            > By using normal development practices instead of TDD, teams can complete
            > projects 14 to 26 percent faster at the expense of defect densities that
            > are 150 to 900 percent higher.
            >
            > Dave
            >
            > On Sat, Dec 29, 2012 at 5:31 PM, JeffGrigg <jeffreytoddgrigg@...
            > >wrote:
            >
            >
            > > An important paper by Microsoft research was referenced indirectly in on
            > e
            > > of the current threads.
            > >
            > > It is "Realizing quality improvement through test driven
            > > development: results and experiences of four industrial
            > > teams" by Nachiappan Nagappan, E. Michael Maximilien,
            > > Thirumalesh Bhat, and Laurie Williams.
            > >
            > > reference:
            > > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
            > >
            > > full paper:
            > > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
            > >
            > > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
            > >
            > > HOWEVER, before everyone runs off and plans projects based on those
            > > numbers, I'd like to point out that while the defect reduction numbers
            > were
            > > measurements, the "cost increases" were "shoot from the hip estimates" by
            > > the bosses.
            > >
            > > The paper says...
            > > "Another interesting observation from the outcome measures in Table 3 is
            > > the increase in time to develop the features attributed to the usage of
            > the
            > > TDD practice, as subjectively estimated by management. The increase in
            > > development time ranges from 15% to 35%. From an efficacy perspective
            > this
            > > increase in development time is offset by the by the reduced maintenance
            > > costs due to the improvement in quality (Erdogmus and Williams 2003), an
            > > observation that was backed up the product teams at Microsoft and IBM."
            > >
            > > I'm emphasizing the "as subjectively estimated by management" part.
            > >
            > >
            > >
            > > ------------------------------------
            >
            > >
            > > To Post a message, send it to: extremeprogramming@...
            > >
            > > To Unsubscribe, send a blank message to:
            > > extremeprogramming-unsubscribe@...
            > >
            > > ad-free courtesy of objectmentor.comYahoo! Groups Links
            > >
            > >
            > >
            > >
            >
            > [Non-text portions of this message have been removed]
            >
            >
            >


            [Non-text portions of this message have been removed]
          • George Dinwiddie
            Pooya, ... If you re interested in such papers, you might find http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of use. There are a few other
            Message 5 of 10 , Dec 30, 2012
            • 0 Attachment
              Pooya,

              On 12/29/12 11:48 PM, pooya shahbazian wrote:
              > hi jeff
              > thanks for introducing the paper.

              If you're interested in such papers, you might find
              http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
              use. There are a few other papers along this line.

              - George

              --
              ----------------------------------------------------------------------
              * George Dinwiddie * http://blog.gdinwiddie.com
              Software Development http://www.idiacomputing.com
              Consultant and Coach http://www.agilemaryland.org
              ----------------------------------------------------------------------
            • Ron Jeffries
              Hi Jeff, Good stuff. ... What is unfortunate in these studies, and in all the studies I ve seen, is that they seem to be estimating the time to produce
              Message 6 of 10 , Dec 30, 2012
              • 0 Attachment
                Hi Jeff,

                Good stuff.
                On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:

                > An important paper by Microsoft research was referenced indirectly in on e of the current threads.
                >
                > It is "Realizing quality improvement through test driven
                > development: results and experiences of four industrial
                > teams" by Nachiappan Nagappan, E. Michael Maximilien,
                > Thirumalesh Bhat, and Laurie Williams.
                >
                > reference:
                > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                >
                > full paper:
                > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                >
                > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

                > HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

                What is unfortunate in these studies, and in all the studies I've seen, is that they seem to be estimating the time to produce bug-free code on the TDD side, and buggy code on the non-TDD side, and do not add in the time to fix the bugs that need to be fixed. So not only are those numbers pulled from the hip or lower, they're probably not comparable even if they were accurate.

                But still good stuff!

                Ron Jeffries
                www.XProgramming.com
                Wisdom begins when we learn the difference between "that makes no sense" and "I don't understand". -- Mary Doria Russell



                [Non-text portions of this message have been removed]
              • M. Manca
                Il 30/12/2012 02:31, JeffGrigg ha scritto: Hi Jeff, I have similar results about quality but about cost I have a reduction between 25% to 45% compared with
                Message 7 of 10 , Dec 30, 2012
                • 0 Attachment
                  Il 30/12/2012 02:31, JeffGrigg ha scritto:
                  Hi Jeff,
                  I have similar results about quality but about cost I have a reduction
                  between 25% to 45% compared with similar embedded projects.
                  >
                  >
                  > An important paper by Microsoft research was referenced indirectly in
                  > on e of the current threads.
                  >
                  > It is "Realizing quality improvement through test driven
                  > development: results and experiences of four industrial
                  > teams" by Nachiappan Nagappan, E. Michael Maximilien,
                  > Thirumalesh Bhat, and Laurie Williams.
                  >
                  > reference:
                  > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                  >
                  > full paper:
                  > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                  >
                  > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
                  >
                  > HOWEVER, before everyone runs off and plans projects based on those
                  > numbers, I'd like to point out that while the defect reduction numbers
                  > were measurements, the "cost increases" were "shoot from the hip
                  > estimates" by the bosses.
                  >
                  > The paper says...
                  > "Another interesting observation from the outcome measures in Table 3
                  > is the increase in time to develop the features attributed to the
                  > usage of the TDD practice, as subjectively estimated by management.
                  > The increase in development time ranges from 15% to 35%. From an
                  > efficacy perspective this increase in development time is offset by
                  > the by the reduced maintenance costs due to the improvement in quality
                  > (Erdogmus and Williams 2003), an observation that was backed up the
                  > product teams at Microsoft and IBM."
                  >
                  > I'm emphasizing the "as subjectively estimated by management" part.
                  >
                  >



                  [Non-text portions of this message have been removed]
                • George Dinwiddie
                  Ron, ... Yes, I can write code really fast if it doesn t have to work. - George -- ... * George Dinwiddie * http://blog.gdinwiddie.com
                  Message 8 of 10 , Dec 30, 2012
                  • 0 Attachment
                    Ron,

                    On 12/30/12 6:10 AM, Ron Jeffries wrote:
                    > Hi Jeff,
                    >
                    > Good stuff.
                    > On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:
                    >
                    >> An important paper by Microsoft research was referenced indirectly in on e of the current threads.
                    >>
                    >> It is "Realizing quality improvement through test driven
                    >> development: results and experiences of four industrial
                    >> teams" by Nachiappan Nagappan, E. Michael Maximilien,
                    >> Thirumalesh Bhat, and Laurie Williams.
                    >>
                    >> reference:
                    >> http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
                    >>
                    >> full paper:
                    >> http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
                    >>
                    >> It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
                    >
                    >> HOWEVER, before everyone runs off and plans projects based on those
                    >> numbers, I'd like to point out that while the defect reduction
                    >> numbers were measurements, the "cost increases" were "shoot from
                    >> the hip estimates" by the bosses.
                    >
                    > What is unfortunate in these studies, and in all the studies I've
                    > seen, is that they seem to be estimating the time to produce bug-free
                    > code on the TDD side, and buggy code on the non-TDD side, and do not
                    > add in the time to fix the bugs that need to be fixed. So not only
                    > are those numbers pulled from the hip or lower, they're probably not
                    > comparable even if they were accurate.

                    Yes, I can write code really fast if it doesn't have to work.

                    - George

                    --
                    ----------------------------------------------------------------------
                    * George Dinwiddie * http://blog.gdinwiddie.com
                    Software Development http://www.idiacomputing.com
                    Consultant and Coach http://www.agilemaryland.org
                    ----------------------------------------------------------------------
                  • pooya shahbazian
                    hi thanks, it s so useful  Pooya Shahbazian http://www.pooyablog.blogfa.com/ ________________________________ From: George Dinwiddie
                    Message 9 of 10 , Dec 30, 2012
                    • 0 Attachment
                      hi
                      thanks, it's so useful 

                      Pooya Shahbazian
                      http://www.pooyablog.blogfa.com/



                      ________________________________
                      From: George Dinwiddie <lists@...>
                      To: extremeprogramming@yahoogroups.com
                      Sent: Sunday, December 30, 2012 2:03 PM
                      Subject: Re: [XP] Microsoft Research on TDD


                       

                      Pooya,

                      On 12/29/12 11:48 PM, pooya shahbazian wrote:
                      > hi jeff
                      > thanks for introducing the paper.

                      If you're interested in such papers, you might find
                      http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
                      use. There are a few other papers along this line.

                      - George

                      --
                      ----------------------------------------------------------
                      * George Dinwiddie * http://blog.gdinwiddie.com
                      Software Development http://www.idiacomputing.com
                      Consultant and Coach http://www.agilemaryland.org
                      ----------------------------------------------------------




                      [Non-text portions of this message have been removed]
                    • JeffGrigg
                      ... Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the
                      Message 10 of 10 , Dec 30, 2012
                      • 0 Attachment
                        --- "M. Manca" <m.manca@...> wrote:
                        > Hi Jeff,
                        > I have similar results about quality but about cost I
                        > have a reduction between 25% to 45% compared with
                        > similar embedded projects.

                        Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the decreased maintenance costs of having fewer bugs, and more readable and maintainable code, and in-place test suites to prevent regression, are often overlooked.

                        Even during the initial development phase, I've seen numerous benefits to the TDD approach. A friend of mine in the early 2000's studied a number of projects done in the same company with similar teams, similar management, the same tools, etc. -- giving him a good comparison of XP to their more conventional approaches. His numbers showed that not using XP cost the company two to four times as much -- just during the initial development project.
                      Your message has been successfully submitted and would be delivered to recipients shortly.