Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Microsoft Research on TDD

Expand Messages
  • Ron Jeffries
    Hi Jeff, Good stuff. ... What is unfortunate in these studies, and in all the studies I ve seen, is that they seem to be estimating the time to produce
    Message 1 of 10 , Dec 30, 2012
    • 0 Attachment
      Hi Jeff,

      Good stuff.
      On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:

      > An important paper by Microsoft research was referenced indirectly in on e of the current threads.
      >
      > It is "Realizing quality improvement through test driven
      > development: results and experiences of four industrial
      > teams" by Nachiappan Nagappan, E. Michael Maximilien,
      > Thirumalesh Bhat, and Laurie Williams.
      >
      > reference:
      > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
      >
      > full paper:
      > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
      >
      > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.

      > HOWEVER, before everyone runs off and plans projects based on those numbers, I'd like to point out that while the defect reduction numbers were measurements, the "cost increases" were "shoot from the hip estimates" by the bosses.

      What is unfortunate in these studies, and in all the studies I've seen, is that they seem to be estimating the time to produce bug-free code on the TDD side, and buggy code on the non-TDD side, and do not add in the time to fix the bugs that need to be fixed. So not only are those numbers pulled from the hip or lower, they're probably not comparable even if they were accurate.

      But still good stuff!

      Ron Jeffries
      www.XProgramming.com
      Wisdom begins when we learn the difference between "that makes no sense" and "I don't understand". -- Mary Doria Russell



      [Non-text portions of this message have been removed]
    • M. Manca
      Il 30/12/2012 02:31, JeffGrigg ha scritto: Hi Jeff, I have similar results about quality but about cost I have a reduction between 25% to 45% compared with
      Message 2 of 10 , Dec 30, 2012
      • 0 Attachment
        Il 30/12/2012 02:31, JeffGrigg ha scritto:
        Hi Jeff,
        I have similar results about quality but about cost I have a reduction
        between 25% to 45% compared with similar embedded projects.
        >
        >
        > An important paper by Microsoft research was referenced indirectly in
        > on e of the current threads.
        >
        > It is "Realizing quality improvement through test driven
        > development: results and experiences of four industrial
        > teams" by Nachiappan Nagappan, E. Michael Maximilien,
        > Thirumalesh Bhat, and Laurie Williams.
        >
        > reference:
        > http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
        >
        > full paper:
        > http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
        >
        > It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
        >
        > HOWEVER, before everyone runs off and plans projects based on those
        > numbers, I'd like to point out that while the defect reduction numbers
        > were measurements, the "cost increases" were "shoot from the hip
        > estimates" by the bosses.
        >
        > The paper says...
        > "Another interesting observation from the outcome measures in Table 3
        > is the increase in time to develop the features attributed to the
        > usage of the TDD practice, as subjectively estimated by management.
        > The increase in development time ranges from 15% to 35%. From an
        > efficacy perspective this increase in development time is offset by
        > the by the reduced maintenance costs due to the improvement in quality
        > (Erdogmus and Williams 2003), an observation that was backed up the
        > product teams at Microsoft and IBM."
        >
        > I'm emphasizing the "as subjectively estimated by management" part.
        >
        >



        [Non-text portions of this message have been removed]
      • George Dinwiddie
        Ron, ... Yes, I can write code really fast if it doesn t have to work. - George -- ... * George Dinwiddie * http://blog.gdinwiddie.com
        Message 3 of 10 , Dec 30, 2012
        • 0 Attachment
          Ron,

          On 12/30/12 6:10 AM, Ron Jeffries wrote:
          > Hi Jeff,
          >
          > Good stuff.
          > On Dec 29, 2012, at 8:31 PM, JeffGrigg <jeffreytoddgrigg@...> wrote:
          >
          >> An important paper by Microsoft research was referenced indirectly in on e of the current threads.
          >>
          >> It is "Realizing quality improvement through test driven
          >> development: results and experiences of four industrial
          >> teams" by Nachiappan Nagappan, E. Michael Maximilien,
          >> Thirumalesh Bhat, and Laurie Williams.
          >>
          >> reference:
          >> http://link.springer.com/article/10.1007%2Fs10664-008-9062-z
          >>
          >> full paper:
          >> http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf
          >>
          >> It reports 40% to 90% defect reduction and 15% to 35% increase in cost.
          >
          >> HOWEVER, before everyone runs off and plans projects based on those
          >> numbers, I'd like to point out that while the defect reduction
          >> numbers were measurements, the "cost increases" were "shoot from
          >> the hip estimates" by the bosses.
          >
          > What is unfortunate in these studies, and in all the studies I've
          > seen, is that they seem to be estimating the time to produce bug-free
          > code on the TDD side, and buggy code on the non-TDD side, and do not
          > add in the time to fix the bugs that need to be fixed. So not only
          > are those numbers pulled from the hip or lower, they're probably not
          > comparable even if they were accurate.

          Yes, I can write code really fast if it doesn't have to work.

          - George

          --
          ----------------------------------------------------------------------
          * George Dinwiddie * http://blog.gdinwiddie.com
          Software Development http://www.idiacomputing.com
          Consultant and Coach http://www.agilemaryland.org
          ----------------------------------------------------------------------
        • pooya shahbazian
          hi thanks, it s so useful  Pooya Shahbazian http://www.pooyablog.blogfa.com/ ________________________________ From: George Dinwiddie
          Message 4 of 10 , Dec 30, 2012
          • 0 Attachment
            hi
            thanks, it's so useful 

            Pooya Shahbazian
            http://www.pooyablog.blogfa.com/



            ________________________________
            From: George Dinwiddie <lists@...>
            To: extremeprogramming@yahoogroups.com
            Sent: Sunday, December 30, 2012 2:03 PM
            Subject: Re: [XP] Microsoft Research on TDD


             

            Pooya,

            On 12/29/12 11:48 PM, pooya shahbazian wrote:
            > hi jeff
            > thanks for introducing the paper.

            If you're interested in such papers, you might find
            http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment of
            use. There are a few other papers along this line.

            - George

            --
            ----------------------------------------------------------
            * George Dinwiddie * http://blog.gdinwiddie.com
            Software Development http://www.idiacomputing.com
            Consultant and Coach http://www.agilemaryland.org
            ----------------------------------------------------------




            [Non-text portions of this message have been removed]
          • JeffGrigg
            ... Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the
            Message 5 of 10 , Dec 30, 2012
            • 0 Attachment
              --- "M. Manca" <m.manca@...> wrote:
              > Hi Jeff,
              > I have similar results about quality but about cost I
              > have a reduction between 25% to 45% compared with
              > similar embedded projects.

              Yes; some studies seem to ignore the fact that companies typically spend 90% to 99% or more of their software development budgets on maintenance. So the decreased maintenance costs of having fewer bugs, and more readable and maintainable code, and in-place test suites to prevent regression, are often overlooked.

              Even during the initial development phase, I've seen numerous benefits to the TDD approach. A friend of mine in the early 2000's studied a number of projects done in the same company with similar teams, similar management, the same tools, etc. -- giving him a good comparison of XP to their more conventional approaches. His numbers showed that not using XP cost the company two to four times as much -- just during the initial development project.
            Your message has been successfully submitted and would be delivered to recipients shortly.