Loading ...
Sorry, an error occurred while loading the content.

Failed and Challenged Projects (Re: Agile Method Research Study)

Expand Messages
  • woynam
    Right. It s the over budget, and over time criteria that gets me steaming. The last large project I worked on was over by 250% on time and budget, but the
    Message 1 of 41 , Jul 23, 2013
    View Source
    • 0 Attachment
      Right. It's the over budget, and over time criteria that gets me steaming.

      The last large project I worked on was over by 250% on time and budget, but the project was considered a success because it finally got us off our legacy mainframe, and allowed us to business functionality in the future that couldn't be done on the old system.

      The team that created the initial estimates and plan consisted of over a dozen senior engineers that collectively had over 150 years experience with the company. We were locked in a conference room for close to two weeks breaking down the functions into epics/stories, and using the three-point technique to estimate them. IMHO, based on the understanding of the requirements at that time, the estimates were very good.

      Of course, as this huge project progressed, we discovered a lot of "hidden" functionality, and the backlog continued to grow. It was ended up 250% larger than the original backlog.

      Now, given that we didn't add any new functionality, and were merely porting (and re-architecting) the existing functionality deemed essential by the business, I would say that the final cost and schedule was correct for the eventual functionality. It just took us 2 1/2 years to discover the truth.

      Mark


      --- In scrumdevelopment@yahoogroups.com, Diana Young <diana.young@...> wrote:
      >
      > Hi, Mark. I believe the Standish Reports actually apply three criteria to label a project as a challenge or a failure: over budget, over time, and not satisfying users expectations. Intuitively, it would seem that agile would help teams better meet all three of those criteria. However, data is needed to test that intuition.
      >
      > In a related vein, a colleague of mine is conducting research to determine if there differences in stakeholder groups' definitions and criteria concerning software development project success. If we are using metrics that mean different things to different people, no knowledge is gained.
      >
      > Your idea for a research study concerning changes in project estimates based upon knowledge learned through the process is really interesting and could produce some very valuable results. However, getting real companies to participate is going to be very challenging.
      >
      > Diana
      > From: scrumdevelopment@yahoogroups.com [mailto:scrumdevelopment@yahoogroups.com] On Behalf Of woynam
      > Sent: Monday, July 22, 2013 11:46 AM
      > To: scrumdevelopment@yahoogroups.com
      > Subject: [scrumdevelopment] Failed and Challenged Projects (Re: Agile Method Research Study)
      >
      >
      >
      > Sorry, but I'm not buying the plug. If it's wrong, please tell us why.
      >
      > I agree that it's probably not "scientific". As we've been discussing, getting real numbers is tough in the SW field.
      >
      > Based on my experience, I'd say the cone is very close to correct, given a fixed-sized starting backlog, which is almost a certainty in a traditional contract-upfront project.
      >
      > My most recent "large" project, a legacy mainframe migration project, was 2.5 years long, and the final costs were 2.5 times higher than our initial estimates. Of course, as we peeled away the layers of the legacy system, there was more junk in there than even the biggest pessimists imagined. You can see our burn-up chart in the 'Files' section of this group (Burnup Chart Example.jpg).
      >
      > Mark
      >
      > --- In scrumdevelopment@yahoogroups.com<mailto:scrumdevelopment%40yahoogroups.com>, Yves Hanoulle <mailing@<mailto:mailing@>> wrote:
      > >
      > > 2013/7/22 woynam <woyna@<mailto:woyna@>>
      > >
      > > > **
      > > >
      > > >
      > > >
      > > > The figures from Standish need to be taken with a *huge* piece of salt.
      > > >
      > > > A project is considered a "failure" or "challenged" based on its ability
      > > > to come it at, or under budget. We all know in the agile community that the
      > > > initial budget estimate is the *worst* possible estimate, given that its
      > > > derived with the *least* amount of information.
      > > >
      > > I assume that statement is based on the cone of uncertainty.
      > > I encourage you to read Laurent Bossavit's book
      > > https://leanpub.com/leprechauns
      > > You will learn that the cone is not scientific at all (yes I agree it feels
      > > right, well it's not correct..) I won't disclose at what level it is wrong,
      > > let me just say it feels counter intuïtive. (mm isn't agile about doing
      > > some counter intuïtive things ;-) )
      > >
      > >
      > >
      > > > Lately, I've made sure that I refer to projects as being "under budgeted",
      > > > rather than "over budget".
      > > >
      > > > I'd like to see a report that critically reviews projects to determine if
      > > > the actual money spent was inline with the knowledge gained during the
      > > > project. In other words, if you discover something on day 100 that you
      > > > didn't know on day 1, would you have changed your estimate on day 1 if you
      > > > knew what you didn't know. I'm guessing these percentages would flip-flop.
      > > >
      > > > Mark
      > > >
      > > > --- In scrumdevelopment@yahoogroups.com<mailto:scrumdevelopment%40yahoogroups.com>, Diana Young <diana.young@>
      > > > wrote:
      > > > >
      > > > > That being said, the reality is that globally a lot of money is spent
      > > > each year on software development projects and the results are less than
      > > > stellar. The Standish Reports estimate that approximately 25% of all
      > > > software development project are considered failures, about 30% are
      > > > considered successful, and the remainder are challenged in some way
      > > >
      > > >
      > > >
      > >
      >
    • Cass Dalton
      That s exactly the point. Here, we are defining the end as the ACTUAL end when you stop charging your time. When you think you re going to be done in 2 weeks
      Message 41 of 41 , Jul 29, 2013
      View Source
      • 0 Attachment
        That's exactly the point.  Here, we are defining the end as the ACTUAL end when you stop charging your time.  When you think you're going to be done in 2 weeks (the estimate) but you don't finish for 2 months (the actual end), you obviously have some amount of uncertainty in your estimate even if you don't realize it.  That uncertainty is supposed to be reflected in the cone, but in practice, that cone doesn't taper; it is hangs around your estimates like a thundercloud until you stop charging your time to it.


        On Tue, Jul 23, 2013 at 10:58 AM, Yves Hanoulle <mailing@...> wrote:
         

        2013/7/22 Cass Dalton <cassdalton73@...>

         

        The high level concept that the cone portrays (estimation uncertainty is IN GENERAL higher the the farther away you are from the end) is true.  
         
        well  you have to keep in mind that there is a BIG difference between being close to the end and thinking you are close to the end.


         
        However, the shape of the cone is based completely on someone's subjective theory, not on objective, empirical data.  That is the only real point that Laurent is trying to make.  He backs the argument up with intuition that 1) estimates in software development usually tend toward UNDER estimation, not OVER estimation, so the cone is not symmetrical as the original plot suggests, and that 2) the smooth tapering in the curve often doesn't happen as the last 90% of the work takes the last 40-50% of the time.

        Based on my experience in a traditional environment, I would say that the cone is rarely correct as presented in the plot.  Estimates are low at least 85% of the time, and the uncertainty often doesn't taper anything like how the plot suggests.  The times when estimates are high come from people who have been bitten by the always low estimates enough that the add in so much padding that their estimates are always unrealistically high.  And then you have the rule that the work will tend to fill the estimate, completely skewing any empirical evidence you think you have.  (The empirical evidence or lack thereof being the entire crux of Laurent's argument).


        On Mon, Jul 22, 2013 at 1:20 PM, George Dinwiddie <lists@...> wrote:
         

        Mark,



        On 7/22/13 12:46 PM, woynam wrote:
        >
        > Sorry, but I'm not buying the plug. If it's wrong, please tell us why.

        You can read some of what Laurent says about it at
        https://plus.google.com/115091715679003832601/posts/FKLauKLZECm


        > I agree that it's probably not "scientific". As we've been
        > discussing, getting real numbers is tough in the SW field.
        >
        > Based on my experience, I'd say the cone is very close to correct,
        > given a fixed-sized starting backlog, which is almost a certainty in
        > a traditional contract-upfront project.

        Laurent questions that
        - the cone is presented a symetrical, with as much room for
        underestimating as overestimating, even though it's impossible to
        complete a project in negative time
        - that the cone seems to say that we necessarily get tighter estimates
        when we approach the end, though in reality some projects stay at "90%
        done" for a long time
        - that the cone is taken for empirical data, but is based on Boehm's
        subjective opinion

        - George


        > My most recent "large" project, a legacy mainframe migration project,
        > was 2.5 years long, and the final costs were 2.5 times higher than
        > our initial estimates. Of course, as we peeled away the layers of the
        > legacy system, there was more junk in there than even the biggest
        > pessimists imagined. You can see our burn-up chart in the 'Files'
        > section of this group (Burnup Chart Example.jpg).
        >
        > Mark
        >
        >
        > --- In scrumdevelopment@yahoogroups.com, Yves Hanoulle <mailing@...> wrote:
        >>
        >> 2013/7/22 woynam <woyna@...>
        >>
        >>> **
        >>>
        >>>
        >>>
        >>> The figures from Standish need to be taken with a *huge* piece of salt.
        >>>
        >>> A project is considered a "failure" or "challenged" based on its ability
        >>> to come it at, or under budget. We all know in the agile community that the
        >>> initial budget estimate is the *worst* possible estimate, given that its
        >>> derived with the *least* amount of information.
        >>>
        >> I assume that statement is based on the cone of uncertainty.
        >> I encourage you to read Laurent Bossavit's book
        >> https://leanpub.com/leprechauns
        >> You will learn that the cone is not scientific at all (yes I agree it feels
        >> right, well it's not correct..) I won't disclose at what level it is wrong,
        >> let me just say it feels counter intuïtive. (mm isn't agile about doing
        >> some counter intuïtive things ;-) )
        >>
        >>
        >>
        >>> Lately, I've made sure that I refer to projects as being "under budgeted",
        >>> rather than "over budget".
        >>>
        >>> I'd like to see a report that critically reviews projects to determine if
        >>> the actual money spent was inline with the knowledge gained during the
        >>> project. In other words, if you discover something on day 100 that you
        >>> didn't know on day 1, would you have changed your estimate on day 1 if you
        >>> knew what you didn't know. I'm guessing these percentages would flip-flop.
        >>>
        >>> Mark
        >>>
        >>> --- In scrumdevelopment@yahoogroups.com, Diana Young <diana.young@>
        >>> wrote:
        >>>>
        >>>> That being said, the reality is that globally a lot of money is spent
        >>> each year on software development projects and the results are less than
        >>> stellar. The Standish Reports estimate that approximately 25% of all
        >>> software development project are considered failures, about 30% are
        >>> considered successful, and the remainder are challenged in some way

        --
        Want to speak at AgileDC October 8, 2013? http://agiledc.org/speak/
        ----------------------------------------------------------
        * George Dinwiddie * http://blog.gdinwiddie.com
        Software Development http://www.idiacomputing.com
        Consultant and Coach http://www.agilemaryland.org
        ----------------------------------------------------------





      Your message has been successfully submitted and would be delivered to recipients shortly.