Loading ...
Sorry, an error occurred while loading the content.

RE: [scrumdevelopment] What's wrong with tracking estimates vs. actuals?

Expand Messages
  • David J. Anderson
    Marc, I think there is a danger of doing agile for the sake of it. The real goal is to make more money by delivering more value faster to the end of value
    Message 1 of 20 , Oct 6, 2003
    • 0 Attachment
      Marc,

      I think there is a danger of "doing agile" for the
      sake of it. The real goal is to make more money by
      delivering more value faster to the end of value
      chain. If agilists lose sight of this then agile will
      be a fad rather than a genuine trend.

      I would contest your observation that making the best
      ROI or the most money is always best done using the
      very pure agile techniques you describe.

      For example, in the Boehm and Turner book they
      describe a 3 year long XP project which using more
      than 30 developers in a 50 person team produced around
      500,000 SLOC. Compare this to the CLS project at UOB
      in Singapore (the original FDD project) which was
      performed using the fine-grained planned versus actual
      which with slightly less people (47) but with only 23
      developers produced 1,500,000 SLOC in only 18 months.

      If SLOC is a reliable metric (and I don't believe it
      is but given a lack of a function point assessment of
      both projects it is all we have) then the first FDD
      project seems to be up to 6 times more effective than
      the textbook XP project reported by Boehm and Turner.

      Not only was the CLS project a huge success but after
      it was rolled live - the bank (UOB) was able to take
      over its nearest competitor. This was achieved through
      improved competitiveness and the lending system
      delivered by the CLS project played a key part in
      providing that competitiveness.

      One of the big problems with agile successes is that
      they get reported in a relative manner e.g. we
      implemented (pick a method you like e.g. Scrum) and we
      produced a four fold improvement. There are lots of
      such anecdotes. Ken even includes one in the Scrum
      book. However, none of these results are reported as
      absolute figures in a normalized manner. The success
      has a lot to do with the starting position. It's
      always easy to improve a very poor organization.

      So I fail to accept that you can assert that adhoc,
      self-organizing processes can outperform lean
      processes which still include aspects such as planned
      versus actual dates. When you can show me the metrics
      from real projects reported in a normalized fashion,
      from projects performed in large businesses with large
      value-added or revenue generation potential then I
      might start to believe.

      Cordially,
      David
      --
      David J. Anderson
      author of "Agile Management for Software Engineering"
      http://www.agilemanagement.net/

      --- Marc Hamann <marc@...> wrote:

      David,

      My claim is not that your approach doesn't work; there
      a innumerable
      systems that have been built using BDUF or even ad hoc
      methodologies.
      I'm sure that many of these conform to various sorts
      of acceptable metrics.

      However, such practices are not Agile in general or
      Scrum in
      particular. Moreover, they work against the goals and
      values that Agile
      and Scrum promote, which is counter-productive if you
      are trying to combine
      them with Agile practices.

      My implication that the team should be fired was
      provocative, but "that
      won't work here because of our culture/our
      programmers/our boss" is the
      most common objection I hear to using Agile
      techniques. This seems
      counter-intuitive to me, since I believe that the
      great virtue of Agile
      techniques is that they can be used even in adverse
      circumstances.

      You don't have to change the culture to start using
      the techniques; just do
      your best to implement the practices, and avoid any
      practices which work
      against the goals and values you are trying to
      promote.

      It's not magic, it's not easy, but if you don't at
      least try the best you
      can, you can't claim to be doing Scrum or Agile.

      Regards,

      Marc



      =====
      David J Anderson
      Author of "Agile Management for Software Engineering"
      http://www.agilemanagement.net/

      __________________________________
      Do you Yahoo!?
      The New Yahoo! Shopping - with improved product search
      http://shopping.yahoo.com
    • Marc Hamann
      David, ... I agree that doing agile is not an end in and of itself. Providing return on investment and return on expectation is the real goal. We do this by
      Message 2 of 20 , Oct 6, 2003
      • 0 Attachment
        David,

        >I think there is a danger of "doing agile" for the
        >sake of it. The real goal is to make more money by
        >delivering more value faster to the end of value
        >chain.

        I agree that "doing agile" is not an end in and of itself. Providing
        return on investment and return on expectation is the real goal.

        We do this by producing working software that fulfills the needs of the
        customer. I'm quite skeptical that we can do this by producing "metrics".

        The only metric that counts is how many of the customer's requirements we
        clear out of the backlog by delivering well-made, working software for a
        given cost.
        Any other metric may or may not be a helpful guideline to help you
        accomplish that primary goal, but some metrics are blatantly
        counter-productive because they encourage the wrong behaviour.

        I'm glad you agree that SLOC is not a good metric; crappy code can eat up
        WAY more lines than good code. ;-)

        >One of the big problems with agile successes is that
        >they get reported in a relative manner

        I suppose part of our disagreement comes from our different concerns. You
        want to bring a scientific study to the successes of particular cases.
        Though I'm skeptical that you can ever factor out all the variables that go
        into a large project and isolate with absolute objectivity that one method
        is better than another, I can understand why you want to do this.

        However, I'm in the trenches, and the only reason I give a fig about the
        Agile methodologies is that they offer approaches and techniques that solve
        my real day to day challenges in building good, reliable, low cost
        software. What Chrysler did, or some offshore bank, though perhaps of
        academic interest will not sway me one way or another as to whether those
        techniques will work for me. If they work for me, I use them, if they
        don't, I won't.

        >So I fail to accept that you can assert that adhoc,
        >self-organizing processes can outperform lean
        >processes which still include aspects such as planned
        >versus actual dates.

        Well, since we are unlikely to find mutually satisfactory metrics for
        determining performance, I don't think I would make that claim. ;-)

        However, I do know that professionals who are too valuable to be fired are
        smart enough to tell when they are being micro-managed or babied, and
        neither of these brings out high morale, creativity and "can do" spirit.

        Remember that people will focus their energies on whatever you measure to
        evaluate them. If you measure their ability to estimate rather than their
        ability to produce working software, you will encourage them to pace their
        work to the plan, rather than doing the best they can always, to avoid
        taking risks that might throw their estimations off, even if it would
        improve the product.

        I can't see how that will improve productivity. In fact, I wouldn't
        believe it were so even if you showed me some metrics. ;-)

        >When you can show me the metrics
        >from real projects reported in a normalized fashion,
        >from projects performed in large businesses with large
        >value-added or revenue generation potential then I
        >might start to believe.

        When the Buddha was asked "Why should I believe what your saying?", he
        replied "Don't believe it. Try it for yourself ."

        I can't improve on that. ;-)

        Marc
      • Daniel Gackle
        Thanks to everyone who replied on estimates vs. actuals. The best thing for me was seeing the diversity of opinion on it, which helped me realize we can just
        Message 3 of 20 , Oct 10, 2003
        • 0 Attachment
          Thanks to everyone who replied on estimates vs. actuals. The best thing for
          me was seeing the diversity of opinion on it, which helped me realize we can
          just be pragmatic. Both Jeff's and Ron's reply of "why not give them what
          they want if it doesn't cost you anything" (my paraphrase) seems applicable
          to our situation. For me this is part of learning to distinguish the
          essential stuff from the secondary stuff (which can just flex).

          I remain perplexed about exactly what people think they're going to get out
          of such numbers. Maybe my problem is that word "exactly". Maybe they just
          have a fuzzy idea of what they think they'll get... something vague but
          comforting, like "more control".

          Daniel
        • David J. Anderson
          ... I remain perplexed about exactly what people think they re going to get out of such numbers. Maybe my problem is that word exactly . Maybe they just have
          Message 4 of 20 , Oct 11, 2003
          • 0 Attachment
            --- Daniel Gackle <gackle@...> wrote:

            I remain perplexed about exactly what people think
            they're going to get out
            of such numbers. Maybe my problem is that word
            "exactly". Maybe they just
            have a fuzzy idea of what they think they'll get...
            something vague but
            comforting, like "more control".

            Daniel

            -=-=-

            Daniel,

            There is a real danger that some mangers/directors
            will try to use the data for staff evaluation and use
            it to filter out targets for layoffs. I worked with a
            director at Sprint who tried to do this. I (somewhat)
            successfully turned this into athe more benign,
            monitoring of our ability to estimate. This made it
            more a metric measuring line managers like me, than a
            metric measuring my staff. I saw that as part of my
            job - taking the heat away from the workforce.

            Hence, I think that you are right to be wary. I don't
            think that anyone believes they have more control from
            metrics like these - other than they believe that they
            will get early warning of slippages.

            People I work with who believe in short-time window
            planned versus actual, i.e. only planned out up to two
            weeks ahead, believe that it provides commitment and
            focus when that can't be achieved through other means
            such as daily standups.

            I don't know anyone who genuinely believes that
            fine-grained planned versus actual dates works beyond
            that time period. Trying to make a full release plan
            in advance is just inviting Gantt Chart hell and
            providing busy work for project managers anxious to
            keep their jobs and their PMI qualifications.
            Sometimes, you should look to the personal motivation
            and the extremely local optimum for an explanation of
            why something is the way it is.

            David



            =====
            David J Anderson
            Author of "Agile Management for Software Engineering"
            http://www.agilemanagement.net/

            __________________________________
            Do you Yahoo!?
            The New Yahoo! Shopping - with improved product search
            http://shopping.yahoo.com
          • Ken Schwaber
            In some instances we re going to be stuck with tracking actuals. Mel Pullen pointed out that this may be a symptom of displaced management trying to figure out
            Message 5 of 20 , Oct 12, 2003
            • 0 Attachment
              In some instances we're going to be stuck with tracking actuals. Mel Pullen
              pointed out that this may be a symptom of displaced management trying to
              figure out how they are adding value, and relying on what they have used in
              the past to add that value. They have always tried to improve predictabity
              in the past, let's try to do it in the future.

              I've always tried to get management to do another job. I've tried to get
              them to see how well or badly a team does, what it can produce Sprint by
              Sprint. Then I ask them to actually do the job of management - figure out
              what to do based on what the team has been able to build. Should release
              dates be changed? Should the project be decommissioned? Should functionality
              be dropped from this release? Should the team's expertise be improved in
              some areas? To me, these are all reasonable areas of management expertise,
              not throwing the problem back on the team by saying, "improve your
              estimates."

              Ken

              -----Original Message-----
              From: Mike Cohn [mailto:mike@...]
              Sent: Sunday, October 05, 2003 9:57 AM
              To: scrumdevelopment@yahoogroups.com
              Subject: RE: [scrumdevelopment] What's wrong with tracking estimates vs.
              actuals?


              I'd say you shouldn't do it because it doesn't add value commensurate with
              its cost. Don't argue with your bosses that it "adds no value" because
              comparing what you originally thought a task would take with what it did
              take can help make you a better estimator.

              But, it can be time-consuming to track actuals, especially for a full team
              where some on the team are probably already decent estimators.

              Because Scrum already has solid mechanisms for monitoring whether all the
              work gets done in a sprint (high team commitment, daily burndown charts,
              daily scrum, and so on), Scrum does not have the same reliance on early and
              accurating estimating that a predictive or waterfall approach does.

              So--the cost to gather actuals is the same in Scrum or waterfall. The
              benefit in Scrum is greatly reduced.

              --Mike



              > -----Original Message-----
              > From: Daniel Gackle [mailto:gackle@...]
              > Sent: Sunday, October 05, 2003 12:35 AM
              > To: scrumdevelopment@yahoogroups.com
              > Subject: [scrumdevelopment] What's wrong with tracking estimates vs.
              > actuals?
              >
              > At my company, some managers believe in tracking estimates vs. actuals.
              > They
              > like what my team is doing with Scrum, but they'd like it better if we
              > gave
              > them a spreadsheet every month matching the estimate for each task with
              > the
              > hours actually spent on that task.
              >
              > I feel bad about this and don't want to do it. It feels contrary to the
              > spirit of self-organizing teams, like someone is looking over our
              > shoulder.
              > Yet I can't fully articulate what's wrong with it.
              >
              > Can anyone help me get clearer on this?
              >
              > - Daniel
              >
              > -----Original Message-----
              > From: "Gamble, Ken" <ken.gamble@...>
              > Subject: New Scrum Article Available
              >
              > No matter how well someone measures past estimates against actuals even a
              > small change in the estimate can have a big effect on the outcome of a
              > chaotic/complex process no matter how good the model is or the resolution
              > of
              > the measurements.
              >
              > What this means for software development is that even though we can use
              > previous estimates as part of a process model for delivering software we
              > have to keep a sharp eye (Scrum management) on the process because it can
              > wonder off course simply because of these small values, even if the
              > process
              > model is very tightly defined.
              >
              >
              >
              > To Post a message, send it to: scrumdevelopment@...
              > To Unsubscribe, send a blank message to: scrumdevelopment-
              > unsubscribe@...
              >
              > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/




              To Post a message, send it to: scrumdevelopment@...
              To Unsubscribe, send a blank message to:
              scrumdevelopment-unsubscribe@...

              Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
            • Steven Gordon
              I have worked on many projects in the past where the elimination of labor was considered a significant benefit, if not the greatest benefit of the resulting
              Message 6 of 20 , Oct 12, 2003
              • 0 Attachment
                I have worked on many projects in the past where the elimination of labor was considered a significant benefit, if not the greatest benefit of the resulting process and/or product.

                Why would the elimination of unnecessary management labor not be a benefit to a client? Maybe, the case needs to be made further up the food chain?


                -----Original Message-----
                From: Ken Schwaber [mailto:ken.schwaber@...]
                Sent: Sun 10/12/2003 4:03 AM
                To: scrumdevelopment@yahoogroups.com
                Cc:
                Subject: RE: [scrumdevelopment] What's wrong with tracking estimates vs. actuals?

                In some instances we're going to be stuck with tracking actuals. Mel Pullen
                pointed out that this may be a symptom of displaced management trying to
                figure out how they are adding value, and relying on what they have used in
                the past to add that value. They have always tried to improve predictabity
                in the past, let's try to do it in the future.

                I've always tried to get management to do another job. I've tried to get
                them to see how well or badly a team does, what it can produce Sprint by
                Sprint. Then I ask them to actually do the job of management - figure out
                what to do based on what the team has been able to build. Should release
                dates be changed? Should the project be decommissioned? Should functionality
                be dropped from this release? Should the team's expertise be improved in
                some areas? To me, these are all reasonable areas of management expertise,
                not throwing the problem back on the team by saying, "improve your
                estimates."

                Ken

                -----Original Message-----
                From: Mike Cohn [mailto:mike@...]
                Sent: Sunday, October 05, 2003 9:57 AM
                To: scrumdevelopment@yahoogroups.com
                Subject: RE: [scrumdevelopment] What's wrong with tracking estimates vs.
                actuals?


                I'd say you shouldn't do it because it doesn't add value commensurate with
                its cost. Don't argue with your bosses that it "adds no value" because
                comparing what you originally thought a task would take with what it did
                take can help make you a better estimator.

                But, it can be time-consuming to track actuals, especially for a full team
                where some on the team are probably already decent estimators.

                Because Scrum already has solid mechanisms for monitoring whether all the
                work gets done in a sprint (high team commitment, daily burndown charts,
                daily scrum, and so on), Scrum does not have the same reliance on early and
                accurating estimating that a predictive or waterfall approach does.

                So--the cost to gather actuals is the same in Scrum or waterfall. The
                benefit in Scrum is greatly reduced.

                --Mike



                > -----Original Message-----
                > From: Daniel Gackle [mailto:gackle@...]
                > Sent: Sunday, October 05, 2003 12:35 AM
                > To: scrumdevelopment@yahoogroups.com
                > Subject: [scrumdevelopment] What's wrong with tracking estimates vs.
                > actuals?
                >
                > At my company, some managers believe in tracking estimates vs. actuals.
                > They
                > like what my team is doing with Scrum, but they'd like it better if we
                > gave
                > them a spreadsheet every month matching the estimate for each task with
                > the
                > hours actually spent on that task.
                >
                > I feel bad about this and don't want to do it. It feels contrary to the
                > spirit of self-organizing teams, like someone is looking over our
                > shoulder.
                > Yet I can't fully articulate what's wrong with it.
                >
                > Can anyone help me get clearer on this?
                >
                > - Daniel
                >
                > -----Original Message-----
                > From: "Gamble, Ken" <ken.gamble@...>
                > Subject: New Scrum Article Available
                >
                > No matter how well someone measures past estimates against actuals even a
                > small change in the estimate can have a big effect on the outcome of a
                > chaotic/complex process no matter how good the model is or the resolution
                > of
                > the measurements.
                >
                > What this means for software development is that even though we can use
                > previous estimates as part of a process model for delivering software we
                > have to keep a sharp eye (Scrum management) on the process because it can
                > wonder off course simply because of these small values, even if the
                > process
                > model is very tightly defined.
                >
                >
                >
                > To Post a message, send it to: scrumdevelopment@...
                > To Unsubscribe, send a blank message to: scrumdevelopment-
                > unsubscribe@...
                >
                > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/




                To Post a message, send it to: scrumdevelopment@...
                To Unsubscribe, send a blank message to:
                scrumdevelopment-unsubscribe@...

                Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/





                Yahoo! Groups Sponsor
                ADVERTISEMENT
                Click Here! <http://rd.yahoo.com/M=244522.3707890.4968055.1261774/D=egroupweb/S=1707209021:HM/A=1595054/R=0/SIG=124ukap9t/*http://ashnin.com/clk/muryutaitakenattogyo?YH=3707890&yhad=1595054>
                <http://us.adserver.yahoo.com/l?M=244522.3707890.4968055.1261774/D=egroupmail/S=:HM/A=1595054/rand=270339561>

                To Post a message, send it to: scrumdevelopment@...
                To Unsubscribe, send a blank message to: scrumdevelopment-unsubscribe@...

                Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service <http://docs.yahoo.com/info/terms/> .
              • Edmund Schweppe
                ... What will the client do with the managers whose unnecessary management labor will be eliminated? Or, more to the point, what do the affected managers
                Message 7 of 20 , Oct 12, 2003
                • 0 Attachment
                  Steven Gordon wrote:
                  > I have worked on many projects in the past where the
                  > elimination of labor was considered a significant benefit,
                  > if not the greatest benefit of the resulting process and/or
                  > product.
                  >
                  > Why would the elimination of unnecessary management labor not
                  > be a benefit to a client? Maybe, the case needs to be made
                  > further up the food chain?

                  What will the client do with the managers whose "unnecessary management
                  labor" will be eliminated? Or, more to the point, what do the affected
                  managers *think* the client will do with them?

                  If the particular client has a history of dealing with "unnecessary"
                  positions by firing the persons therein, the affected managers are going
                  to be *very* strongly motivated to make sure *their* positions are seen
                  as "necessary".

                  --
                  Edmund Schweppe -- schweppe@... -- http://schweppe.home.tiac.net
                  The opinions expressed herein are at best coincidentally related to
                  those of any past, present or future employer.
                • Steven Gordon
                  The main difference is whose position is being eliminated. It bothers me that projects which reduce how much labor it takes to do production
                  Message 8 of 20 , Oct 12, 2003
                  • 0 Attachment
                    The main difference is whose position is being eliminated.

                    <Rant mode on>

                    It bothers me that projects which reduce how much labor it takes to do production will lead to reductions in force, but projects that reduce how much management is required just leads to managers with time on their hands.

                    And managers with time on their hands leads to micromanagement activities like tracking estimates vs. actuals in a process where the metric is not highly correlated to success. Next, they will try to weigh the process down with collecting even more metrics that are not success factors, and start applying 6-sigma optimizations on them. Will anybody notice that these measured metrics will improve, but actual productivity and true successes will decrease? Probably not, because they will define productivity and success in terms of the metrics they are collecting and analyzing.

                    <Rant mode off>

                    Sorry if I offended any managers here.

                    -----Original Message-----
                    From: Edmund Schweppe [mailto:schweppe@...]
                    Sent: Sun 10/12/2003 10:21 AM
                    To: scrumdevelopment@yahoogroups.com
                    Cc:
                    Subject: Re: [scrumdevelopment] What's wrong with tracking estimates vs. actuals?

                    Steven Gordon wrote:
                    > I have worked on many projects in the past where the
                    > elimination of labor was considered a significant benefit,
                    > if not the greatest benefit of the resulting process and/or
                    > product.
                    >
                    > Why would the elimination of unnecessary management labor not
                    > be a benefit to a client? Maybe, the case needs to be made
                    > further up the food chain?

                    What will the client do with the managers whose "unnecessary management
                    labor" will be eliminated? Or, more to the point, what do the affected
                    managers *think* the client will do with them?

                    If the particular client has a history of dealing with "unnecessary"
                    positions by firing the persons therein, the affected managers are going
                    to be *very* strongly motivated to make sure *their* positions are seen
                    as "necessary".

                    --
                    Edmund Schweppe -- schweppe@... -- http://schweppe.home.tiac.net
                    The opinions expressed herein are at best coincidentally related to
                    those of any past, present or future employer.



                    Yahoo! Groups Sponsor
                    <http://rd.yahoo.com/M=259395.3614674.4902533.1261774/D=egroupweb/S=1707209021:HM/A=1524963/R=0/SIG=12o885gmo/*http://hits.411web.com/cgi-bin/autoredir?camp=556&lineid=3614674?=egroupweb&pos=HM>
                    <http://us.adserver.yahoo.com/l?M=259395.3614674.4902533.1261774/D=egroupmail/S=:HM/A=1524963/rand=240962557>

                    To Post a message, send it to: scrumdevelopment@...
                    To Unsubscribe, send a blank message to: scrumdevelopment-unsubscribe@...

                    Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service <http://docs.yahoo.com/info/terms/> .
                  • David J. Anderson
                    Steve, You are defining what bad managers do - managers who don t know what management is and probably have never been trained as proper managers. For the
                    Message 9 of 20 , Oct 12, 2003
                    • 0 Attachment
                      Steve,

                      You are defining what bad managers do - managers who
                      don't know what management "is" and probably have
                      never been trained as proper managers.

                      For the rest of us good managers - we just tune you
                      out :-)

                      David


                      --- Steven Gordon <sagordon@...> wrote:

                      The main difference is whose position is being
                      eliminated.

                      <Rant mode on>

                      It bothers me that projects which reduce how much
                      labor it takes to do production will lead to
                      reductions in force, but projects that reduce how much
                      management is required just leads to managers with
                      time on their hands.

                      And managers with time on their hands leads to
                      micromanagement activities like tracking estimates vs.
                      actuals in a process where the metric is not highly
                      correlated to success. Next, they will try to weigh
                      the process down with collecting even more metrics
                      that are not success factors, and start applying
                      6-sigma optimizations on them. Will anybody notice
                      that these measured metrics will improve, but actual
                      productivity and true successes will decrease?
                      Probably not, because they will define productivity
                      and success in terms of the metrics they are
                      collecting and analyzing.

                      <Rant mode off>

                      Sorry if I offended any managers here.



                      __________________________________
                      Do you Yahoo!?
                      The New Yahoo! Shopping - with improved product search
                      http://shopping.yahoo.com
                    Your message has been successfully submitted and would be delivered to recipients shortly.