David,

Beware metrics. We are enamored with them from the days of waterfall, when we couldn’t tell what was going on until the end of the project. So, we devised metrics to attempt to read the tea leaves of what might be going on so we could get early warnings. Earned value is a great example of this. Also, we developed metrics to prove that things were improving to our customers even though over ˝ of our projects failed. See, we are getting better, so leave us alone and please don’t fire us.

The two metrics that matter most in Scrum are the inspect and adapt of reality that happen within teams at the daily Scrum, and the inspect and adapt between teams and customers at the Sprint Review, end of iteration. Burndown charts show trends in these inspections so we can make changes as necessary.

We use metrics when there are problems. We use such metrics to see where we are – quantitatively – regarding a problem. We then track our improvement with the problem until it is better. Then we remove the metric. Because, gathering the data and then analyzing the data is not developing software for our customers. The more metrics we track the less code we develop.

Plus, the problem of suboptimal metrics and the law of unintended consequences. When you measure something, something else changes as everyone optimizes to that measured area. For instance, if you reward for a combination of productivity and quality, you may discourage collaboration to produce really important functionality that requires more thinking.

The primary metric used in the software, product development side of the house in Scrum is actionable product backlog items/$100,000 developed with a number of defects. With this metric, obstensibly, 12/$100,000 with 20 defects is better than 10/$100,000 with 40 defects. However, to balance this, we ask the business to use the metric of return on investment. If the ROI of the first measure is 24% and the ROI of the second is 36%, we have metrics at odds with each other.

Don’t measure, do!

Ken

Hello Ken:

I've assembled a "shoot-from-the-hip" list of metrics/questions I would like to ask in a survey in order to analyze the links between Agile Methods and System Success ...

Basically, I'm attempting to:

1. Identify the major tenets, principles, and factors of Agile Methods.
2. Identify a small set of relevant metrics.
3. Design a doctoral dissertation to test the hypotheses that Agile Methods do indeed enhance system success.

Of course, I figure there are only a handful of metrics that really count (e.g., Stefan Thomke said that a high number of frequent iterations results in 100% customer acceptance, customer satisfaction, and system performance) ...

If you know of a framework of metrics for Agile Methods, please share it with me, thanks ...

-Dave Rico-

EARLY CUSTOMER INVOLVEMENT

Number of Customers
Degree of Customer Involvement
Degree of Stories Produced by Customers
Degree of Tests Produced by Customers
Degree of Testing Performed by Customers
Degree of Hands On Evaluation and Feedback by Customers
Degree of Stop Work Authority by Customers
Ratio of End Users to Non-End Users

ITERATIVE RELEASES

Number of Iterations
Duration of Iterations
Number of New Stories Per Iteration 

Number of Bugs Found Per Iteration

Number of Bugs Fixed Per Iteration 
Number of Iterations Evaluated by Customers
Number of Iterations Evaluated by End Users
Ratio of Iterations to Releases

SELF ORGANIZING TEAMS

Number of Teams
Number of Team Members Per Team
Degree of Team Autonomy
Degree of Team Authority and Empowerment
Degree of Pair Programming Used
Degree of Team Cooperation
Degree of Team Conflict
Ratio of Programmers to Non-Programmers

FLEXIBILITY

Degree of Requirements Informality
Degree of Process Informality
Degree of Documentation Informality
Degree of Architecture Extensibility

SYSTEM SUCCESS

Number of Systems Completed Per Project
Number of Stories Per Project
Degree of System Usability, Quality, and Reliability
Number of Systems Deployed to End Users
Number of Deployments Used by End Users
Degree of Customer Satisfaction
Degree of End User Satisfaction

CONTROL

Total Number of Agile Methods Training Hours Required Per Project
Total Cost of Agile Methods Training Required Per Project
Size of Project in Total Number of People (including Customers)
Size of Project in Staff Hours
Size of Project in Lines of Code
Size of Project in Number of Function Points
Number of Bugs Found
Number of Bugs Fixed 

Shortest Period of Time to Fix Bug

Longest Period of Time to Fix Bug

Average Period of Time to Fix Bug 
Industry Code of Organization Producing System
Firm/Business Unit Name
Firm/Business Unit Revenues