On Tuesday, September 30, 2003, at 12:12:59 PM, Tom Stambaugh wrote:
> I wonder if some of the apparently reduced cost of change in XP is instead
> an illusion resulting from moving those costs to the test development area -
> and then not measuring them as carefully. When we calculate the "cost of
> change", do we include the cost of maintaining or extending the test
> suite/harness in our calculations?
I doubt whether anyone has done (or could do) a comparison, as no two
projects are comparable.
However, those of who do more testing than we used to are quite sure that
our total time for development is reduced thereby, and that conformance to
requirements is increased.
> Could some of the practitioners perhaps contribute a brief summary of how
> they measure this "cost of change", and what costs are included? For
> example - do we include the cost of enumerating the user stories, then
> collecting and structuring the example data used to generate test cases in
> our cost summaries? Do we include the cost of refactoring the test harnesses
> when a new story or bug reveals fundamental errors or omissions?
>From this list, I'd note:
- All the information in the user stories must be created in any case,
and there is no reason to think that doing it in story style is more
costly. If anything, owing to reduced formality, I would expect it to be
- Unless we are comparing a tested project to an untested one, all the
information used for tests needs to be created somewhere in the project
context in any case. Are you thinking that perhaps agile projects test
- <sigh> Refactoring as part of incremental development is not a matter
of doing large amounts of work over. It is more like sanding a surface
prior to painting. First the coarse sandpaper takes down the rough spots,
then finer and finer grades until the surface is as smooth as we want.
People who sand surfaces don't think of sanding multiple times as doing
things over again. Neither should people who develop software, or test
harnesses, incrementally, think of refactoring as doing things over
again. It's doing the thing first roughly, then refining it.
> Perhaps because agile methods are still new to me, I find my head-scratching
> time often moves from the code under test to the test harness, rather than
> disappearing altogether.
I may not be correctly apprehending your use of the word "harness".
Certainly one thinks about testing. If one is spending much time thinking
about the harness in the sense of framework, it might be too much. We don't
need a testing framework, we need tests.
More thinking about the tests seems to reduce the time spend coding by more
than the same amount.
> I find that an important part of my development
> time for new functionality is determining how to think about it. I have
> always rejected waterfall-style "white-board engineering" or "word
> engineering", in favor of Smalltalk-style prototyping. I'm not sure whether
> or how much XP has improved my prototyping - but I'm pretty sure it has NOT
> been exponential. Perhaps this falls under the umbrella of "spike
I'm not sure that XP claims to improve anything exponentially, most
especially not a good evolution-from-prototype approach. What XP claims is
that the old saw about the cost of change being exponential gets translated
in the minds of people as "change is bad, very bad", and that in fact, in
incremental design supported by tests and refactoring, change is not all
that bad after all.
> For example, I'm just rediscovering the power of streams, sequences, and
> delayed evaluation as described in the Wizard Book, because I'm working on a
> project where I'm manipulating LOTS of hierarchical structures. I can write
> the code (often just transcribing it from the Scheme examples in the text)
> easily enough - but how do I even think about *testing* it? Do I bother?
I don't know. The first time I spent more than five minutes in the
debugger, I'd start trying to figure it out.
> I wonder how the *total* cost of new functionality (including
> headscratching, prototyping, and test development time) using XP compares
> to, for instance, my habit of prototyping and iterating until it works "good
> enough". I ask this because I find myself just as able to get lost in
> designing and coding unit tests as I used to get doing "prototypes".
It could happen. XP and Scrum and so on are probably more about taming the
effects of change, and making results more predictable thereby, than about
reducing the overall cost of projects. We get projects done that would
otherwise not get done; we provide information to enable knowing when
things will be done; and we generally produce results that satisfy the
If you're already doing those things, XP or Scrum might help you do them
more easily, but Done At Right Quality As Predicted is what it's really all
No one expects the Spanish Inquisition ...