> > > Are you doing pairwise combinations or all combinations? The lore
> > > seems to be that pairwise is enough, though I'm not up enough on
> > > configuration testing to say that with any certainty.
> >And I'm no expert on configuration testing either, but we
> have seen the
> >most value
> >in pairwise combinations. Initially we tested all
> combinations, but as the
> >permutations increased and we gained confidence that pairwise was
> >sufficient, then
> >we adjusted. We still test all combinations from time to
> time if there's
> >a fear
> >that a new feature will break, and then learn from that
> experience. It's a
> >constant game of cost-benefit analysis, and we've made
> mistakes along the way.
I wrote an all-pairs coverage test case generation tool, which you can
download at http://www.satisfice.com/tools/pairs.zip.
Given a table of
variables and values, it tries to find the smallest set of test cases
that pair each value of each variable at least once. An example is
included that shows how a print configuration test suite that would
require 36,864,000 cases to cover for all combinations can achieve
all-pairs coverage in 35 cases.