Re: [hackers-il] Automatic Test Suites
- On Fri, 3 Jan 2003, Shlomi Fish wrote:
> My question is: what experience have you had with automatic test suites?i've seen several good test suites, and i've participated in writing 2
> How much do they help identifying bugs and broken code? What kind of tests
> would you recommend?
others. i distinguish between correctness tests and stress tests.
correctness tests deal with checking that a specific function is done
properly (gives the expected reuslts). this is mostly useful when you
re-write your code and want to make sure it didn't break, or when you add
new features to a large feature-full project.
stress tests deal with bombarding your code in some manner, and making
sure it does not crash. this is useful both in early development stage,
and when the project is mature.
i've seen stress testing doing a lot of good, both for the developers, and
for users. as a developer - it helps finding instability bugs, races and
the like. its not somehting that comes instead of manual testing - but
rather complements it.
when a project is small, or does something small, it is rather easy to
test it. when it becomes large, and contains a lot of features, manual
tests simply don't cope well with the task. when i was working on large
industry project, which was first only being tested manually, i saw that
we got bugs re-crawling into the system (sometimes the same bugs,
soemtimes new bugs with the same symptoms). after a while, the testing
team started writing automatic tests for the code (it was driven by a GUI
on Unix, so they used X-runner for the testing) - and eventually, a single
machine was dedicated to run automatic regression tests on the latest
'internal release' version of the code all the time - and this helped us
find bugs faster. finding a bug close to when you create it is very
helpful - since it usually lies in the new code you just added (thought
someitmes its a bug in the old code, that the new code now exposed due to
using the old code in new ways).
thus, i think my (current) conclusion is that automatic tests should be
developed along side the development of the software, and the ammount of
effort spent on it, should be proportional to the size and complexity of
the software, as well as to the maturity stage of the software (the more
mature it is - the more tests you need in order to make sure it keeps
"For world domination - press 1,
or dial 0, and please hold, for the creator." -- nob o. dy