Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Unit Testing against multiple configurations

Expand Messages
  • Gary Feldman
    ... So you re saying that you did all 4000 unit tests before every commit? If so, then what percentage of those tests didn t touch any piece of code that was
    Message 1 of 30 , Jan 26, 2005
    • 0 Attachment
      Steve Bate wrote:
      > It depends on the team. We had 4000 unit tests that ran in 15-20 seconds
      > total and had about 500 integration tests that ran in 4-5 minutes. If
      > all those tests took 500ms each, the test suite would have taken
      > 37.5 minutes to run. Just considering the unit tests, if those took
      > 500ms to run it would have take over 30 minutes to run the precommit
      > tests versus 15-20 seconds. For us, that would have been a very
      > significant difference. It may not be for other teams.

      So you're saying that you did all 4000 unit tests before every commit? If so, then what percentage of those tests didn't touch any piece of code that was changed by the commit? Would it have made a difference, in terms of the quality of the test, if the unit tests that didn't touch the changed code were omitted?

      Suppose you could easily run both the unit and integration tests that touch a set of code changes, while leaving out the ones that didn't. Or suppose a dozen of the integration tests cut a broad swatch, and ran in 6 seconds (12 x 500msec). Would running those dozen integration tests, while omitting a couple thousand irrelevant unit tests (saying cutting the unit test cost in half), produce a better or worse testing strategy? If the answer is that it wouldn't have worked for you, can you at least conceive that it might work well for many other projects?

      > The distinction doesn't create confusion for everyone. It definitely
      > didn't confuse our team (we used the same guidelines that Michael
      > described). See my previous message about benefits of unit tests beyond
      > performance considerations. Another potential benefit that I didn't
      > mention earlier is simpler test setup. With integration tests, not only
      > does the environment have to be configured for the unit under test, but
      > also for the transitive closure of all functionality needed by that
      > tested code.

      Ten years ago those complaints would have been about running unit tests, particularly according to your definition - you had to write a special harness instead of simply running the CLI of the program being tested; it was too much work to write stubs instead of simply testing against the lower levels that already existed; you'd have to complicate the configuration in order to link against the stubs instead of the real thing, as well as making the build do both, etc.

      Yet I remember writing unit tests twenty years ago. I was working on a character cell, but we had hooks to fire up editor commands from the command line, a tool for comparing screen results (though my team didn't use it much, as the underlying editing and display engines weren't in our scope), a tool for doing somewhat smart text matching (what Phlip called fuzzy matching recently) and a tool for measuring coverage to the branch level. We didn't worry about them not being unit tests according to your definition, yet they achieved the same result. The architecture was such that we could write tests that focused on individual sections of code. By diffing the code, we could isolate almost all bugs quickly - and the ones we couldn't weren't likely to be caught by your style of unit tests, either (unit tests aren't very good at detecting bad pointer references in the absence of special hardware/debugger support). We weren't doing TDD, of course, nor did we have 100% coverage
      , but at the time I left, policy was that all new code had to have tests, and adding tests for existing code was a virtuous activity.

      Granted, today's unit testing frameworks are more than a SMOP over the technology of the eighties. They rely upon reflection and inheritance - language technologies that weren't widely then. But the point is that you can either construct rigid boxes and live within them - or you can focus on the needs, and redesign the boxes to fit.

    Your message has been successfully submitted and would be delivered to recipients shortly.