Loading ...
Sorry, an error occurred while loading the content.

Re: AT run times (was [XP] Re: Continous integration without version control)

Expand Messages
  • Buddha Buck
    ... We have a mixture of fully-automated and partially-automated acceptance tests for the project I m working on. We only fully acceptance-test release
    Message 1 of 5 , Sep 1 10:26 AM
      wecaputo@... writes:

      > I am curious how long people's AT's take to run. I have often seen comments
      > about hours and days. Is this because they are not automated? Something
      > else?

      We have a mixture of fully-automated and partially-automated
      acceptance tests for the project I'm working on. We only fully
      acceptance-test release candidates. So between cutting a CD, cleaning
      a machine and installing the software onto it, etc, all but one of the
      tests take about a half-day (4 hours).

      That one remaining test takes a lot longer to run. It covers several
      story-equivalents (we aren't an XP shop, although I'm trying to bring
      in XP principles). The biggest two being "It must be able to process
      at least 1600 transactions/day" and "It must be able to run
      continuously, 24/7, for months at a time without the need to reboot,
      except for a daily, scheduled, quiescent period to allow for database

      We can, and do, push the system as hard as possible to test
      performance, so testing the 1600 transactions/day is not hard. 1600
      transactions could be processed without much problem in an hour or so
      of testing. It's the longevity requirement that's the issue. Once we
      discovered, by letting the program run (at about 50,000
      transactions/day for a week) that we were leaking approximately 100
      bytes per transaction. That's a small leak, but it is sufficient that
      over time, it would degrade the performance and eventually cause the
      program to fail.

      So one of our acceptance tests is to run the program on a longevity
      test bed for a week, checking memory and performance criteria (as well
      as verifying that it handles enterring and exiting the scheduled
      quiescent periods OK).

      > We had 300 AT's running for our last project, and they only took about 15
      > minutes to run. I want to find out what other people are doing, to see if:
      > a)We were doing something differnt and calling them AT's
      > b)We were doing something different, and were lucky/insightful
      > c)The hours and days AT run time is actually not the case.
      > d)Any other phenomena that I can't think of.

      I think this is a case of d)
    Your message has been successfully submitted and would be delivered to recipients shortly.