Loading ...
Sorry, an error occurred while loading the content.

Re: AT run times (was [XP] Re: Continous integration without version control)

Expand Messages
  • Doug Swartz
    ... Our Acceptance tests are automated. Each test runs somehere in the range from a few minutes to 30 minutes or so. Each test checks a number of
    Message 1 of 5 , Sep 1, 2001
    • 0 Attachment
      wecaputo@... wrote:

      > I am curious how long people's AT's take to run. I have often seen comments
      > about hours and days. Is this because they are not automated? Something
      > else?
      >
      > We had 300 AT's running for our last project, and they only took about 15
      > minutes to run. I want to find out what other people are doing, to see if:
      >
      > a)We were doing something differnt and calling them AT's
      > b)We were doing something different, and were lucky/insightful
      > c)The hours and days AT run time is actually not the case.
      > d)Any other phenomena that I can't think of.
      >

      Our Acceptance tests are automated. Each test runs somehere in the range from a few
      minutes to 30 minutes or so. Each test checks a number of scenarios/conditions. Some of
      them test upwards of 400 to 600 scenarios. Each of our four products has about a dozen,
      or so tests.

      Our AT's can run long because the customers like to start with a (sometimes rather
      large) set of data extracted from production situations to build the tests. They
      manually add conditions not covered in the beginning set of data.

      Doug Swartz
      daswartz@...
    • Buddha Buck
      ... We have a mixture of fully-automated and partially-automated acceptance tests for the project I m working on. We only fully acceptance-test release
      Message 2 of 5 , Sep 1, 2001
      • 0 Attachment
        wecaputo@... writes:

        > I am curious how long people's AT's take to run. I have often seen comments
        > about hours and days. Is this because they are not automated? Something
        > else?

        We have a mixture of fully-automated and partially-automated
        acceptance tests for the project I'm working on. We only fully
        acceptance-test release candidates. So between cutting a CD, cleaning
        a machine and installing the software onto it, etc, all but one of the
        tests take about a half-day (4 hours).

        That one remaining test takes a lot longer to run. It covers several
        story-equivalents (we aren't an XP shop, although I'm trying to bring
        in XP principles). The biggest two being "It must be able to process
        at least 1600 transactions/day" and "It must be able to run
        continuously, 24/7, for months at a time without the need to reboot,
        except for a daily, scheduled, quiescent period to allow for database
        backups."

        We can, and do, push the system as hard as possible to test
        performance, so testing the 1600 transactions/day is not hard. 1600
        transactions could be processed without much problem in an hour or so
        of testing. It's the longevity requirement that's the issue. Once we
        discovered, by letting the program run (at about 50,000
        transactions/day for a week) that we were leaking approximately 100
        bytes per transaction. That's a small leak, but it is sufficient that
        over time, it would degrade the performance and eventually cause the
        program to fail.

        So one of our acceptance tests is to run the program on a longevity
        test bed for a week, checking memory and performance criteria (as well
        as verifying that it handles enterring and exiting the scheduled
        quiescent periods OK).

        >
        > We had 300 AT's running for our last project, and they only took about 15
        > minutes to run. I want to find out what other people are doing, to see if:
        >
        > a)We were doing something differnt and calling them AT's
        > b)We were doing something different, and were lucky/insightful
        > c)The hours and days AT run time is actually not the case.
        > d)Any other phenomena that I can't think of.

        I think this is a case of d)
      Your message has been successfully submitted and would be delivered to recipients shortly.