Loading ...
Sorry, an error occurred while loading the content.
 

AT run times (was [XP] Re: Continous integration without version control)

Expand Messages
  • wecaputo@thoughtworks.com
    ... I am curious how long people s AT s take to run. I have often seen comments about hours and days. Is this because they are not automated? Something else?
    Message 1 of 5 , Aug 31, 2001
      Russell Gold:
      >> That acceptance tests are always able to
      >> be completed in a few hours?

      Ron Jeffries:
      >I would think that strictly they only have to be completed in the
      >shortest time you ever wish to release a version in. But a few hours
      >would be good, yes. Whenever acceptance tests take a long time, they
      >make us fearful to release code, or make us release code that may not
      >work. Either is bad. So I'd work hard to make ATs fast.

      I am curious how long people's AT's take to run. I have often seen comments
      about hours and days. Is this because they are not automated? Something
      else?

      We had 300 AT's running for our last project, and they only took about 15
      minutes to run. I want to find out what other people are doing, to see if:

      a)We were doing something differnt and calling them AT's
      b)We were doing something different, and were lucky/insightful
      c)The hours and days AT run time is actually not the case.
      d)Any other phenomena that I can't think of.

      Best,
      Bill
    • Ron Jeffries
      ... C3 s ran in hours, because they were paying lots of people in lots of combinations. We generally ran them overnight, with a selection of them run more
      Message 2 of 5 , Aug 31, 2001
        Around Friday, August 31, 2001, 5:37:13 PM, wecaputo@... wrote:

        > I am curious how long people's AT's take to run. I have often seen comments
        > about hours and days. Is this because they are not automated? Something
        > else?

        C3's ran in hours, because they were paying lots of people in lots of
        combinations. We generally ran them overnight, with a selection of
        them run more frequently. They were automated, just big and slow. It
        worked for us, but maybe we weren't extreme enough.

        Ronald E Jeffries
        www.XProgramming.com
        You can observe a lot by watching. --Yogi Berra
      • Laurent Bossavit
        ... The ones for the Web content management system (project before current) took at most 10 minutes. A more recent project elsewhere ran all tests (unit and
        Message 3 of 5 , Sep 1, 2001
          > I am curious how long people's AT's take to run.

          The ones for the Web content management system (project before
          current) took at most 10 minutes. A more recent project elsewhere ran
          all tests (unit and acceptance) in under a minute - small project
          though.


          -[Morendil]-
          This message will self-destruct in 5 seconds...4...3...2.
        • Doug Swartz
          ... Our Acceptance tests are automated. Each test runs somehere in the range from a few minutes to 30 minutes or so. Each test checks a number of
          Message 4 of 5 , Sep 1, 2001
            wecaputo@... wrote:

            > I am curious how long people's AT's take to run. I have often seen comments
            > about hours and days. Is this because they are not automated? Something
            > else?
            >
            > We had 300 AT's running for our last project, and they only took about 15
            > minutes to run. I want to find out what other people are doing, to see if:
            >
            > a)We were doing something differnt and calling them AT's
            > b)We were doing something different, and were lucky/insightful
            > c)The hours and days AT run time is actually not the case.
            > d)Any other phenomena that I can't think of.
            >

            Our Acceptance tests are automated. Each test runs somehere in the range from a few
            minutes to 30 minutes or so. Each test checks a number of scenarios/conditions. Some of
            them test upwards of 400 to 600 scenarios. Each of our four products has about a dozen,
            or so tests.

            Our AT's can run long because the customers like to start with a (sometimes rather
            large) set of data extracted from production situations to build the tests. They
            manually add conditions not covered in the beginning set of data.

            Doug Swartz
            daswartz@...
          • Buddha Buck
            ... We have a mixture of fully-automated and partially-automated acceptance tests for the project I m working on. We only fully acceptance-test release
            Message 5 of 5 , Sep 1, 2001
              wecaputo@... writes:

              > I am curious how long people's AT's take to run. I have often seen comments
              > about hours and days. Is this because they are not automated? Something
              > else?

              We have a mixture of fully-automated and partially-automated
              acceptance tests for the project I'm working on. We only fully
              acceptance-test release candidates. So between cutting a CD, cleaning
              a machine and installing the software onto it, etc, all but one of the
              tests take about a half-day (4 hours).

              That one remaining test takes a lot longer to run. It covers several
              story-equivalents (we aren't an XP shop, although I'm trying to bring
              in XP principles). The biggest two being "It must be able to process
              at least 1600 transactions/day" and "It must be able to run
              continuously, 24/7, for months at a time without the need to reboot,
              except for a daily, scheduled, quiescent period to allow for database
              backups."

              We can, and do, push the system as hard as possible to test
              performance, so testing the 1600 transactions/day is not hard. 1600
              transactions could be processed without much problem in an hour or so
              of testing. It's the longevity requirement that's the issue. Once we
              discovered, by letting the program run (at about 50,000
              transactions/day for a week) that we were leaking approximately 100
              bytes per transaction. That's a small leak, but it is sufficient that
              over time, it would degrade the performance and eventually cause the
              program to fail.

              So one of our acceptance tests is to run the program on a longevity
              test bed for a week, checking memory and performance criteria (as well
              as verifying that it handles enterring and exiting the scheduled
              quiescent periods OK).

              >
              > We had 300 AT's running for our last project, and they only took about 15
              > minutes to run. I want to find out what other people are doing, to see if:
              >
              > a)We were doing something differnt and calling them AT's
              > b)We were doing something different, and were lucky/insightful
              > c)The hours and days AT run time is actually not the case.
              > d)Any other phenomena that I can't think of.

              I think this is a case of d)
            Your message has been successfully submitted and would be delivered to recipients shortly.