Loading ...
Sorry, an error occurred while loading the content.
 

RE: [XP] Acceptance tests

Expand Messages
  • Ted M. Young
    ... Both. We run a set of acceptance (Customer) tests where each test can be run independently. Since that doesn t provide an adequate level of testing (in
    Message 1 of 48 , Nov 30, 2001
      Glen Stampoultzis asked on Wednesday, November 28, 2001 3:15 PM:
      >
      > I'd be interested in peoples experiences with acceptance testing JSP.
      >
      > * How do you handle database interactions?
      > - Do you set up the database at the start of your test suite with
      > known values then run a whole bunch of tests or do you try to make each
      > acceptance test run independently?

      Both. We run a set of acceptance (Customer) tests where each test can be run
      independently. Since that doesn't provide an adequate level of testing (in
      terms of coverage and/or complexity) for certain (many?) situations, we'll also
      define tests which look like multiple tests running in a specific order.

      Depending on the level of the story, the level of Customer Tests will vary.
      There will be the simple ones, such as Create a new Client, and the more
      complex ones such as "Limit the Number of Active Cases to 6 Per Client".

      > - Do you use a dedicated database for acceptance testing?

      Yes. This would be running on an integration machine which has its own
      database. It gets reset to a known state prior to each test and/or test run:
      sometimes set to empty, other times set with some data, and eventually set to
      lots of data for tests that need it.

      > * How do you test things like email being sent?

      Create a test (mock) SMTP server and verify that it received the mail that it
      was supposed to receive. I believe there's at least a couple of such mock
      servers.

      > * What do you do about interactions external systems?
      > - Stub them out somehow?
      > - Set them up to a known state?
      > - Pretent the problem doesn't exist?

      Well, since we're talking Customer tests, we'll start off with Mock systems,
      such as an interface to a mainframe system or to an SMTP server, where we tell
      the Mock system what interactions to expect ("run report JABC", "send email to
      xyz@...", etc.) and then test to see if they indeed had those
      interactions. If we're building the external system, then we'll do the same
      thing from the other point of view, where the first system becomes the external
      system and the previously external system becomes the system under test.

      At lower levels, these interactions would be tested using unit tests and more
      Mock objects.

      > * What testing tools do you use?
      > - Do you use junit for acceptance testing?
      > - HttpUnit?
      > - Anyone tried latka?
      > - Some commerical tool?
      > - Home grown tool?

      We use JUnit as the basis for all testing. HttpUnit builds on JUnit and we use
      that. We also built our own tool on top of HttpUnit to do the actual Customer
      tests. We'll be moving our tool to sit on top of Cactus instead of HttpUnit as
      our user interface comes into existence -- right now we use Struts for the
      presentation layer, but we don't really have much of a user interface, just
      forms for input and tables for output.

      I've never heard of latka ('cept for the yummy spinach latkes that my sister
      makes!).

      > * How do you handle time based dependancies? Eg, user presses a button to
      > schedule a test to run tomorrow.

      Good question. Right now we've only run into a story for the first half of
      that: store/schedule the notification alert. Right now we just look to see if
      the proper entry was made in the scheduling table in the database. My current
      plan for testing this is to have a Mock object that represents the current
      date/time, so that we can do things like set a specific date/time, have it run
      faster than real time (real time seconds = system hours), etc.

      > * How do you schedule your acceptance tests?
      > - Entered into the schedule as a story?
      > - Done before or after you write the your story?
      > - In the same iteration or another iteration?

      Every story has a customer test defined, otherwise we don't know when we can
      cross the story off the list and put it into the completed pile. When we
      estimate stories, we need to take into account how long the test is going to
      take to write. Since we already have a pretty decent automated customer test
      tool, we feel we can do them fairly quickly. I'm sure we'll have to schedule
      some stories that are aimed at improving our tool to sustain our ability to
      write customer tests quickly and easily.

      > * What do you still test manually?
      > - Javascript?

      We're not planning on using JavaScript for exactly that reason: it's too hard
      to test, not to mention that if I never have to write another line of
      JavaScript, it'll be far too soon.

      The only thing that we'll test manually is the user interface: making sure it
      looks good and is usable. We may automate some of that with HttpUnit for more
      regression type testing, but user interfaces are too much of a moving target to
      automate right now.

      > * How fast do your acceptance tests run and how frequently do you run them?

      I think they currently take less than a minute and we run them every hour. An
      email goes out to tell us how it went and a web page is automatically built to
      report the status as well (the email is generally used to indicate if someone
      broke the build).

      > We are starting acceptance testing soon so I'm really interested in peoples
      > real world experiences dealing with these sorts of issues.

      Let us know how it goes!

      ;ted

      --
      Ted M. Young, Java Architect
      Caribou Lake -- Custom E-Solutions for Your Organization
      http://www.CaribouLake.com
      No plan survives first contact with the enemy.
    • Ted M. Young
      ... Both. We run a set of acceptance (Customer) tests where each test can be run independently. Since that doesn t provide an adequate level of testing (in
      Message 48 of 48 , Nov 30, 2001
        Glen Stampoultzis asked on Wednesday, November 28, 2001 3:15 PM:
        >
        > I'd be interested in peoples experiences with acceptance testing JSP.
        >
        > * How do you handle database interactions?
        > - Do you set up the database at the start of your test suite with
        > known values then run a whole bunch of tests or do you try to make each
        > acceptance test run independently?

        Both. We run a set of acceptance (Customer) tests where each test can be run
        independently. Since that doesn't provide an adequate level of testing (in
        terms of coverage and/or complexity) for certain (many?) situations, we'll also
        define tests which look like multiple tests running in a specific order.

        Depending on the level of the story, the level of Customer Tests will vary.
        There will be the simple ones, such as Create a new Client, and the more
        complex ones such as "Limit the Number of Active Cases to 6 Per Client".

        > - Do you use a dedicated database for acceptance testing?

        Yes. This would be running on an integration machine which has its own
        database. It gets reset to a known state prior to each test and/or test run:
        sometimes set to empty, other times set with some data, and eventually set to
        lots of data for tests that need it.

        > * How do you test things like email being sent?

        Create a test (mock) SMTP server and verify that it received the mail that it
        was supposed to receive. I believe there's at least a couple of such mock
        servers.

        > * What do you do about interactions external systems?
        > - Stub them out somehow?
        > - Set them up to a known state?
        > - Pretent the problem doesn't exist?

        Well, since we're talking Customer tests, we'll start off with Mock systems,
        such as an interface to a mainframe system or to an SMTP server, where we tell
        the Mock system what interactions to expect ("run report JABC", "send email to
        xyz@...", etc.) and then test to see if they indeed had those
        interactions. If we're building the external system, then we'll do the same
        thing from the other point of view, where the first system becomes the external
        system and the previously external system becomes the system under test.

        At lower levels, these interactions would be tested using unit tests and more
        Mock objects.

        > * What testing tools do you use?
        > - Do you use junit for acceptance testing?
        > - HttpUnit?
        > - Anyone tried latka?
        > - Some commerical tool?
        > - Home grown tool?

        We use JUnit as the basis for all testing. HttpUnit builds on JUnit and we use
        that. We also built our own tool on top of HttpUnit to do the actual Customer
        tests. We'll be moving our tool to sit on top of Cactus instead of HttpUnit as
        our user interface comes into existence -- right now we use Struts for the
        presentation layer, but we don't really have much of a user interface, just
        forms for input and tables for output.

        I've never heard of latka ('cept for the yummy spinach latkes that my sister
        makes!).

        > * How do you handle time based dependancies? Eg, user presses a button to
        > schedule a test to run tomorrow.

        Good question. Right now we've only run into a story for the first half of
        that: store/schedule the notification alert. Right now we just look to see if
        the proper entry was made in the scheduling table in the database. My current
        plan for testing this is to have a Mock object that represents the current
        date/time, so that we can do things like set a specific date/time, have it run
        faster than real time (real time seconds = system hours), etc.

        > * How do you schedule your acceptance tests?
        > - Entered into the schedule as a story?
        > - Done before or after you write the your story?
        > - In the same iteration or another iteration?

        Every story has a customer test defined, otherwise we don't know when we can
        cross the story off the list and put it into the completed pile. When we
        estimate stories, we need to take into account how long the test is going to
        take to write. Since we already have a pretty decent automated customer test
        tool, we feel we can do them fairly quickly. I'm sure we'll have to schedule
        some stories that are aimed at improving our tool to sustain our ability to
        write customer tests quickly and easily.

        > * What do you still test manually?
        > - Javascript?

        We're not planning on using JavaScript for exactly that reason: it's too hard
        to test, not to mention that if I never have to write another line of
        JavaScript, it'll be far too soon.

        The only thing that we'll test manually is the user interface: making sure it
        looks good and is usable. We may automate some of that with HttpUnit for more
        regression type testing, but user interfaces are too much of a moving target to
        automate right now.

        > * How fast do your acceptance tests run and how frequently do you run them?

        I think they currently take less than a minute and we run them every hour. An
        email goes out to tell us how it went and a web page is automatically built to
        report the status as well (the email is generally used to indicate if someone
        broke the build).

        > We are starting acceptance testing soon so I'm really interested in peoples
        > real world experiences dealing with these sorts of issues.

        Let us know how it goes!

        ;ted

        --
        Ted M. Young, Java Architect
        Caribou Lake -- Custom E-Solutions for Your Organization
        http://www.CaribouLake.com
        No plan survives first contact with the enemy.
      Your message has been successfully submitted and would be delivered to recipients shortly.