Loading ...
Sorry, an error occurred while loading the content.

automating Customer Tests

Expand Messages
  • Michael Finney
    Which Customer Tests (aka Acceptance Tests) should be automated? The situation is that the testers have given the Onsite-Customer suggestions as to what the
    Message 1 of 9 , Dec 18, 2001
    • 0 Attachment
      Which Customer Tests (aka Acceptance Tests) should be automated?

      The situation is that the testers have given the Onsite-Customer suggestions
      as to what the Customer may want to specify. The Customer looked at the
      specifications and said "looks great with some tweaks. These are my
      specifications for these stories." There is a lot of Look and Feel GUI
      checking such as making sure a window minimizes, closes via 'X button' and
      so on.

      A thought was that maybe automation can happen and the testers can run their
      manual tests when they want to too.

      I skimmed:
      http://www.xp2001.org/xp2001/conference/papers/Chapter26-Finsterwalder.pdf

      Do you believe in what the paper says? It sounds like it downplays
      automation a lot. I noticed it dicourages GUI test tools and the author
      achieves automation using the same language as what the application is built
      in. I understand why. The GUI test tools have had a lot of energy and time
      spent on them without return.

      By the way, we have a traditional Swing Application hitting the DB via JDBC.

      Thank you for your help.
      Michael Finney
      Sun Certified Programmer for the Java 2 Platform
      Sun Certified Developer for the Java 2 Platform
      Cofounder of PPJDG - http://www.ppjdg.org
      Cofounder of cosAgile - Colorado Springs XP Users Group -
      http://groups.yahoo.com/group/cosAgile
    • Ron Jeffries
      ... Just the ones that matter. Ron Jeffries www.XProgramming.com Sorry about your cow ...
      Message 2 of 9 , Dec 18, 2001
      • 0 Attachment
        Around Tuesday, December 18, 2001, 4:28:38 PM, Michael Finney wrote:

        > Which Customer Tests (aka Acceptance Tests) should be automated?

        Just the ones that matter.

        Ron Jeffries
        www.XProgramming.com
        Sorry about your cow ...
      • yet another bill smith
        ... My dentist has a sign in his office that says, You don t have to floss _all_ your teeth--just the ones you want to keep!
        Message 3 of 9 , Dec 18, 2001
        • 0 Attachment
          Ron Jeffries wrote:
          >
          > Around Tuesday, December 18, 2001, 4:28:38 PM, Michael Finney wrote:
          >
          > > Which Customer Tests (aka Acceptance Tests) should be automated?
          >
          > Just the ones that matter.
          >
          My dentist has a sign in his office that says, "You don't have to floss
          _all_ your teeth--just the ones you want to keep!"
        • Ralph Johnson
          ... Ron is here making the claim that all tests should be automated. Automated tests are good. Most groups don t automate enough tests. But not all tests
          Message 4 of 9 , Dec 19, 2001
          • 0 Attachment
            Ron Jeffries <ronjeffries@...> wrote:
            >Around Tuesday, December 18, 2001, 4:28:38 PM, Michael Finney wrote:

            >> Which Customer Tests (aka Acceptance Tests) should be automated?

            >Just the ones that matter.

            Ron is here making the claim that all tests should be automated.
            Automated tests are good. Most groups don't automate enough tests.
            But not all tests should be automated. Sometimes the cost of
            automating a test is so high that if you require all tests to
            be automated, people will not do the tests. It is better to
            have a few manual tests than not to test at all.

            Suppose we are building a system that is processing some images.
            We want to know whether the images that come out are easier to
            read than the ones that go in. This is nearly impossible to
            automate, but humans can tell very quickly whether the system
            is good enough.

            Unit tests can almost always be automated, but acceptance tests
            less so. But if if takes a tester several days to run through
            the script for the manual acceptance tests, something is wrong!
            Half an hour is one thing. If it takes several days, you will
            not run your acceptance tests every day, and you should.

            -Ralph
          • Keith Nicholas
            ... want to know whether the images that come out are easier to read than the ones that go in. This is nearly impossible to automate, but humans can tell
            Message 5 of 9 , Dec 19, 2001
            • 0 Attachment
              >From: Ralph Johnson [mailto:johnson@...]

              >Suppose we are building a system that is processing some images. We
              want to know whether the images that come out >are easier to read than
              the ones that go in. This is nearly impossible to automate, but humans
              can tell very
              >quickly whether the system is good enough.


              Being that we do image processing.....what you want to do here is have
              all the unit tests on your code. The acceptance tests for the imaging
              is only put in place once you have developed the software so that it
              meets the requirements (ie, you look at the image and say, yep, that's
              what it should look like) then you can take those visually inspected
              images and turn them into and automated test so that each time you run
              the software it produces the same images. If you end up changing the
              processing so a few pixels change because you have come up with a better
              or faster algorithm, then you take those pics and put them back into the
              automated test as your "proofs". If the process for putting the
              "proofs" into your automated tests is tedious....write some software to
              speed it up.

              Most things can have automated testing in some form....some things you
              cant predict the results up front in which case you put tests in
              afterwards to lock things down once you have your desired results. You
              just need to engage your brain a bit and try and find ways to test, all
              too often I hear "this is too hard to test.." and after a bit of thought
              it turns out through slightly different approaches it isn't really that
              hard to test.

              What we do have that's hard to automate is control of mechanical
              devices...end of the day, we have to do manual testing on the actual
              machines...however, for the most, we can simulate the hardware with a
              large amount of confidence that it will work when it is put on the
              actual machine.

              Keith
            • Charlie Poole
              ... have ... imaging ... it ... that s ... inspected ... run ... the ... better ... into the ... software to ... I ve found it useful to separate filtering
              Message 6 of 9 , Dec 19, 2001
              • 0 Attachment
                Keith Nicholas said:
                > Being that we do image processing.....what you want to do here is
                have
                > all the unit tests on your code. The acceptance tests for the
                imaging
                > is only put in place once you have developed the software so that
                it
                > meets the requirements (ie, you look at the image and say, yep,
                that's
                > what it should look like) then you can take those visually
                inspected
                > images and turn them into and automated test so that each time you
                run
                > the software it produces the same images. If you end up changing
                the
                > processing so a few pixels change because you have come up with a
                better
                > or faster algorithm, then you take those pics and put them back
                into the
                > automated test as your "proofs". If the process for putting the
                > "proofs" into your automated tests is tedious....write some
                software to
                > speed it up.

                I've found it useful to separate filtering operations into two
                tests:

                1: Pixels match the expected output
                2: The desired qualitative improvement is there

                It has helped when somebody claimed that the images no longer looked
                right and I was able to demonstrate that the pixels were not changed
                from an earlier version. Of course, we still might need to improve
                the algorithm but it's helpful to know right up front where the
                issue is.

                --------------------------------------------------------------------
                ----------
                Charlie Poole phone: (360) 697-4574
                Poole Consulting mobile: (360) 981-3010
                20224 Pond View Lane NE web:
                www.pooleconsulting.com
                Poulsbo, WA 98370, U.S.A. email: cpoole@...
              • Ron Jeffries
                ... Ralph made these points, which I have reordered for comment. ... Yes, certainly I d agree. Testing not at all is very risky, and if we re anywhere near
                Message 7 of 9 , Dec 21, 2001
                • 0 Attachment
                  Around Wednesday, December 19, 2001, 9:51:21 AM, Ralph Johnson wrote:

                  > Ron Jeffries <ronjeffries@...> wrote:
                  >>Around Tuesday, December 18, 2001, 4:28:38 PM, Michael Finney wrote:

                  >>> Which Customer Tests (aka Acceptance Tests) should be automated?

                  >>Just the ones that matter.

                  Ralph made these points, which I have reordered for comment.

                  > Sometimes the cost of
                  > automating a test is so high that if you require all tests to
                  > be automated, people will not do the tests. It is better to
                  > have a few manual tests than not to test at all.

                  Yes, certainly I'd agree. Testing not at all is very risky, and if
                  we're anywhere near that point, some testing is better than none.

                  > Ron is here making the claim that all tests should be automated.
                  > Automated tests are good. Most groups don't automate enough tests.
                  > But not all tests should be automated.

                  While Ralph's statement is true in a theoretical sense, this is in my
                  opinion, very dangerous advice. I have seen far more teams get in
                  trouble from under-automation than I've ever seen get in trouble from
                  automating too much.

                  > Suppose we are building a system that is processing some images.
                  > We want to know whether the images that come out are easier to
                  > read than the ones that go in. This is nearly impossible to
                  > automate, but humans can tell very quickly whether the system
                  > is good enough.

                  Even this is difficult to automate _once_. Having checked out the
                  image visually, it is often sufficient to capture that input and
                  output and compare the output thereafter. Naturally, when the
                  processing algorithm changes, we'll have to recheck visually. But even
                  in a system whose purpose is to process images and little else, we're
                  probably not tweaking that algorithm every day. In the systems most of
                  us do day to day, there's a lot of work being done beyond the picture
                  processor.

                  > Unit tests can almost always be automated, but acceptance tests
                  > less so. But if if takes a tester several days to run through
                  > the script for the manual acceptance tests, something is wrong!
                  > Half an hour is one thing. If it takes several days, you will
                  > not run your acceptance tests every day, and you should.

                  Here's the rub. When the pressure gets on, and it will, manual tests
                  get skipped. Few of us are disciplined enough to do all the manual
                  tests every time we should. It's easier to run the automated ones as a
                  matter of course. ThoughtWorks runs them at each build ...
                  automatically.

                  Ralph, I hesitate always to disagree with you (thus the delay on this
                  reply). In this case I agree with the fact that not everything can or
                  should be automated. I disagree with saying flatly that not all tests
                  should be automated. Here's what I say:

                  Not all tests should be automated. But the one you are thinking of
                  right now probably should be.

                  Ron Jeffries
                  www.XProgramming.com
                  The Great and Powerful Oz has spoken.
                • nails762
                  ... Although this is a slightly old thread, my team has been thrashing lately about writing and automating Customer Tests, so I would like to get into
                  Message 8 of 9 , Jan 9, 2002
                  • 0 Attachment
                    --- In extremeprogramming@y..., Ralph Johnson <johnson@c...> wrote:
                    > Ron Jeffries <ronjeffries@a...> wrote:
                    > >Around Tuesday, December 18, 2001, 4:28:38 PM, Michael Finney
                    wrote:
                    >
                    > >> Which Customer Tests (aka Acceptance Tests) should be automated?
                    >
                    > >Just the ones that matter.
                    >
                    > Ron is here making the claim that all tests should be automated.
                    > Automated tests are good. Most groups don't automate enough tests.
                    > But not all tests should be automated.
                    <snip />

                    Although this is a slightly old thread, my team has been thrashing
                    lately about writing and automating Customer Tests, so I would like
                    to get into the discussion a little.

                    I have clipped Ralph's description of a particular kind of Customer
                    Test that is difficult to automate. I don't question his reasoning,
                    because I'm not a domain expert; however, I find that not all teams
                    have the required discipline to use Ralph's reasoning in a positive
                    way. Since I'm not sure that was well-expressed, let me clarify this
                    way.

                    As soon as a team agrees that "this kind of test is not worth
                    automating", it may begin to say this about tests that *are* worth
                    automating, if only because the decision is easier to make in the
                    short term. For this reason, I think it's important to make every
                    effort to automate all tests until it's clear that it's not worth it.

                    That said, my reasoning is just "gut feel". I have nothing to support
                    the idea.
                  • Ron Jeffries
                    ... I agree entirely. XP is dials up to ten for a reason: the extreme position of automate EVERY test is the only position that isn t a slippery slope. No
                    Message 9 of 9 , Jan 9, 2002
                    • 0 Attachment
                      Around Wednesday, January 09, 2002, 9:07:47 AM, nails762 wrote:

                      > As soon as a team agrees that "this kind of test is not worth
                      > automating", it may begin to say this about tests that *are* worth
                      > automating, if only because the decision is easier to make in the
                      > short term. For this reason, I think it's important to make every
                      > effort to automate all tests until it's clear that it's not worth it.

                      I agree entirely. XP is "dials up to ten" for a reason: the extreme
                      position of "automate EVERY test" is the only position that isn't a
                      slippery slope.

                      No one automates every test. Everyone should feel badly when they fail
                      to automate even one.

                      Ron Jeffries
                      www.XProgramming.com
                      First they ignore you, then they laugh at you, then they fight you, then you win. - Ghandi
                    Your message has been successfully submitted and would be delivered to recipients shortly.