Loading ...
Sorry, an error occurred while loading the content.

More considerations for not automating

Expand Messages
  • Petersen, Erik
    Some other reasons for low automation takeup. 1) The software is expensive, typically too expensive for a single project to afford 2) It does take longer to
    Message 1 of 25 , Nov 13, 2001
    View Source
    • 0 Attachment
      Some other reasons for low automation takeup.
      1) The software is expensive, typically too expensive for a single project
      to afford
      2) It does take longer to automate. If the testing project timeframe is
      tight, then automation can be eliminated immediately.
      3 Automation paradox: Any rapid development may never be stable enough to
      justify the time to automate it, until later releases. The cost of
      automation and time required may prevent it being done in a later release,
      especially if the life of the product being tested is uncertain.
      4) Any automation is a complex mini-project that comes with a risk of
      failure. If a test or project manager has to guarantee a tested product, it
      is safer (and faster) to go the manual road.

      One of the solutions is the one raised already by Erik Meade, "Add
      an acceptance testing task to every story". Give a healthy estimate for it,
      and argue that it is the only sensible way for a customer to ensure that
      they can check that their app works over time, with the cost effectiveness
      coming in the long term.
      I would argue that the automation team needs to be a closely managed
      sub-project, with staff dedicated just to the automation. with a good
      people-person manager to liase with the customers. It is too easy for a
      developer to decide that their priorities lie in polishing the automation
      programming, rather than working on the real app, or for a tester to decide
      that they will learn to program by overworking the automation.
      cheers,
      Erik

      Software Testing Consultant
      ANZ Banking Group
      Australia


      > -----Original Message-----
      > From: Lisa Crispin [mailto:lisa.crispin@...]
      > Sent: 09 November 2001 05:28
      > To: agile-testing@yahoogroups.com
      > Subject: [agile-testing] Re: testing non-web GUI
      >
      >
      > My experience is like Ron's in that many organizations I'm familiar
      > with do not automate at all, or very little, possibly for
      > load testing
      > and nothing else. This includes organizations who say they are doing
      > XP. Some potential reason:
      >
      > 1. A lot of testers do not have a programming backround so it can be
      > difficult for them to learn how to create automated test scripts and
      > they have a lot of fear about it.
      > 2. Capture/playback tools alone don't help, see 1.
      > 3. Lack of support from management and even from development: "just
      > get the testing done so we can launch". This is worst when you have
      > testing on the "back end".
      > 4. Lack of understanding of the value of automating tests other than
      > unit tests, even among developers on XP projects.
      >
      > Some things that can be done about it (although I admit to struggling
      > in this area myself):
      > 1. Put testers and developers on one team
      > 2. Make the whole team responsible for executing and for automating
      > acceptance tests
      > 3. For each release, if you have any automated tests at all,
      > document
      > as best you can what value they added (time saved, bugs found)
      > 4. The getting support from management and development one is hard,
      > but find someone who sees the value of test automation and
      > get them to
      > help champion the cause
      >
      > Other suggestions? Because these are working some for me, but
      > sometimes I get discouraged fighting the same battles all the time.
      > -- Lisa
      >
      <snip>
      > > I might mention that in the projects I visit, even the ones that
      > have
      > > large and presumably competent QA departments, few (usually zero)
      > > tests are automated. That seems to be very much the wrong proportion
      > > to me. What do folks here think?
      > >
      > > What is the cost of automating a test that doesn't need it? What is
      > > the cost of NOT automating a test that DOES need it?
      > >
      > > Ron Jeffries
      > > www.XProgramming.com
      > > Life tough is. Then die you do. --Yoda (personal communication)
      >
    • wecaputo@thoughtworks.com
      ... Since I have been causing gastric disturbances on at least two lists this week, I figure I am on a roll, so here goes: Writing testing code, and testing
      Message 2 of 25 , Nov 13, 2001
      View Source
      • 0 Attachment
        Erik Peterson:

        >I would argue that the automation team needs to be a closely managed
        >sub-project, with staff dedicated just to the automation. with a good
        >people-person manager to liase with the customers.

        Since I have been causing gastric disturbances on at least two lists this
        week, I figure I am on a roll, so here goes:

        Writing testing code, and testing frameworks is no different than writing
        any other code. We have testing requirements, and tasks to accomplish for
        making them a reality.

        Why is it necessary to reduce the communication of these items by
        separating them out into a separate team and project? If one insists on
        making testing separate from development, then yes, I would say it will be
        very risky, expensive and time-consuming.

        But what happens if we make this a visible part of the team's effort,
        asking the programmers to handle the tasks, ask the customer to steer their
        prioritization, and having our testing experts helping both them?

        What do we lose if we do this? What do we gain?

        Best,
        Bill
      • STEURS Stefan
        ... I d agree on the principal. ... Why does one separate things out in separate teams? Know that is a universal question if there has ever been one. Size of
        Message 3 of 25 , Nov 13, 2001
        View Source
        • 0 Attachment
          > Writing testing code, and testing frameworks is no different
          > than writing
          > any other code. We have testing requirements, and tasks to
          > accomplish for
          > making them a reality.

          I'd agree on the principal.
          >
          > Why is it necessary to reduce the communication of these items by
          > separating them out into a separate team and project? If one
          > insists on
          > making testing separate from development, then yes, I would
          > say it will be
          > very risky, expensive and time-consuming.

          Why does one separate things out in separate teams? Know that is a
          universal question if there has ever been one. Size of the project may be
          one, specialised knowledge may be one, number of projects to be covered may
          be one (generalised testing frameworks), parallel rather than serialised
          work/activities may be one, legal obligations may be one, contractual
          obligations may be one, etc. Sometimes it is just the opposite of what you
          say, it is risk-reducing, more efficient, and faster.
          >
          > But what happens if we make this a visible part of the team's effort,
          > asking the programmers to handle the tasks, ask the customer
          > to steer their
          > prioritization, and having our testing experts helping both them?

          In the right context it leads to success, in the wrong context, it leads to
          extra delays, higher risks, and a slowdown.
          >
          > What do we lose if we do this? What do we gain?

          There aren't any universal truths. Sometimes it is even worse, the truth it
          what you make it to be and in that case it is not repeatable.
          >
          > Best,
          > Bill

          Stefan Steurs.
        • Ron Jeffries
          ... I ll answer these as if Erik thinks the questions are valid. I suspect he is just reporting reasons that people give, not reasons that he personally
          Message 4 of 25 , Nov 14, 2001
          View Source
          • 0 Attachment
            Around Wednesday, November 14, 2001, 12:32:42 AM, Petersen, Erik wrote:

            > Some other reasons for low automation takeup. 

            I'll answer these as if Erik thinks the questions are valid. I suspect
            he is just reporting reasons that people give, not reasons that he
            personally believes.

            > 1) The software is expensive, typically too expensive for a single project
            > to afford

            No real need for expensive software. Tests can be written using jUnit
            or equivalent, or with a simple scripting language (all free), or with
            custom code in the language of the application.

            > 2) It does take longer to automate.  If the testing project timeframe is
            > tight, then automation can be eliminated immediately.

            I suggest that it rarely takes longer to automate a test than it does
            to do it manually two or three times.

            > 3  Automation paradox: Any rapid development may never be stable enough to
            > justify the time to automate it, until later releases.  The cost of
            > automation and time required may prevent it being done in a later release,
            > especially if the life of the product being tested is uncertain.  

            I suspect that if automated testing is not justifiable on the basis of
            instability, either you're not building user-identifiable business
            value, or you aren't doing automation correctly. (I acknowledge that
            the latter isn't entirely helpful because it doesn't say _how_ to do
            it "correctly". I'll give a general algorithm below.)

            > 4) Any automation is a complex mini-project that comes with a risk of
            > failure.  If a test or project manager has to guarantee a tested product, it
            > is safer (and faster) to go the manual road.

            Rarely if ever true. More likely is that tests will be skipped in the
            interest of saving time.

            >       One of the solutions is the one raised already by Erik Meade, "Add
            > an acceptance testing task to every story".  Give a healthy estimate for it,
            > and argue that it is the only sensible way for a customer to ensure that
            > they can check that their app works over time, with the cost effectiveness
            > coming in the long term.

            Yes. Even better is to automate simply and incrementally, so that
            estimates for automating the tests are low, but honest. I believe that
            an incremental development project that delivers software to users
            every couple of weeks will benefit from automated acceptance tests
            within a month or so, because fewer defects will come back from manual
            users or testers, resulting in less disruption of the customer's
            plans.

            >       I would argue that the automation team needs to be a closely managed
            > sub-project, with staff dedicated just to the automation. with a good
            > people-person manager to liase with the customers.  It is too easy for a
            > developer to decide that their priorities lie in polishing the automation
            > programming, rather than working on the real app, or for a tester to decide
            > that they will learn to program by overworking the automation.

            I recognize this fear but do not see it coming to pass in real life.
            I've never seen an integrated team build too much test automation:
            there is too much pressure to get features done for the customer. I
            have often seen a separate QA organization be so busy building
            automation tools that they are too busy to do any actual testing.

            I promised a general algorithm for automating acceptance tests:

            1. For each story, get a _definition_ of what needs to be tested from
            the customer. This will typically be a set of inputs and corresponding
            "correct" outputs.

            2. Have the programmers task and build automated acceptance tests for
            these inputs and outputs, for each story. Do this in the simplest
            possible way, which at first will probably consist of writing code in
            your unit testing framework (typically jUnit) that sets up the inputs,
            calls the system code, checks the outputs.

            3. As these tests grow, refactor them. Note that the setup code
            invariably looks like
            issue this or that input transaction with these values
            issue another input transaction
            ...
            call one of a very few system entry points
            check this or that output value with this provided value
            check another output value with this other provided value
            ...

            Refactor in the direction of simple text scripts. Perhaps
            comma-separated value strings like this:
            transaction1,valuet11,valuet12,...
            transaction2,valuet21,valuet22
            ...
            entrycall1
            compare,outputname1,expectedresult1
            compare,outputname2,expectedresult2
            ...

            With very little refactoring, you'll have a simple language for
            automating tests.

            Ron Jeffries
            www.XProgramming.com
            "How do I get to XP?" "Practice, man, practice."
          • Ron Jeffries
            ... Separate teams are rarely if ever faster, and rarely more efficient than integrated teams made up of people of equivalent skills. Ron Jeffries
            Message 5 of 25 , Nov 14, 2001
            View Source
            • 0 Attachment
              Around Wednesday, November 14, 2001, 2:15:23 AM, STEURS Stefan wrote:

              > Why does one separate things out in separate teams?  Know that is a
              > universal question if there has ever been one.  Size of the project may be
              > one, specialised knowledge may be one, number of projects to be covered may
              > be one (generalised testing frameworks), parallel rather than serialised
              > work/activities may be one, legal obligations may be one, contractual
              > obligations may be one, etc.  Sometimes it is just the opposite of what you
              > say, it is risk-reducing, more efficient, and faster.

              Separate teams are rarely if ever faster, and rarely more efficient
              than integrated teams made up of people of equivalent skills.

              Ron Jeffries
              www.XProgramming.com
              Wisdom begins when we discover the difference between
              "That makes no sense" and "I don't understand". --Mary Doria Russell
            • STEURS Stefan
              ... Give me an example of a tool that allows to write tests at the GUI interface level, that supports a wide variety of programming languages and 4th
              Message 6 of 25 , Nov 14, 2001
              View Source
              • 0 Attachment
                > -----Original Message-----
                > From: Ron Jeffries [mailto:ronjeffries@...]
                > Sent: Wednesday, November 14, 2001 10:18 AM
                > To: agile-testing@yahoogroups.com
                > Subject: Re: [agile-testing] More considerations for not automating
                >
                >
                > Around Wednesday, November 14, 2001, 12:32:42 AM, Petersen,
                > Erik wrote:
                >
                > > Some other reasons for low automation takeup. 
                >
                > I'll answer these as if Erik thinks the questions are valid. I suspect
                > he is just reporting reasons that people give, not reasons that he
                > personally believes.
                >
                > > 1) The software is expensive, typically too expensive for a
                > single project
                > > to afford
                >
                > No real need for expensive software. Tests can be written using jUnit
                > or equivalent, or with a simple scripting language (all free), or with
                > custom code in the language of the application.

                Give me an example of a tool that allows to write tests at the GUI interface
                level, that supports a wide variety of programming languages and 4th
                generation tools (a la PowerBuilder), that has functions built in for using
                a data driven approach, that has an efficient test logging/results reporting
                interface, that has an elegant debugger, that can be used by people with
                limited development background, that allows to compare results against
                database contents, that performs event synchronisation, etc.

                There are different tool needs at different levels of testing. jUnit,
                PerlUnit, CppUnit, etc. are all fine tools at the developer level, but I
                don't think you will want to tell your customers to write their tests in
                jUnit.
                >
                > > 2) It does take longer to automate.  If the testing project
                > timeframe is
                > > tight, then automation can be eliminated immediately.
                >
                > I suggest that it rarely takes longer to automate a test than it does
                > to do it manually two or three times.

                It is the wrong argument! First, after many reports I've read about test
                automation it is suggested that more often it takes 10 test runs than 2 or 3
                times. 2 or 3 times is marketing pitch. Even so, it doesn't even matter
                that much. Test automation allows to shorten the product development cycle
                because it shortens test execution. You don't even have to automate all
                your tests, you can automate those which have the most earned value first.
                There are tests that are impossible to automate and there are tests that are
                impossible to do manually.

                If the product development cycle is too short, whatever the reason, then
                test automation will always be difficult.
                >
                > > 3  Automation paradox: Any rapid development may never be
                > stable enough to
                > > justify the time to automate it, until later releases.  The cost of
                > > automation and time required may prevent it being done in a
                > later release,
                > > especially if the life of the product being tested is uncertain.  
                >
                > I suspect that if automated testing is not justifiable on the basis of
                > instability, either you're not building user-identifiable business
                > value, or you aren't doing automation correctly. (I acknowledge that
                > the latter isn't entirely helpful because it doesn't say _how_ to do
                > it "correctly". I'll give a general algorithm below.)

                I mostly agree with what Ron says here. Although there are one shot
                projects which not very complex, short term, etc for which test automation
                would make it too costly or too late. Take for instance a one-off database
                conversion to port data from one DBMS to another. It's not something you
                will do twice.
                >
                > > 4) Any automation is a complex mini-project that comes with
                > a risk of
                > > failure.  If a test or project manager has to guarantee a
                > tested product, it
                > > is safer (and faster) to go the manual road.
                >
                > Rarely if ever true. More likely is that tests will be skipped in the
                > interest of saving time.

                It's usually always a mixture so it's not a good point of discussion. You
                will never have total automation so the risk is low.
                >
                > >       One of the solutions is the one raised already by
                > Erik Meade, "Add
                > > an acceptance testing task to every story".  Give a healthy
                > estimate for it,
                > > and argue that it is the only sensible way for a customer
                > to ensure that
                > > they can check that their app works over time, with the
                > cost effectiveness
                > > coming in the long term.
                >
                > Yes. Even better is to automate simply and incrementally, so that
                > estimates for automating the tests are low, but honest. I believe that
                > an incremental development project that delivers software to users
                > every couple of weeks will benefit from automated acceptance tests
                > within a month or so, because fewer defects will come back from manual
                > users or testers, resulting in less disruption of the customer's
                > plans.

                Good test automation is not simple but it can be done incrementally.
                Unfortunately you have to build framework function for your tests upfront,
                the kind of stuff all your tests will use like acquiring test data, logging,
                application start-up/terminate, configuration stuff, etc. Most of the time
                however you can apply this kind of generic framework stuff to multiple
                projects so the cost becomes smaller as the number of projects increases.
                The first one has to pay the full price though.
                >
                > >       I would argue that the automation team needs to be a
                > closely managed
                > > sub-project, with staff dedicated just to the automation.
                > with a good
                > > people-person manager to liase with the customers.  It is
                > too easy for a
                > > developer to decide that their priorities lie in polishing
                > the automation
                > > programming, rather than working on the real app, or for a
                > tester to decide
                > > that they will learn to program by overworking the automation.
                >
                > I recognize this fear but do not see it coming to pass in real life.
                > I've never seen an integrated team build too much test automation:
                > there is too much pressure to get features done for the customer. I
                > have often seen a separate QA organization be so busy building
                > automation tools that they are too busy to do any actual testing.

                I've seen both. I know Ron has frictions with QA organisations, it gets his
                hair in a mess. QA stands for Quality Assurance, and many people think that
                QA's primary job is testing. Poor people. QA is not QC and QC is not
                testing. Testing is a subset of QC, and QC is a subset of QA. If QA is too
                busy building test tools, they are not even worth the name QA.

                Anyway, I've seen Test Teams do very good work indeed.
                >
                > I promised a general algorithm for automating acceptance tests:
                >
                > 1. For each story, get a _definition_ of what needs to be tested from
                > the customer. This will typically be a set of inputs and corresponding
                > "correct" outputs.

                In fact, the input should be the business process description. You don't
                only need what in and what out but also definitions of user/security
                profiles (who is allowed to do what), production databases used as a
                reference, timing (when), raw input material (to see if they convert this
                input material to the input that the tool accepts - engine trust in Newtons
                or Pascal?), ...

                If the customers have to do the tests, they should also create them. The
                developers don't know the business in most of the cases and they make more
                than just once in a while the same logical mistakes in interpreting customer
                data as interpreting customer requirements.
                >
                > 2. Have the programmers task and build automated acceptance tests for
                > these inputs and outputs, for each story. Do this in the simplest
                > possible way, which at first will probably consist of writing code in
                > your unit testing framework (typically jUnit) that sets up the inputs,
                > calls the system code, checks the outputs.

                But this means you are not doing any tests at the user level which is at the
                HMI level. I wouldn't call this acceptance tests, they are just functional
                tests with a customer flavour.

                I don't see how acceptance tests can be done out of a business context.
                >
                > 3. As these tests grow, refactor them. Note that the setup code
                > invariably looks like
                > issue this or that input transaction with these values
                > issue another input transaction
                > ...
                > call one of a very few system entry points
                > check this or that output value with this provided value
                > check another output value with this other provided value
                > ...
                >
                > Refactor in the direction of simple text scripts. Perhaps
                > comma-separated value strings like this:
                > transaction1,valuet11,valuet12,...
                > transaction2,valuet21,valuet22
                > ...
                > entrycall1
                > compare,outputname1,expectedresult1
                > compare,outputname2,expectedresult2
                > ...

                The refactoring can be very costly! It's better to start immediately from
                day one with a data/keyword driven framework based approach. It's a bit
                slower but it's much cheaper over-all.
                >
                > With very little refactoring, you'll have a simple language for
                > automating tests.

                Very little refactoring? If you don't do it right from day one, you loose
                time and money. Refactoring tests starts with good test design.
                Refactoring poor tests still makes you end up with poor tests.
                >
                > Ron Jeffries

                Regards, Stefan Steurs.
              • STEURS Stefan
                Small teams of a couple of persons rarely do very big projects of hundreths of manyears. They are rarely involved in safety or mission critical projects. When
                Message 7 of 25 , Nov 14, 2001
                View Source
                • 0 Attachment
                  Small teams of a couple of persons rarely do very big projects of hundreths
                  of manyears. They are rarely involved in safety or mission critical
                  projects.

                  When it comes to software development, "Fast" is not the only quality we are
                  after.

                  Depending on your quality attributes and context you should make careful
                  decisions. There is no magical recipe.

                  Regards, Stefan Steurs.

                  > -----Original Message-----
                  > From: Ron Jeffries [mailto:ronjeffries@...]
                  > Sent: Wednesday, November 14, 2001 10:50 AM
                  > To: agile-testing@yahoogroups.com
                  > Subject: Re: [agile-testing] More considerations for not automating
                  >
                  >
                  > Around Wednesday, November 14, 2001, 2:15:23 AM, STEURS Stefan wrote:
                  >
                  > > Why does one separate things out in separate teams?  Know that is a
                  > > universal question if there has ever been one.  Size of the
                  > project may be
                  > > one, specialised knowledge may be one, number of projects
                  > to be covered may
                  > > be one (generalised testing frameworks), parallel rather
                  > than serialised
                  > > work/activities may be one, legal obligations may be one,
                  > contractual
                  > > obligations may be one, etc.  Sometimes it is just the
                  > opposite of what you
                  > > say, it is risk-reducing, more efficient, and faster.
                  >
                  > Separate teams are rarely if ever faster, and rarely more efficient
                  > than integrated teams made up of people of equivalent skills.
                  >
                  > Ron Jeffries
                  > www.XProgramming.com
                  > Wisdom begins when we discover the difference between
                  > "That makes no sense" and "I don't understand". --Mary Doria Russell
                  >
                  >
                  > ------------------------ Yahoo! Groups Sponsor
                  > ---------------------~-->
                  > Universal Inkjet Refill Kit $29.95
                  > Refill any ink cartridge for less!
                  > Includes black and color ink.
                  > http://us.click.yahoo.com/1_Y1qC/MkNDAA/ySSFAA/03wwlB/TM
                  > --------------------------------------------------------------
                  > -------~->
                  >
                  > To unsubscribe from this group, send an email to:
                  > agile-testing-unsubscribe@yahoogroups.com
                  >
                  >
                  >
                  > Your use of Yahoo! Groups is subject to
                  http://docs.yahoo.com/info/terms/
                • Ron Jeffries
                  ... I don t know of one. And we don t need all that to write effective ... Don t write many tests at the GUI level: keep the GUI very thin so that these can be
                  Message 8 of 25 , Nov 14, 2001
                  View Source
                  • 0 Attachment
                    Around Wednesday, November 14, 2001, 7:52:21 AM, STEURS Stefan wrote:

                    > Give me an example of a tool that allows to write tests at the GUI interface
                    > level, that supports a wide variety of programming languages and 4th
                    > generation tools (a la PowerBuilder), that has functions built in for using
                    > a data driven approach, that has an efficient test logging/results reporting
                    > interface, that has an elegant debugger, that can be used by people with
                    > limited development background, that allows to compare results against
                    > database contents, that performs event synchronisation, etc.

                    I don't know of one. And we don't need all that to write effective
                    tests. Here's a bit about why and how:

                    > tests at the GUI interface

                    Don't write many tests at the GUI level: keep the GUI very thin so
                    that these can be minimized. Use an available GUI-testing tool for the
                    rest. Do not expect this tool to solve other problems.

                    > wide variety of programming languages

                    Scripting languages support most programming languages. Most
                    projects, in addition, use only one programming language. They don't
                    need a tool that supports the language they use. And there is no
                    reason to use the same testing tool for different languages, though it
                    would sometimes be nice.

                    > data driven approach

                    All the programming languages that I know of have input and output
                    from files. This is sufficient to write data driven tests.

                    > test logging/results reporting interface

                    Use I/O to write simple logging records. Extract and report with
                    your scripting language. Use conventional language compiler error
                    reporting as the model: extract from the error printout into a list.

                    > that has an elegant debugger

                    Write tests that don't need debugging. Or write them test first if
                    you have some valid reason for writing complex tests.

                    > people with limited development background

                    There is no tool that will substitute for people who are under
                    qualified. Upgrade the people through training, or when really
                    necessary, through replacement.

                    > compare results against database contents

                    Print with any reporting tool. Compare the files for identity.

                    > event synchronisation

                    I don't think I understand this question. Can programming languages
                    do this? Use them?

                    Ron Jeffries
                    www.XProgramming.com
                    You keep using that word. I do not think it means what you think it means.
                    --Inigo Montoya
                  • Ron Jeffries
                    ... Right. Customers, in my model, must define tests. If tools are available, they can type them in. Most testing, in my experience, can be done with rather
                    Message 9 of 25 , Nov 14, 2001
                    View Source
                    • 0 Attachment
                      Around Wednesday, November 14, 2001, 7:52:21 AM, STEURS Stefan wrote:

                      > There are different tool needs at different levels of testing.  jUnit,
                      > PerlUnit, CppUnit, etc. are all fine tools at the developer level, but I
                      > don't think you will want to tell your customers to write their tests in
                      > jUnit.

                      Right. Customers, in my model, must define tests. If tools are
                      available, they can type them in. Most testing, in my experience, can
                      be done with rather simple scripting. Customers can readily write
                      sequential scripts.

                      Often testing can be done by using automation tools, letting customers
                      type data for testing into spreadsheets, and automatically extracting
                      that data and driving the system, exporting the results back to the
                      spreadsheet or elsewhere for reporting.

                      Ron Jeffries
                      www.XProgramming.com
                      Fear is the mindkiller. --Bene Gesserit Litany Against Fear
                    • Ron Jeffries
                      ... I ve never seen a case where doing good testing was harder than doing the programming that needed it. I ve seen cases where people MADE it hard, and many
                      Message 10 of 25 , Nov 14, 2001
                      View Source
                      • 0 Attachment
                        Around Wednesday, November 14, 2001, 7:52:21 AM, STEURS Stefan wrote:

                        > If the product development cycle is too short, whatever the reason, then
                        > test automation will always be difficult.

                        I've never seen a case where doing good testing was harder than doing
                        the programming that needed it. I've seen cases where people MADE it
                        hard, and many more cases where they ASSUMED it was too hard.

                        Ron Jeffries
                        www.XProgramming.com
                        Life tough is. Then die you do. --Yoda (personal communication)
                      • Ron Jeffries
                        ... It certainly can be done incrementally. It can also be done simply. ... Untrue. The agile methods (and all the incremental methods) show that software
                        Message 11 of 25 , Nov 14, 2001
                        View Source
                        • 0 Attachment
                          Around Wednesday, November 14, 2001, 7:52:21 AM, STEURS Stefan wrote:

                          > Good test automation is not simple but it can be done incrementally.

                          It certainly can be done incrementally. It can also be done simply.

                          > Unfortunately you have to build framework function for your tests upfront,
                          > the kind of stuff all your tests will use like acquiring test data, logging,
                          > application start-up/terminate, configuration stuff, etc. 

                          Untrue. The agile methods (and all the incremental methods) show that
                          software applications can be written incrementally, delivering
                          business value as they do. Testing is merely an example.

                          It is quite possible to build framework as you go in building
                          applications. The same is true of testing frameworks.

                          > Most of the time
                          > however you can apply this kind of generic framework stuff to multiple
                          > projects so the cost becomes smaller as the number of projects increases.
                          > The first one has to pay the full price though.

                          If the cost of testing is higher than justified for a single
                          application, it's my opinion that the testing is not being done
                          correctly. This includes automation.

                          With the greatest love and respect, I'm starting to think we need to
                          teach an XP Immersion class for testers wanting to automate.

                          Ron Jeffries
                          www.XProgramming.com
                          Wisdom begins when we discover the difference between
                          "That makes no sense" and "I don't understand". --Mary Doria Russell
                        • Ron Jeffries
                          ... According to Jim Highsmith, something like 80% of all software projects are done with fewer than 20 people. ... No one suggested it was. ... Yes, you
                          Message 12 of 25 , Nov 14, 2001
                          View Source
                          • 0 Attachment
                            Around Wednesday, November 14, 2001, 8:20:00 AM, STEURS Stefan wrote:

                            > Small teams of a couple of persons rarely do very big projects of hundreths
                            > of manyears.  They are rarely involved in safety or mission critical
                            > projects.

                            According to Jim Highsmith, something like 80% of all software
                            projects are done with fewer than 20 people.

                            > When it comes to software development, "Fast" is not the only quality we are
                            > after.

                            No one suggested it was.

                            > Depending on your quality attributes and context you should make careful
                            > decisions. 

                            Yes, you should.

                            > There is no magical recipe.

                            No one suggested there was.

                            Ron Jeffries
                            www.XProgramming.com
                            Sigs are like I Ching or Tarot. They don't mean anything,
                            but sometimes if you think about them you'll get a useful idea.
                          • Bret Pettichord
                            ... I d be interested. I know a lot about automation, but would like to learn the XP perspective on it. Bret
                            Message 13 of 25 , Nov 14, 2001
                            View Source
                            • 0 Attachment
                              At 07:56 AM 11/14/01 , Ron Jeffries wrote:
                              >With the greatest love and respect, I'm starting to think we need to
                              >teach an XP Immersion class for testers wanting to automate.

                              I'd be interested. I know a lot about automation, but would like to learn
                              the XP perspective on it.

                              Bret
                              __________________________________________________________________
                              Bret Pettichord
                              Editor, Software Testing Hotlist - http://www.testinghotlist.com
                              Consultant, Pettichord Consulting LLC - http://www.pettichord.com

                              Lessons Learned in Software Testing: A Context-Driven Approach
                              By Cem Kaner, James Bach & Bret Pettichord (Wiley, Dec 2001)
                              http://www.testinglessons.com
                            • Lisa Crispin
                              If you have appropriate lightweight test tools and framework, I think it is always feasible to automate testing at least the major functionality for any
                              Message 14 of 25 , Nov 14, 2001
                              View Source
                              • 0 Attachment
                                If you have appropriate lightweight test tools and framework, I think
                                it is always feasible to automate testing at least the major
                                functionality for any application. Maybe not through the GUI, if it
                                is not a web app. I haven't found an XP project yet where the code
                                was too unstable to maintain automated tests. If the test fails we
                                look to see what changed and we fix the test. If it's too hard or too
                                expensive to automate ANY testing, maybe you are using the wrong tools
                                or the wrong approach?

                                That said, I don't think ALL testing should be automated, but I think
                                there needs to be SOME automated testing for every application, not
                                only to verify the most critical business functionality for that
                                release, but to use as regression tests for future releases.
                                -- Lisa

                                --- In agile-testing@y..., "Petersen, Erik" <PETERSEE@a...> wrote:
                                > Some other reasons for low automation takeup.
                                > 1) The software is expensive, typically too expensive for a single
                                project
                                > to afford
                                > 2) It does take longer to automate. If the testing project
                                timeframe is
                                > tight, then automation can be eliminated immediately.
                                > 3 Automation paradox: Any rapid development may never be stable
                                enough to
                                > justify the time to automate it, until later releases. The cost of
                                > automation and time required may prevent it being done in a later
                                release,
                                > especially if the life of the product being tested is uncertain.
                                > 4) Any automation is a complex mini-project that comes with a risk
                                of
                                > failure. If a test or project manager has to guarantee a tested
                                product, it
                                > is safer (and faster) to go the manual road.
                                >
                                > One of the solutions is the one raised already by Erik Meade,
                                "Add
                                > an acceptance testing task to every story". Give a healthy estimate
                                for it,
                                > and argue that it is the only sensible way for a customer to ensure
                                that
                                > they can check that their app works over time, with the cost
                                effectiveness
                                > coming in the long term.
                                > I would argue that the automation team needs to be a closely
                                managed
                                > sub-project, with staff dedicated just to the automation. with a
                                good
                                > people-person manager to liase with the customers. It is too easy
                                for a
                                > developer to decide that their priorities lie in polishing the
                                automation
                                > programming, rather than working on the real app, or for a tester to
                                decide
                                > that they will learn to program by overworking the automation.
                                > cheers,
                                > Erik
                                >
                                > Software Testing Consultant
                                > ANZ Banking Group
                                > Australia
                                >
                                >
                                > > -----Original Message-----
                                > > From: Lisa Crispin [mailto:lisa.crispin@a...]
                                > > Sent: 09 November 2001 05:28
                                > > To: agile-testing@y...
                                > > Subject: [agile-testing] Re: testing non-web GUI
                                > >
                                > >
                                > > My experience is like Ron's in that many organizations I'm
                                familiar
                                > > with do not automate at all, or very little, possibly for
                                > > load testing
                                > > and nothing else. This includes organizations who say they are
                                doing
                                > > XP. Some potential reason:
                                > >
                                > > 1. A lot of testers do not have a programming backround so it can
                                be
                                > > difficult for them to learn how to create automated test scripts
                                and
                                > > they have a lot of fear about it.
                                > > 2. Capture/playback tools alone don't help, see 1.
                                > > 3. Lack of support from management and even from development:
                                "just
                                > > get the testing done so we can launch". This is worst when you
                                have
                                > > testing on the "back end".
                                > > 4. Lack of understanding of the value of automating tests other
                                than
                                > > unit tests, even among developers on XP projects.
                                > >
                                > > Some things that can be done about it (although I admit to
                                struggling
                                > > in this area myself):
                                > > 1. Put testers and developers on one team
                                > > 2. Make the whole team responsible for executing and for
                                automating
                                > > acceptance tests
                                > > 3. For each release, if you have any automated tests at all,
                                > > document
                                > > as best you can what value they added (time saved, bugs found)
                                > > 4. The getting support from management and development one is
                                hard,
                                > > but find someone who sees the value of test automation and
                                > > get them to
                                > > help champion the cause
                                > >
                                > > Other suggestions? Because these are working some for me, but
                                > > sometimes I get discouraged fighting the same battles all the
                                time.
                                > > -- Lisa
                                > >
                                > <snip>
                                > > > I might mention that in the projects I visit, even the ones that
                                > > have
                                > > > large and presumably competent QA departments, few (usually
                                zero)
                                > > > tests are automated. That seems to be very much the wrong
                                proportion
                                > > > to me. What do folks here think?
                                > > >
                                > > > What is the cost of automating a test that doesn't need it? What
                                is
                                > > > the cost of NOT automating a test that DOES need it?
                                > > >
                                > > > Ron Jeffries
                                > > > www.XProgramming.com
                                > > > Life tough is. Then die you do. --Yoda (personal communication)
                                > >
                              • Lisa Crispin
                                ... effort, ... steer their ... We gain a much higher quality product. I have done XP projects both ways, and the way Bill describes works a whole lot
                                Message 15 of 25 , Nov 14, 2001
                                View Source
                                • 0 Attachment
                                  > But what happens if we make this a visible part of the team's
                                  effort,
                                  > asking the programmers to handle the tasks, ask the customer to
                                  steer their
                                  > prioritization, and having our testing experts helping both them?
                                  >
                                  > What do we lose if we do this? What do we gain?
                                  >
                                  > Best,
                                  > Bill

                                  We gain a much higher quality product. I have done "XP" projects both
                                  ways, and the way Bill describes works a whole lot better.
                                  -- Lisa
                                • Lisa Crispin
                                  ... hundreths ... It s dangerous to make this kind of blanket statement. What are you basing this on? There are some giant companies, some in the telecom
                                  Message 16 of 25 , Nov 14, 2001
                                  View Source
                                  • 0 Attachment
                                    --- In agile-testing@y..., STEURS Stefan <stefan.steurs@e...> wrote:
                                    > Small teams of a couple of persons rarely do very big projects of

                                    hundreths
                                    > of manyears. They are rarely involved in safety or mission critical
                                    > projects.

                                    It's dangerous to make this kind of blanket statement. What are you
                                    basing this on? There are some giant companies, some in the telecom
                                    space, using XP now. I know of at least one planning to use it for
                                    all their development projects, which might include the kinds of big
                                    projects you mention. Maybe a project of hundreds of personyears
                                    would be divided into lots of small subteams. Where I work, we do
                                    this on a smaller scale. It's possible.
                                  • Lisa Crispin
                                    ... reason, then ... doing ... Short releases aren t any harder for test automation than longer ones, because there is less to test.
                                    Message 17 of 25 , Nov 14, 2001
                                    View Source
                                    • 0 Attachment
                                      --- In agile-testing@y..., Ron Jeffries <ronjeffries@a...> wrote:
                                      > Around Wednesday, November 14, 2001, 7:52:21 AM, STEURS Stefan
                                      wrote:
                                      >
                                      > > If the product development cycle is too short, whatever the
                                      reason, then
                                      > > test automation will always be difficult.
                                      >
                                      > I've never seen a case where doing good testing was harder than
                                      doing
                                      > the programming that needed it. I've seen cases where people MADE it
                                      > hard, and many more cases where they ASSUMED it was too hard.
                                      >
                                      > Ron Jeffries
                                      > www.XProgramming.com
                                      > Life tough is. Then die you do. --Yoda (personal communication)

                                      Short releases aren't any harder for test automation than longer
                                      ones, because there is less to test.
                                    • Lisa Crispin
                                      ... HEY! HEY! Let s not make generalizations about testers here! Many of us are out here automating, thank you very much! We are just continually looking
                                      Message 18 of 25 , Nov 14, 2001
                                      View Source
                                      • 0 Attachment
                                        >
                                        > With the greatest love and respect, I'm starting to think we need to
                                        > teach an XP Immersion class for testers wanting to automate.
                                        >
                                        > Ron Jeffries
                                        > www.XProgramming.com
                                        > Wisdom begins when we discover the difference between
                                        > "That makes no sense" and "I don't understand". --Mary Doria Russell

                                        HEY! HEY! Let's not make generalizations about testers here! Many
                                        of us are out here automating, thank you very much! We are just
                                        continually looking for ways to do it better, just as our team looks
                                        for ways to code better and all that stuff.
                                        -- Lisa
                                      • Ron Jeffries
                                        ... Since I m at an Immersion now, I ve already raised the issue with the [other?] Powers. We ll see if we can figure out something that would be useful and
                                        Message 19 of 25 , Nov 14, 2001
                                        View Source
                                        • 0 Attachment
                                          Around Wednesday, November 14, 2001, 11:25:38 AM, Bret Pettichord wrote:

                                          > At 07:56 AM 11/14/01 , Ron Jeffries wrote:
                                          >>With the greatest love and respect, I'm starting to think we need to
                                          >>teach an XP Immersion class for testers wanting to automate.

                                          > I'd be interested. I know a lot about automation, but would like to learn
                                          > the XP perspective on it.

                                          Since I'm at an Immersion now, I've already raised the issue with the
                                          [other?] Powers. We'll see if we can figure out something that would
                                          be useful and saleable. Thanks for your interest!

                                          Ron Jeffries
                                          www.XProgramming.com
                                          You keep using that word. I do not think it means what you think it means.
                                          --Inigo Montoya
                                        • Ron Jeffries
                                          ... I m not generalizing about testers. I m observing that there are lots of testers who think automation is really hard, and that therefore there might be a
                                          Message 20 of 25 , Nov 14, 2001
                                          View Source
                                          • 0 Attachment
                                            Around Wednesday, November 14, 2001, 1:32:44 PM, Lisa Crispin wrote:

                                            >> With the greatest love and respect, I'm starting to think we need to
                                            >> teach an XP Immersion class for testers wanting to automate.
                                            >>
                                            >> Ron Jeffries
                                            >> www.XProgramming.com
                                            >> Wisdom begins when we discover the difference between
                                            >> "That makes no sense" and "I don't understand". --Mary Doria Russell

                                            > HEY! HEY!  Let's not make generalizations about testers here!  Many
                                            > of us are out here automating, thank you very much!  We are just
                                            > continually looking for ways to do it better, just as our team looks
                                            > for ways to code better and all that stuff.

                                            I'm not generalizing about testers. I'm observing that there are lots
                                            of testers who think automation is really hard, and that therefore
                                            there might be a market for a course. I didn't say anything bad about
                                            testers. Just because you're paranoid doesn't mean people are out to
                                            get you. Now get back to writing that book.

                                            Ron Jeffries
                                            www.XProgramming.com
                                            I must create a system, or be enslaved by another man's;
                                            I will not reason and compare; my business is to create. --William Blake
                                          • Lisa Crispin
                                            ... Many ... looks ... lots ... about ... You are making me laugh so much I am starting to snort. My coworkers believe the bugs have put me around the bend.
                                            Message 21 of 25 , Nov 14, 2001
                                            View Source
                                            • 0 Attachment
                                              >
                                              > > HEY! HEY!  Let's not make generalizations about testers here! 
                                              Many
                                              > > of us are out here automating, thank you very much!  We are just
                                              > > continually looking for ways to do it better, just as our team
                                              looks
                                              > > for ways to code better and all that stuff.
                                              >
                                              > I'm not generalizing about testers. I'm observing that there are
                                              lots
                                              > of testers who think automation is really hard, and that therefore
                                              > there might be a market for a course. I didn't say anything bad
                                              about
                                              > testers. Just because you're paranoid doesn't mean people are out to
                                              > get you. Now get back to writing that book.
                                              >
                                              > Ron Jeffries
                                              You are making me laugh so much I am starting to snort. My coworkers
                                              believe the bugs have put me around the bend. You are correct, there
                                              is a huge market for your course. I would go too and I'd try to get
                                              all the programmers on my team to go.
                                              -- Lisa
                                            • Ron Jeffries
                                              ... Actually I was thinking of asking you to help teach it ... Ron Jeffries www.XProgramming.com Accroches-toi a ton reve. --ELO
                                              Message 22 of 25 , Nov 14, 2001
                                              View Source
                                              • 0 Attachment
                                                Around Wednesday, November 14, 2001, 2:13:00 PM, Lisa Crispin wrote:

                                                > You are making me laugh so much I am starting to snort.  My coworkers
                                                > believe the bugs have put me around the bend.  You are correct, there
                                                > is a huge market for your course.  I would go too and I'd try to get
                                                > all the programmers on my team to go.

                                                Actually I was thinking of asking you to help teach it ...

                                                Ron Jeffries
                                                www.XProgramming.com
                                                Accroches-toi a ton reve. --ELO
                                              • STEURS Stefan
                                                ... So? 80% of the work is done in 20% of the time but it still doesn t mean that if you cut your scope to 80% that you implement software 5 times as fast.
                                                Message 23 of 25 , Nov 14, 2001
                                                View Source
                                                • 0 Attachment
                                                  > -----Original Message-----
                                                  > From: Ron Jeffries [mailto:ronjeffries@...]
                                                  > Sent: Wednesday, November 14, 2001 3:55 PM
                                                  > To: agile-testing@yahoogroups.com
                                                  > Subject: Re: [agile-testing] More considerations for not automating
                                                  >
                                                  >
                                                  > Around Wednesday, November 14, 2001, 8:20:00 AM, STEURS Stefan wrote:
                                                  >
                                                  > > Small teams of a couple of persons rarely do very big
                                                  > projects of hundreths
                                                  > > of manyears.  They are rarely involved in safety or mission critical
                                                  > > projects.
                                                  >
                                                  > According to Jim Highsmith, something like 80% of all software
                                                  > projects are done with fewer than 20 people.

                                                  So? 80% of the work is done in 20% of the time but it still doesn't mean
                                                  that if you cut your scope to 80% that you implement software 5 times as
                                                  fast.
                                                  >
                                                  > > There is no magical recipe.
                                                  >
                                                  > No one suggested there was.

                                                  You make it sound as if XP is.
                                                  >
                                                  > Ron Jeffries
                                                  > www.XProgramming.com
                                                  > Sigs are like I Ching or Tarot. They don't mean anything,
                                                  > but sometimes if you think about them you'll get a useful idea.

                                                  Regards, Stefan Steurs.
                                                • STEURS Stefan
                                                  I m trying to understand how. Making the transition isn t easy but I m sure that we need more scalability in XP to make it more apt to deal with different
                                                  Message 24 of 25 , Nov 14, 2001
                                                  View Source
                                                  • 0 Attachment
                                                    I'm trying to understand how. Making the transition isn't easy but I'm sure
                                                    that we need more scalability in XP to make it more apt to deal with
                                                    different contexts.

                                                    Stefan

                                                    > -----Original Message-----
                                                    > From: Lisa Crispin [mailto:lisa.crispin@...]
                                                    > Sent: Wednesday, November 14, 2001 7:25 PM
                                                    > To: agile-testing@yahoogroups.com
                                                    > Subject: [agile-testing] Re: More considerations for not automating
                                                    >
                                                    >
                                                    > --- In agile-testing@y..., STEURS Stefan <stefan.steurs@e...> wrote:
                                                    > > Small teams of a couple of persons rarely do very big projects of
                                                    >
                                                    > hundreths
                                                    > > of manyears. They are rarely involved in safety or mission critical
                                                    > > projects.
                                                    >
                                                    > It's dangerous to make this kind of blanket statement. What are you
                                                    > basing this on? There are some giant companies, some in the telecom
                                                    > space, using XP now. I know of at least one planning to use it for
                                                    > all their development projects, which might include the kinds of big
                                                    > projects you mention. Maybe a project of hundreds of personyears
                                                    > would be divided into lots of small subteams. Where I work, we do
                                                    > this on a smaller scale. It's possible.
                                                    >
                                                    >
                                                    > ------------------------ Yahoo! Groups Sponsor
                                                    > ---------------------~-->
                                                    > Universal Inkjet Refill Kit $29.95
                                                    > Refill any ink cartridge for less!
                                                    > Includes black and color ink.
                                                    > http://us.click.yahoo.com/Vv.L9D/MkNDAA/ySSFAA/03wwlB/TM
                                                    > --------------------------------------------------------------
                                                    > -------~->
                                                    >
                                                    > To unsubscribe from this group, send an email to:
                                                    > agile-testing-unsubscribe@yahoogroups.com
                                                    >
                                                    >
                                                    >
                                                    > Your use of Yahoo! Groups is subject to
                                                    http://docs.yahoo.com/info/terms/
                                                  • Lisa Crispin
                                                    ... coworkers ... there ... get ... I would love to! -- Lisa
                                                    Message 25 of 25 , Nov 15, 2001
                                                    View Source
                                                    • 0 Attachment
                                                      --- In agile-testing@y..., Ron Jeffries <ronjeffries@a...> wrote:
                                                      > Around Wednesday, November 14, 2001, 2:13:00 PM, Lisa Crispin wrote:
                                                      >
                                                      > > You are making me laugh so much I am starting to snort.  My
                                                      coworkers
                                                      > > believe the bugs have put me around the bend.  You are correct,
                                                      there
                                                      > > is a huge market for your course.  I would go too and I'd try to
                                                      get
                                                      > > all the programmers on my team to go.
                                                      >
                                                      > Actually I was thinking of asking you to help teach it ...
                                                      >
                                                      > Ron Jeffries
                                                      > www.XProgramming.com
                                                      > Accroches-toi a ton reve. --ELO

                                                      I would love to!
                                                      -- Lisa
                                                    Your message has been successfully submitted and would be delivered to recipients shortly.