Loading ...
Sorry, an error occurred while loading the content.
 

Is it better to have bad tests than no tests? Understanding growth in a project

Expand Messages
  • Pat Maddox
    A friend on another list asked are bad tests better than no tests? Here s my theory. Ranked, in ascending order of desirability: No tests Green suite, poor
    Message 1 of 14 , Feb 28, 2008
      A friend on another list asked "are bad tests better than no tests?"

      Here's my theory.

      Ranked, in ascending order of desirability:
      No tests
      Green suite, poor tests
      Red suite, poor tests
      Red suite, quality tests
      Green suite, quality tests

      A passing suite of high-quality tests gives you confidence that your
      system works well. If your tests are of low quality, then you will be
      confident, but your confidence will be misguided, and so you disrupt
      the balance of what the team knows vs what the team thinks it knows.
      The team has good values but does not understand the principles.

      A broken suite of low-quality tests sounds discouraging, but I believe
      it is a positive situation, because the team's understanding of the
      values and principles can grow at the same rate, rather than one
      dominating the other like a weed. Some teams and team members will be
      cynical, some will be eager but struggle. Most importantly though,
      they will be realistic.

      A broken suite of high-quality tests signifies a breakdown in process.
      Unlike before, the problem at this stage is no longer a matter of
      growth and understanding within the developers, it is probably
      political. They may be under tight deadlines and so have to let
      practices slip slightly, urged on by misinformed project managers.
      Removing political obstacles is difficult and requires skill in
      itself, but I think it's a tedious work compared to the personal
      growth that developers must experience.

      Willfully choosing not to write tests means you are stupid.

      Pat
    • Paul Campbell
      ... I ve seen plenty of tests that are worse than useless. A common theme in such tests is diffing (rather than interpreting) textual output that has semantic
      Message 2 of 14 , Feb 29, 2008
        --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@...>
        wrote:
        >
        > A friend on another list asked "are bad tests better than no tests?"

        I've seen plenty of tests that are worse than useless. A common theme
        in such tests is diffing (rather than interpreting) textual output
        that has semantic content. Common examples include diffing HTML,
        diffing XML, diffing SQL queries. The problem with such tests is that
        in general they only tell you when something has changed - not whether
        it works, and in my opinion that makes them worse than useless in many
        cases.

        Paul.
      • Ola Ellnestam
        ... I categorize these kind of tests under: Tests holding the code hostage As said. They are beyond useless. They stop you from making improvements and are
        Message 3 of 14 , Feb 29, 2008
          Paul Campbell wrote:
          > --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@...>
          > wrote:
          >
          >> A friend on another list asked "are bad tests better than no tests?"
          >>
          >
          > I've seen plenty of tests that are worse than useless. A common theme
          > in such tests is diffing (rather than interpreting) textual output
          > that has semantic content. Common examples include diffing HTML,
          > diffing XML, diffing SQL queries. The problem with such tests is that
          > in general they only tell you when something has changed - not whether
          > it works, and in my opinion that makes them worse than useless in many
          > cases.
          >
          >

          I categorize these kind of tests under: "Tests holding the code hostage"

          As said. They are beyond useless. They stop you from making improvements
          and are too coupled to the implementation. Not focusing at behavior at all.

          //Ola

          --
          ---------------------------------------------------------
          Ola Ellnestam
          Agical AB
          Västerlånggatan 79, 2 tr
          111 29 Stockholm, SWEDEN

          Mobile: +46-708-754000
          E-mail: ola.ellnestam@...
          Blog: http://ellnestam.wordpress.com
        • Manuel Klimek
          Hi Paul, ... Wouldn t you change those tests anyway before changing the code? Just curious, since I do those diffing tests sometimes (ever tried to interpret
          Message 4 of 14 , Feb 29, 2008
            Hi Paul,
            On Fri, Feb 29, 2008 at 3:06 PM, Paul Campbell <yahoo@...> wrote:
            > --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@...>
            > wrote:
            > >
            > > A friend on another list asked "are bad tests better than no tests?"
            >
            > I've seen plenty of tests that are worse than useless. A common theme
            > in such tests is diffing (rather than interpreting) textual output
            > that has semantic content. Common examples include diffing HTML,
            > diffing XML, diffing SQL queries. The problem with such tests is that
            > in general they only tell you when something has changed - not whether
            > it works, and in my opinion that makes them worse than useless in many
            > cases.

            Wouldn't you change those tests anyway before changing the code? Just
            curious, since I do those diffing tests sometimes (ever tried to
            "interpret" an image ;-), and I think they're really useful for me.

            Cheers,
            /Manuel

            >
            > Paul.
            >
            >



            --
            http://klimek.box4.net
          • Michael Feathers
            ... When you re refactoring or adding functionality, you do want to know when something has changed. It s not the best general testing strategy, but it has
            Message 5 of 14 , Feb 29, 2008
              Paul Campbell wrote:
              > --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@...>
              > wrote:
              >
              >> A friend on another list asked "are bad tests better than no tests?"
              >>
              >
              > I've seen plenty of tests that are worse than useless. A common theme
              > in such tests is diffing (rather than interpreting) textual output
              > that has semantic content. Common examples include diffing HTML,
              > diffing XML, diffing SQL queries. The problem with such tests is that
              > in general they only tell you when something has changed - not whether
              > it works, and in my opinion that makes them worse than useless in many
              > cases.
              When you're refactoring or adding functionality, you do want to know
              when something has changed. It's not the best general testing strategy,
              but it has its uses.

              Michael Feathers
            • Michael Feathers
              ... The hardest part of dealing with bad tests is knowing when to let go. We ve conditioned ourselves to think more tests: good , fewer tests: bad. If
              Message 6 of 14 , Feb 29, 2008
                Ola Ellnestam wrote:
                > Paul Campbell wrote:
                >
                >> --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@...>
                >> wrote:
                >>
                >>
                >>> A friend on another list asked "are bad tests better than no tests?"
                >>>
                >>>
                >> I've seen plenty of tests that are worse than useless. A common theme
                >> in such tests is diffing (rather than interpreting) textual output
                >> that has semantic content. Common examples include diffing HTML,
                >> diffing XML, diffing SQL queries. The problem with such tests is that
                >> in general they only tell you when something has changed - not whether
                >> it works, and in my opinion that makes them worse than useless in many
                >> cases.
                >>
                >>
                >>
                >
                > I categorize these kind of tests under: "Tests holding the code hostage"
                >
                > As said. They are beyond useless. They stop you from making improvements
                > and are too coupled to the implementation. Not focusing at behavior at all.
                >
                The hardest part of dealing with bad tests is knowing when to let go.
                We've conditioned ourselves to think "more tests: good", "fewer tests:
                bad." If tests are in the way, it's okay to delete them even though it
                feels wrong. If you know that you can always go back to them in version
                control, there is no downside, really, to putting some tests aside and
                writing some new ones.

                Michael Feathers
              • Kent Beck
                I wouldn t want tests that didn t help either. However, I work with a team that has extensive tests that diff PDF or HTML. They are valuable as change
                Message 7 of 14 , Feb 29, 2008
                  I wouldn't want tests that didn't help either. However, I work with a team
                  that has extensive tests that diff PDF or HTML. They are valuable as change
                  detectors. If one breaks and they didn't expect it, they know to be careful.
                  However, the team have made it easy to say, "That change is ok, this test
                  should pass now." I suppose the next step would be to do a root cause
                  analysis whenever one of the "hostage" tests breaks to see what failing unit
                  tests they should have written.

                  Regards,

                  Kent Beck
                  Three Rivers Institute

                  _____

                  From: extremeprogramming@yahoogroups.com
                  [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Ola Ellnestam
                  Sent: Friday, February 29, 2008 6:16 AM
                  To: extremeprogramming@yahoogroups.com
                  Subject: Re: [XP] Re: Is it better to have bad tests than no tests?
                  Understanding growth in a proj



                  Paul Campbell wrote:
                  > --- In extremeprogramming@ <mailto:extremeprogramming%40yahoogroups.com>
                  yahoogroups.com, "Pat Maddox" <pergesu@...>
                  > wrote:
                  >
                  >> A friend on another list asked "are bad tests better than no tests?"
                  >>
                  >
                  > I've seen plenty of tests that are worse than useless. A common theme
                  > in such tests is diffing (rather than interpreting) textual output
                  > that has semantic content. Common examples include diffing HTML,
                  > diffing XML, diffing SQL queries. The problem with such tests is that
                  > in general they only tell you when something has changed - not whether
                  > it works, and in my opinion that makes them worse than useless in many
                  > cases.
                  >
                  >

                  I categorize these kind of tests under: "Tests holding the code hostage"

                  As said. They are beyond useless. They stop you from making improvements
                  and are too coupled to the implementation. Not focusing at behavior at all.

                  //Ola

                  --
                  ---------------------------------------------------------
                  Ola Ellnestam
                  Agical AB
                  Västerlånggatan 79, 2 tr
                  111 29 Stockholm, SWEDEN

                  Mobile: +46-708-754000
                  E-mail: ola.ellnestam@ <mailto:ola.ellnestam%40agical.se> agical.se
                  Blog: http://ellnestam. <http://ellnestam.wordpress.com> wordpress.com





                  [Non-text portions of this message have been removed]
                • Ola Ellnestam
                  Hi Kent, ... Yes. I might have been a bit to harsh in my previous post. It s better to have a test like this, some sort of alarm. /If/ you take the time needed
                  Message 8 of 14 , Feb 29, 2008
                    Hi Kent,
                    > I wouldn't want tests that didn't help either. However, I work with a team
                    > that has extensive tests that diff PDF or HTML. They are valuable as change
                    > detectors. If one breaks and they didn't expect it, they know to be careful.
                    > However, the team have made it easy to say, "That change is ok, this test
                    > should pass now." I suppose the next step would be to do a root cause
                    > analysis whenever one of the "hostage" tests breaks to see what failing unit
                    > tests they should have written.
                    >
                    >
                    >
                    Yes. I might have been a bit to harsh in my previous post.

                    It's better to have a test like this, some sort of alarm. /If/ you take
                    the time needed for the root cause analysis.

                    Recently I've been in a situation where this rarely happened. Hence my
                    strong criticism of such tests. :-(

                    //Ola

                    --
                    ---------------------------------------------------------
                    Ola Ellnestam
                    Agical AB
                    Västerlånggatan 79, 2 tr
                    111 29 Stockholm, SWEDEN

                    Mobile: +46-708-754000
                    E-mail: ola.ellnestam@...
                    Blog: http://ellnestam.wordpress.com
                  • marty.nelson
                    I tend to document via test any assumptions on arbitrary values or behaviors of framework or third-part components specifically for the reason that I want to
                    Message 9 of 14 , Feb 29, 2008
                      I tend to document via test any assumptions on arbitrary values or
                      behaviors of framework or third-part components specifically for the
                      reason that I want to know if they change. That way I'll know right
                      away if something goes awry.

                      --- In extremeprogramming@yahoogroups.com, Michael Feathers
                      <mfeathers@...> wrote:
                      >
                      > Paul Campbell wrote:
                      > > --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@>
                      > > wrote:
                      > >
                      > >> A friend on another list asked "are bad tests better than no
                      tests?"
                      > >>
                      > >
                      > > I've seen plenty of tests that are worse than useless. A common
                      theme
                      > > in such tests is diffing (rather than interpreting) textual output
                      > > that has semantic content. Common examples include diffing HTML,
                      > > diffing XML, diffing SQL queries. The problem with such tests is
                      that
                      > > in general they only tell you when something has changed - not
                      whether
                      > > it works, and in my opinion that makes them worse than useless in
                      many
                      > > cases.
                      > When you're refactoring or adding functionality, you do want to
                      know
                      > when something has changed. It's not the best general testing
                      strategy,
                      > but it has its uses.
                      >
                      > Michael Feathers
                      >
                    • Daniel Pupek
                      Most of time, I prefer to see tests as a design tool. But I have to admit that I do like tests that will tell me when something has changed (broken?). It s a
                      Message 10 of 14 , Feb 29, 2008
                        Most of time, I prefer to see tests as a design tool. But I have to
                        admit that I do like tests that will tell me when something has
                        changed (broken?). It's a nice side affect.

                        On Sat, Mar 1, 2008 at 5:07 AM, marty.nelson <noslenytram@...> wrote:
                        >
                        >
                        >
                        >
                        >
                        >
                        > I tend to document via test any assumptions on arbitrary values or
                        > behaviors of framework or third-part components specifically for the
                        > reason that I want to know if they change. That way I'll know right
                        > away if something goes awry.
                        >
                        > --- In extremeprogramming@yahoogroups.com, Michael Feathers
                        > <mfeathers@...> wrote:
                        > >
                        > > Paul Campbell wrote:
                        > > > --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@>
                        > > > wrote:
                        > > >
                        > > >> A friend on another list asked "are bad tests better than no
                        > tests?"
                        > > >>
                        > > >
                        > > > I've seen plenty of tests that are worse than useless. A common
                        > theme
                        > > > in such tests is diffing (rather than interpreting) textual output
                        > > > that has semantic content. Common examples include diffing HTML,
                        > > > diffing XML, diffing SQL queries. The problem with such tests is
                        > that
                        > > > in general they only tell you when something has changed - not
                        > whether
                        > > > it works, and in my opinion that makes them worse than useless in
                        > many
                        > > > cases.
                        > > When you're refactoring or adding functionality, you do want to
                        > know
                        > > when something has changed. It's not the best general testing
                        > strategy,
                        > > but it has its uses.
                        > >
                        > > Michael Feathers
                        > >
                        >
                        >



                        --


                        Checkout my blog @ http://blog.agilejedi.com
                        Checkout my homepage @ http://www.agilejedi.com
                      • marty.nelson
                        ... the ... I m also use to do a small-medium scale research spikes, and record the results as assumptions. For example, just yesterday I was writing a method
                        Message 11 of 14 , Mar 1, 2008
                          --- In extremeprogramming@yahoogroups.com, "Daniel Pupek" <dan@...>
                          wrote:
                          >
                          > Most of time, I prefer to see tests as a design tool. But I have to
                          > admit that I do like tests that will tell me when something has
                          > changed (broken?). It's a nice side affect.
                          >
                          > On Sat, Mar 1, 2008 at 5:07 AM, marty.nelson <noslenytram@...> wrote:
                          > > I tend to document via test any assumptions on arbitrary values or
                          > > behaviors of framework or third-part components specifically for
                          the
                          > > reason that I want to know if they change. That way I'll know right
                          > > away if something goes awry.

                          I'm also use to do a small-medium scale research spikes, and record the
                          results as assumptions. For example, just yesterday I was writing a
                          method that needed to use .NET reflection to get the type calling my
                          method. It turns out that I needed to get the stack frame and skip so
                          many entries to get to the caller. I started by writing a test that
                          looped from 0 to 5 and asserting that I found a match to the type of my
                          test. When I got an answer, I changed the test to use that value.
                        • Paul Campbell
                          ... whether ... many ... hostage ... improvements ... behavior at all. ... I definately agree with that, indeed TDD is so ingrained in some enviroments now
                          Message 12 of 14 , Mar 7, 2008
                            --- In extremeprogramming@yahoogroups.com, Michael Feathers
                            <mfeathers@...> wrote:
                            >
                            > Ola Ellnestam wrote:
                            > > Paul Campbell wrote:
                            > >
                            > >> --- In extremeprogramming@yahoogroups.com, "Pat Maddox" <pergesu@>
                            > >> wrote:
                            > >>
                            > >>
                            > >>> A friend on another list asked "are bad tests better than no tests?"
                            > >>>
                            > >>>
                            > >> I've seen plenty of tests that are worse than useless. A common theme
                            > >> in such tests is diffing (rather than interpreting) textual output
                            > >> that has semantic content. Common examples include diffing HTML,
                            > >> diffing XML, diffing SQL queries. The problem with such tests is that
                            > >> in general they only tell you when something has changed - not
                            whether
                            > >> it works, and in my opinion that makes them worse than useless in
                            many
                            > >> cases.
                            > >>
                            > >>
                            > >>
                            > >
                            > > I categorize these kind of tests under: "Tests holding the code
                            hostage"
                            > >
                            > > As said. They are beyond useless. They stop you from making
                            improvements
                            > > and are too coupled to the implementation. Not focusing at
                            behavior at all.
                            > >
                            > The hardest part of dealing with bad tests is knowing when to let go.
                            > We've conditioned ourselves to think "more tests: good", "fewer tests:
                            > bad." If tests are in the way, it's okay to delete them even though it
                            > feels wrong.

                            I definately agree with that, indeed TDD is so ingrained in some
                            enviroments now that persuading people to do *less* testing is a real
                            problem :-)

                            Paul.
                          • Paul Campbell
                            ... a team ... change ... careful. ... test ... failing unit ... Of course such tests sometimes have their uses but all too often people write such tests
                            Message 13 of 14 , Mar 7, 2008
                              --- In extremeprogramming@yahoogroups.com, "Kent Beck" <kentb@...> wrote:
                              >
                              > I wouldn't want tests that didn't help either. However, I work with
                              a team
                              > that has extensive tests that diff PDF or HTML. They are valuable as
                              change
                              > detectors. If one breaks and they didn't expect it, they know to be
                              careful.
                              > However, the team have made it easy to say, "That change is ok, this
                              test
                              > should pass now." I suppose the next step would be to do a root cause
                              > analysis whenever one of the "hostage" tests breaks to see what
                              failing unit
                              > tests they should have written.

                              Of course such tests sometimes have their uses but all too often
                              people write such tests because "I must have a unit test for
                              everything" rather then because they are genuinely useful.

                              A good "acid test" for such a test is - can I easily change the test
                              first to predict the new expected outcome ?, or is the new output
                              merely copied into test expectation after the fact. I view the latter
                              as strong indicator of a test that I should simply delete.

                              We must remember that tests create drag on the code base and reduce
                              agility.

                              Paul.
                            • Paul Campbell
                              ... g unit ... Of course I meant *bad* tests create drag ... :-).
                              Message 14 of 14 , Mar 12, 2008
                                --- In extremeprogramming@yahoogroups.com, "Paul Campbell" <yahoo@...>
                                wrote:
                                >
                                > --- In extremeprogramming@yahoogroups.com, "Kent Beck" <kentb@> wrote:
                                > >
                                > > I wouldn't want tests that didn't help either. However, I work with
                                > a team
                                > > that has extensive tests that diff PDF or HTML. They are valuable as
                                > change
                                > > detectors. If one breaks and they didn't expect it, they know to be
                                > careful.
                                > > However, the team have made it easy to say, "That change is ok, this
                                > test
                                > > should pass now." I suppose the next step would be to do a root cause
                                > > analysis whenever one of the "hostage" tests breaks to see what
                                > failin
                                g unit
                                > > tests they should have written.
                                >
                                > Of course such tests sometimes have their uses but all too often
                                > people write such tests because "I must have a unit test for
                                > everything" rather then because they are genuinely useful.
                                >
                                > A good "acid test" for such a test is - can I easily change the test
                                > first to predict the new expected outcome ?, or is the new output
                                > merely copied into test expectation after the fact. I view the latter
                                > as strong indicator of a test that I should simply delete.
                                >
                                > We must remember that tests create drag on the code base and reduce
                                > agility.
                                >
                                > Paul.
                                >

                                Of course I meant "*bad* tests create drag ..." :-).
                              Your message has been successfully submitted and would be delivered to recipients shortly.