Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] A TDD Failure

Expand Messages
  • Randy MacDonald
    I gather there s no test that crashes. I d isolate that. ... From: Andrew Wall To: Sent:
    Message 1 of 5 , Feb 23 1:51 PM
    • 0 Attachment
      I gather there's no test that crashes. I'd isolate that.

      ----- Original Message -----
      From: "Andrew Wall" <ajwall.geo@...>
      To: <extremeprogramming@yahoogroups.com>
      Sent: Wednesday, February 23, 2005 4:31 PM
      Subject: [XP] A TDD Failure


      >
      >
      > Hello,
      >
      > Background
      > My job is to develop C++ programs for an embedded processor. This
      > means that we make the build process go through various hoops to
      > construct a binary file that is loaded onto the target, even though
      > the compiler produces x86 code and runs on a PC. I've developed an
      > xUnit test framework that runs under Windows and shows a form with a
      > progress bar, etc. However, this uses a slightly different
      > compiler/IDE to the target compiler. Oops what a give away.
      >
      > The Program
      > Within the program there is a subsystem that takes two input
      > streams, validates them and forwards valid data to clients. It
      > prefers one of the inputs over the other and switches between them
      > as it sees fit. In fact, when one has precedence, the other is
      > forced to forward its information to a Null Object. I found a
      > problem when trying to signal invalid data. I found that when
      > invalid data occurred in the preferred stream, it wasn't signalled:
      > I'd forgotten about that scenario. I added a new test for this,
      > which failed, naturally. I put the fix in and lo and behold the
      > test passed! Classic TDD. Unfortunately, recompiling and running
      > on the target made it crash! I removed the fix, the tests failed,
      > but the target no longer crashed. By coincidence another oversight
      > on my part also meant that the system still behaved ok without the
      > fix - so to cut a long story short, the code remains without the
      > fix, so in the short-term the customer is happy.
      >
      > I'm not happy. This is a TDD failure because I can't put the fix in
      > without crashing the system, and I can't leave it out without having
      > failing unit tests! The result is that I have no tests and no fix.
      >
      > My conclusion
      > Despite using the dodgy practise of testing with one compiler and
      > producing a binary from another, I will stick with it because I must
      > have written at least 3000 tests over many projects and I have only
      > seen this one failure in TDD.
      >
      > I am hoping that I'll be able to figure this out shortly and get
      > back to you.
      >
      > Andrew Wall
      > Still not doing XP

      Later...
      -----------------------------------------------------------------------
      |\/| Randy A MacDonald | you can't pay for it,
      |\\| randymacdo@... | even if you want to.
      BSc(Math) UNBF'83 Sapere Aude | APL: If you can say it, it's done..
      Natural Born APL'er | demo website: http://24.43.158.135/
      ------------------------------------------------------------{ gnat }-
    • Andrew Wall
      Kent, I can t possibly demonstrate undefined behaviour, but what I can do is keep the code within the bounds of defined behaviour. I can write a test that
      Message 2 of 5 , Mar 1, 2005
      • 0 Attachment
        Kent,
        I can't possibly demonstrate undefined behaviour, but what I can do
        is keep the code within the bounds of defined behaviour.
        I can write a test that attempts to overfill the string in question
        and test for the string ending up being filled exactly to the end.
        Then I can remove any current behaviour I've put in (Yes, that bit
        was not written Test First) and see the test fail. (I'm fairly sure
        that it won't bring down the test project, but I can never be
        sure.) I can then put some behaviour back in to demonstrate the
        test passing.

        I'll do that tomorrow and report on it later.

        Andrew Wall
        Still not doing XP

        --- In extremeprogramming@yahoogroups.com, "Kent Beck" <kentb@e...>
        wrote:
        > Andrew,
        >
        > Thank you for following up on this. I would like to see a test that
        > demonstrates the "undefined" behavior you (accidentally) expected
        and
        > another that demonstrates the behavior you got in the production
        > environment.
        >
        > Kent Beck
        > Three Rivers Institute
        >
        > > -----Original Message-----
        > > From: Andrew Wall [mailto:ajwall.geo@y...]
        > > Sent: Saturday, February 26, 2005 11:35 AM
        > > To: extremeprogramming@yahoogroups.com
        > > Subject: Re: [XP] A TDD Failure
        > >
        > > Follow-up
        > >
        > > Thanks for the replies.
        > > I've not been able to reproduce the crash of the target machine
        in
        > > the test project, although I've been able to avoid the crash on
        the
        > > target machine now. It was something to do with an unchecked
        string
        > > length. So 'undefined behaviour' is different between the two
        > > compilers.
        > >
        > > Anyway, that was just a little bump on the road of practising
        TDD
        > > and not a hiatus.
        > >
        > > Andrew Wall
        > > Still not doing XP
      • Andrew Wall
        Next report: The first thing I did was to remove the patch that avoided the crash on the target machine. However, no tests failed! Oops! Well I knew that
        Message 3 of 5 , Mar 4, 2005
        • 0 Attachment
          Next report:

          The first thing I did was to remove the patch that avoided the crash
          on the target machine. However, no tests failed! Oops! Well I
          knew that was going to happen didn't I.
          I then wrote a new test that fed a long string through the checking
          process. The idea being that this would be stored, in fact with the
          patch removed, this would overrun the fixed length buffer. The test
          then asked for the stored string and asserted that its length was at
          maximum. Ah ha! Now here was a test that failed.

          Since my last post I was thinking about what should happen, and I
          realised that the concept of the fixed length buffer should be
          encapsulated, so now I knew a better patch to apply (i.e. use strnlen
          () ). So in it went and the test passed.
          I recompiled for the target machine and that worked also. Job done!
          Now I am even more confident of TDD: I've encountered a problem and
          solved it using TDD, even though at first I thought TDD had let me
          down.

          Some of my colleagues are a bit paranoid about software: They think
          that even recompiling may vary the behaviour of software. They may
          be right if you change some libraries that you compile with, but TDD
          should expose that. I'm more confident in the libraries that come
          with the two different compilers that I have. If I had a reason to,
          I could probably compile the tests with the target compiler and run
          those under a console app version of my test framework, possibly as
          part of the build process. But 3000+ tests tell me that the time is
          not now.

          Andrew Wall
          Still not doing XP


          --- In extremeprogramming@yahoogroups.com, "Andrew Wall"
          <ajwall.geo@y...> wrote:
          >
          > Kent,
          > I can't possibly demonstrate undefined behaviour, but what I can
          do
          > is keep the code within the bounds of defined behaviour.
          > I can write a test that attempts to overfill the string in
          question
          > and test for the string ending up being filled exactly to the
          end.
          > Then I can remove any current behaviour I've put in (Yes, that bit
          > was not written Test First) and see the test fail. (I'm fairly
          sure
          > that it won't bring down the test project, but I can never be
          > sure.) I can then put some behaviour back in to demonstrate the
          > test passing.
          >
          > I'll do that tomorrow and report on it later.
          >
          > Andrew Wall
          > Still not doing XP
          >
          > --- In extremeprogramming@yahoogroups.com, "Kent Beck"
          <kentb@e...>
          > wrote:
          > > Andrew,
          > >
          > > Thank you for following up on this. I would like to see a test
          that
          > > demonstrates the "undefined" behavior you (accidentally)
          expected
          > and
          > > another that demonstrates the behavior you got in the production
          > > environment.
          > >
          > > Kent Beck
          > > Three Rivers Institute
          > >
          > > > -----Original Message-----
          > > > From: Andrew Wall [mailto:ajwall.geo@y...]
          > > > Sent: Saturday, February 26, 2005 11:35 AM
          > > > To: extremeprogramming@yahoogroups.com
          > > > Subject: Re: [XP] A TDD Failure
          > > >
          > > > Follow-up
          > > >
          > > > Thanks for the replies.
          > > > I've not been able to reproduce the crash of the target
          machine
          > in
          > > > the test project, although I've been able to avoid the crash
          on
          > the
          > > > target machine now. It was something to do with an unchecked
          > string
          > > > length. So 'undefined behaviour' is different between the two
          > > > compilers.
          > > >
          > > > Anyway, that was just a little bump on the road of practising
          > TDD
          > > > and not a hiatus.
          > > >
          > > > Andrew Wall
          > > > Still not doing XP
        • Davide Inglima
          ... Andrew: I don t know if it would be allowed, so please disregard my post if it is impossible or unfeasible... but in my opinion you could apply the new
          Message 4 of 5 , Mar 7, 2005
          • 0 Attachment
            On Fri, 04 Mar 2005 17:05:51 -0000, Andrew Wall <ajwall.geo@...> wrote:

            > I could probably compile the tests with the target compiler and run
            > those under a console app version of my test framework, possibly as
            > part of the build process. But 3000+ tests tell me that the time is
            > not now.

            Andrew: I don't know if it would be allowed, so please disregard my
            post if it is impossible or unfeasible... but in my opinion you could
            apply the new testing framework and practices little by little to the
            new modules or to old modules that need to be debugged and modified.
            Then, when you have time, you could refactor the old tests.

            Start small, create a new testsuite and put, let's say, 10 tests under
            the automatic build. After being sure that the old tests behave
            correctly, remove those 10 tests from the old tree and convert other
            tests. Then, little by little, migrate the old tests to the new ones.

            Of course, apply this only if you dare, care and have resources to do it ;).
          • Andrew Wall
            ... run ... as ... time is ... could ... the ... modified. ... under ... other ... ones. ... do it ;). Thanks for your reply Davide. The reason to compile the
            Message 5 of 5 , Mar 11, 2005
            • 0 Attachment
              --- In extremeprogramming@yahoogroups.com, Davide Inglima
              <limacat@g...> wrote:
              > On Fri, 04 Mar 2005 17:05:51 -0000, Andrew Wall <ajwall.geo@y...>
              wrote:
              >
              > > I could probably compile the tests with the target compiler and
              run
              > > those under a console app version of my test framework, possibly
              as
              > > part of the build process. But 3000+ tests tell me that the
              time is
              > > not now.
              >
              > Andrew: I don't know if it would be allowed, so please disregard my
              > post if it is impossible or unfeasible... but in my opinion you
              could
              > apply the new testing framework and practices little by little to
              the
              > new modules or to old modules that need to be debugged and
              modified.
              > Then, when you have time, you could refactor the old tests.
              >
              > Start small, create a new testsuite and put, let's say, 10 tests
              under
              > the automatic build. After being sure that the old tests behave
              > correctly, remove those 10 tests from the old tree and convert
              other
              > tests. Then, little by little, migrate the old tests to the new
              ones.
              >
              > Of course, apply this only if you dare, care and have resources to
              do it ;).

              Thanks for your reply Davide.
              The reason to compile the tests with the target compiler is to allow
              the tests to experience the same undefined behaviour that the target
              machine might do. The time is not now because 3000+ tests tell me
              that I'm doing OK right now when I use an alternate compiler/IDE.
              If I weren't doing OK and I kept on coming up against quirky
              behaviour, I might reconsider.

              Andrew Wall
              Still not doing XP
            Your message has been successfully submitted and would be delivered to recipients shortly.