Loading ...
Sorry, an error occurred while loading the content.

re: XP, unit testing and accepted failures

Expand Messages
  • smogallstar
    Hello all, I m just getting into XP and unit testing, and I like what I see (particularly the book Agile Software Development ), however I have a question:
    Message 1 of 10 , Dec 31, 2003
    • 0 Attachment
      Hello all,

      I'm just getting into XP and unit testing, and I like what I see
      (particularly the book "Agile Software Development"), however I have a
      question: How does unit testing view acceptable failures, such as a
      class being unable to connect to a remote host? I can tell that some
      things are assumed to work (i.e., not running out of memory), but how
      far does this extend? Should I make judgement calls? I have some
      reservations about assuming disk writes will go as planned, and I'm
      hesitant to the point of immobility to accept that network connections
      will be peachy.

      Thanks for any comments,
      Sean
    • J. B. Rainsberger
      ... If your system is supposed to handle these scenarios, then you need to write tests for how you handle them. This usually involves /simulating/ failures,
      Message 2 of 10 , Dec 31, 2003
      • 0 Attachment
        smogallstar wrote:

        > Hello all,
        >
        > I'm just getting into XP and unit testing, and I like what I see
        > (particularly the book "Agile Software Development"), however I have a
        > question: How does unit testing view acceptable failures, such as a
        > class being unable to connect to a remote host? I can tell that some
        > things are assumed to work (i.e., not running out of memory), but how
        > far does this extend? Should I make judgement calls? I have some
        > reservations about assuming disk writes will go as planned, and I'm
        > hesitant to the point of immobility to accept that network connections
        > will be peachy.

        If your system is supposed to handle these scenarios, then you need to
        write tests for how you handle them. This usually involves /simulating/
        failures, rather than recreating the conditions that lead to them.

        Usually you hide your connect() method behind an interface (if it isn't
        on an interface already), then build your RemoteHostClient with a
        connect method that accepts a remote host as a parameter. (The remote
        host has your connect() method that might fail with a "10061" error or
        something.)

        RemoteHostClient.connectTo(remoteHost)

        Now you write a test for how your RemoteHostClient should behave when
        the remoteHost fails to connect.

        testRemoteHostCannotConnect {
        crashTestDummy = new RemoteHost() {
        public Connection connect() throws Exception {
        throw new CommunicationsException(10061);
        }
        }

        assertNull(RemoteHostClient.connectTo(crashTestDummy))
        }

        The crash test dummy is a RemoteHost that fails /in a predictable
        manner/. In this case, we've decided that when we cannot connect,
        RemoteHostClient.connectTo() should return a null RemoteHostClient.
        (I've made connectTo a creation method, just for fun.)

        The underlying idea is this: /simulate/ failures, rather than trying to
        recreate the conditions that lead to those failures. Here we create a
        RemoteHost that you just can't connect to. This is easier than trying to
        connect to a real remote host that you've just shut off during the
        tests. That's the important part. The rest is just fluff, I guess. :)

        Does this help?
        --
        J. B. Rainsberger,
        Diaspar Software Services
        http://www.diasparsoftware.com :: +1 416 791-8603
        Let's write software that people understand
      • yahoogroups@jhrothjr.com
        From: smogallstar Sent: Wednesday, December 31, 2003 6:02 PM Subject: [XP] re: XP, unit testing and
        Message 3 of 10 , Dec 31, 2003
        • 0 Attachment
          From: "smogallstar" <smogallstar.at.yahoo.com@...>
          Sent: Wednesday, December 31, 2003 6:02 PM
          Subject: [XP] re: XP, unit testing and accepted failures


          > Hello all,
          >
          > I'm just getting into XP and unit testing, and I like what I see
          > (particularly the book "Agile Software Development"),

          I presume you mean Alistair Cockburn's book by that name?

          > however I have a
          > question: How does unit testing view acceptable failures, such as a
          > class being unable to connect to a remote host? I can tell that some
          > things are assumed to work (i.e., not running out of memory), but how
          > far does this extend? Should I make judgement calls? I have some
          > reservations about assuming disk writes will go as planned, and I'm
          > hesitant to the point of immobility to accept that network connections
          > will be peachy.

          This is the point where you split the tests into several categories.
          In the tests you run all the time, you mock out things like network
          connections so that you can concentrate on the program logic.
          Of course, your mocks should simulate network failures, etc., if
          dealing sanely with them is part of the application specs.

          Another category of tests will deal with the real world network.

          There are a number of reasons for the split, one of which is the
          real world concern you just expressed. Other reasons are that
          the bulk of the tests must run fast enough to insure that they will,
          in fact, get run. Also you don't want the mainline tests dependent
          on anything that behaves in a non-deterministic manner. You need
          to simulate error handling in the mainline tests, but you can reserve
          the longer tests against the complete environment for times when
          you can afford it, such as overnight.

          HTH
          John Roth

          >
          > Thanks for any comments,
          > Sean
        • John Brewer
          ... For unit tests, you want to isolate the class under test, and control its environment completely. The usual way to do this is to abstract out interfaces
          Message 4 of 10 , Dec 31, 2003
          • 0 Attachment
            > I'm just getting into XP and unit testing, and I like what I see
            >(particularly the book "Agile Software Development"), however I have a
            >question: How does unit testing view acceptable failures, such as a
            >class being unable to connect to a remote host? I can tell that some
            >things are assumed to work (i.e., not running out of memory), but how
            >far does this extend? Should I make judgement calls? I have some
            >reservations about assuming disk writes will go as planned, and I'm
            >hesitant to the point of immobility to accept that network connections
            >will be peachy.

            For unit tests, you want to isolate the class under test, and control
            its environment completely. The usual way to do this is to abstract
            out interfaces for the classes the class under test. Then you can
            plug in a so-called "mock object" that implements that interface, but
            fakes the implementation. Then you can have your mock object either
            work or as needed, so you can test your class's behavior in both
            cases.

            For acceptance tests and overall system tests, I'd treat the systems
            inability to deal with a network or disk space failure as a system
            failure. I'd probably also get together with a suitably sadistic QA
            person, and figure out how to repeatably cause such a network or disk
            failure, so that we could test the system's response to the situation
            via an automated test suite.
            --

            John Brewer
            Jera Design

            Extreme Programming FAQ: http://www.jera.com/techinfo/xpfaq.html
          • Sean Gilbertson
            Ah yes, the mock object. That seems to slip out of my consciousness once in a while (as I said, I m still getting used to XP ;-). But, your most helpful
            Message 5 of 10 , Dec 31, 2003
            • 0 Attachment
              Ah yes, the mock object. That seems to slip out of
              my consciousness once in a while (as I said, I'm still
              getting used to XP ;-).

              But, your most helpful comment was the confirmation
              of my inclination to treat it as a real defect. "But
              how to deal with that?" I thought. You say, "with
              ACCEPTANCE tests." That sounds very reasonable, and
              fits the framework. However, let me pose a scenario:
              I decide to handle a disconnect. This in turn fails
              the actual unit test. What then? Basically, my
              concern is for those situations where you simply
              cannot guarantee success (in a reasonable manner.. I
              would be lenient in a situation where memory capacity
              is not a concern) -- unit testing seems to take the
              approach that you can guarantee success through the
              presence of 100% code-coverage testing. It occurs to
              me now, though: unit testing IS for ideal situations,
              in some respects (i.e., the network). However, XP has
              not told me (yet) how to deal with failure.

              That was pretty roundabout, but you did reassure me
              and give me plenty to think about. Thank you!

              - Sean


              --- John Brewer <jbrewer@...> wrote:
              > > I'm just getting into XP and unit testing, and
              > I like what I see
              > >(particularly the book "Agile Software
              > Development"), however I have a
              > >question: How does unit testing view acceptable
              > failures, such as a
              > >class being unable to connect to a remote host? I
              > can tell that some
              > >things are assumed to work (i.e., not running out
              > of memory), but how
              > >far does this extend? Should I make judgement
              > calls? I have some
              > >reservations about assuming disk writes will go as
              > planned, and I'm
              > >hesitant to the point of immobility to accept that
              > network connections
              > >will be peachy.
              >
              > For unit tests, you want to isolate the class under
              > test, and control
              > its environment completely. The usual way to do
              > this is to abstract
              > out interfaces for the classes the class under test.
              > Then you can
              > plug in a so-called "mock object" that implements
              > that interface, but
              > fakes the implementation. Then you can have your
              > mock object either
              > work or as needed, so you can test your class's
              > behavior in both
              > cases.
              >
              > For acceptance tests and overall system tests, I'd
              > treat the systems
              > inability to deal with a network or disk space
              > failure as a system
              > failure. I'd probably also get together with a
              > suitably sadistic QA
              > person, and figure out how to repeatably cause such
              > a network or disk
              > failure, so that we could test the system's response
              > to the situation
              > via an automated test suite.
              > --
              >
              > John Brewer
              > Jera Design
              >
              > Extreme Programming FAQ:
              > http://www.jera.com/techinfo/xpfaq.html
              >
              > To Post a message, send it to:
              > extremeprogramming@...
              >
              > To Unsubscribe, send a blank message to:
              > extremeprogramming-unsubscribe@...
              >
              > ad-free courtesy of objectmentor.com
              >
              > Yahoo! Groups Links
              >
              > To visit your group on the web, go to:
              > http://groups.yahoo.com/group/extremeprogramming/
              >
              > To unsubscribe from this group, send an email to:
              > extremeprogramming-unsubscribe@yahoogroups.com
              >
              > Your use of Yahoo! Groups is subject to:
              > http://docs.yahoo.com/info/terms/
              >
              >

              __________________________________
              Do you Yahoo!?
              Free Pop-Up Blocker - Get it now
              http://companion.yahoo.com/
            • John Brewer
              ... Could you elaborate on this scenario? It s not quite clear to me. What do you mean by I decide to handle a disconnect ? And why would doing that fail a
              Message 6 of 10 , Dec 31, 2003
              • 0 Attachment
                >However, let me pose a scenario:
                >I decide to handle a disconnect. This in turn fails
                >the actual unit test. What then?

                Could you elaborate on this scenario? It's not quite clear to me.
                What do you mean by "I decide to handle a disconnect"? And why would
                doing that fail a unit test?
                --

                John Brewer
                Jera Design

                Extreme Programming FAQ: http://www.jera.com/techinfo/xpfaq.html
              • Sean Gilbertson
                By handle, for example: I surround a connect( ) in a try block, and connect throws a CouldNotConnectException (or something. In this scenario, the
                Message 7 of 10 , Dec 31, 2003
                • 0 Attachment
                  By handle, for example: I surround a "connect( )"
                  in a try block, and connect throws a
                  "CouldNotConnectException" (or something. In this
                  scenario, the function fails and either propagates
                  the exception or returns something unexpected or
                  unwanted. If you wrote a unit test that ensured that
                  something was received off the socket, that would fail
                  the unit test.

                  - Sean


                  --- John Brewer <jbrewer@...> wrote:
                  > >However, let me pose a scenario:
                  > >I decide to handle a disconnect. This in turn
                  > fails
                  > >the actual unit test. What then?
                  >
                  > Could you elaborate on this scenario? It's not
                  > quite clear to me.
                  > What do you mean by "I decide to handle a
                  > disconnect"? And why would
                  > doing that fail a unit test?
                  > --
                  >
                  > John Brewer
                  > Jera Design
                  >
                  > Extreme Programming FAQ:
                  > http://www.jera.com/techinfo/xpfaq.html
                  >
                  > To Post a message, send it to:
                  > extremeprogramming@...
                  >
                  > To Unsubscribe, send a blank message to:
                  > extremeprogramming-unsubscribe@...
                  >
                  > ad-free courtesy of objectmentor.com
                  >
                  > Yahoo! Groups Links
                  >
                  > To visit your group on the web, go to:
                  > http://groups.yahoo.com/group/extremeprogramming/
                  >
                  > To unsubscribe from this group, send an email to:
                  > extremeprogramming-unsubscribe@yahoogroups.com
                  >
                  > Your use of Yahoo! Groups is subject to:
                  > http://docs.yahoo.com/info/terms/
                  >
                  >

                  __________________________________
                  Do you Yahoo!?
                  Free Pop-Up Blocker - Get it now
                  http://companion.yahoo.com/
                • yahoogroups@jhrothjr.com
                  From: Sean Gilbertson Sent: Wednesday, December 31, 2003 8:16 PM Subject: Re: [XP] re: XP, unit
                  Message 8 of 10 , Dec 31, 2003
                  • 0 Attachment
                    From: "Sean Gilbertson"
                    <smogallstar.at.yahoo.com@...>
                    Sent: Wednesday, December 31, 2003 8:16 PM
                    Subject: Re: [XP] re: XP, unit testing and accepted failures


                    > By handle, for example: I surround a "connect( )"
                    > in a try block, and connect throws a
                    > "CouldNotConnectException" (or something. In this
                    > scenario, the function fails and either propagates
                    > the exception or returns something unexpected or
                    > unwanted. If you wrote a unit test that ensured that
                    > something was received off the socket, that would fail
                    > the unit test.

                    > - Sean

                    I see what you're saying. You're trying to test
                    exception handling. The easiest way to do this
                    is to mock it out so that you don't call connect()
                    directly. You have your own proxy for the
                    library calls. Then you can set up the mock to throw
                    an exception on the test for whether the exception
                    handler works, and to simulate a connection on
                    the tests where you want to test that function.

                    The last time I had a network project, I did that
                    and it cut my test time way down.

                    Where this gets messy is, of course, the question
                    of whether the mocks are actually modeling the
                    live environment properly. For that test, I'd put
                    a simulator on the other end of the wire so I could
                    force the error conditions I wanted when I wanted
                    them.

                    John Roth
                  • Charlie Poole
                    Hi Sean, If you start having acceptable failures you ll end up with unacceptable failures as well. My current client has one dll which is generating upwards
                    Message 9 of 10 , Dec 31, 2003
                    • 0 Attachment
                      Hi Sean,

                      If you start having "acceptable failures" you'll end up with unacceptable
                      failures as well. My current client has one dll which is generating upwards
                      of 50 failures due to a problem on the server that runs the nightly tests.
                      Predictably, nobody is reading through all those failures to determine which
                      are acceptable and which are not. From a cursory examination, I believe
                      there are some real problems hiding in the large list of failures (perceived
                      to be) caused by the server environment.

                      That said... if your test depends on connecting to a remote host, are you
                      really sure it's a unit test?

                      Charlie Poole
                      cpoole@...
                      www.pooleconsulting.com
                      www.charliepoole.org



                      > -----Original Message-----
                      > From: smogallstar [mailto:smogallstar@...]
                      > Sent: Wednesday, December 31, 2003 3:03 PM
                      > To: extremeprogramming@yahoogroups.com
                      > Subject: [XP] re: XP, unit testing and accepted failures
                      >
                      >
                      > Hello all,
                      >
                      > I'm just getting into XP and unit testing, and I like what I see
                      > (particularly the book "Agile Software Development"), however I have a
                      > question: How does unit testing view acceptable failures, such as a
                      > class being unable to connect to a remote host? I can tell that some
                      > things are assumed to work (i.e., not running out of memory), but how
                      > far does this extend? Should I make judgement calls? I have some
                      > reservations about assuming disk writes will go as planned, and I'm
                      > hesitant to the point of immobility to accept that network connections
                      > will be peachy.
                      >
                      > Thanks for any comments,
                      > Sean
                      >
                      >
                      > To Post a message, send it to: extremeprogramming@...
                      >
                      > To Unsubscribe, send a blank message to:
                      > extremeprogramming-unsubscribe@...
                      >
                      > ad-free courtesy of objectmentor.com
                      >
                      > Yahoo! Groups Links
                      >
                      > To visit your group on the web, go to:
                      > http://groups.yahoo.com/group/extremeprogramming/
                      >
                      > To unsubscribe from this group, send an email to:
                      > extremeprogramming-unsubscribe@yahoogroups.com
                      >
                      > Your use of Yahoo! Groups is subject to:
                      > http://docs.yahoo.com/info/terms/
                      >
                      >
                      >
                    • Gary Brown
                      ... Sean, Adding to John s experience ... A couple of years ago, I worked on a project for a music and video distribution warehouse in Indianapolis. The
                      Message 10 of 10 , Jan 1, 2004
                      • 0 Attachment
                        --- In extremeprogramming@yahoogroups.com, yahoogroups@j... wrote:
                        > From: "Sean Gilbertson"
                        > <smogallstar.at.yahoo.com@y...>
                        > Sent: Wednesday, December 31, 2003 8:16 PM
                        > Subject: Re: [XP] re: XP, unit testing and accepted failures
                        >
                        >
                        > > By handle, for example: I surround a "connect( )"
                        > > in a try block, and connect throws a
                        > > "CouldNotConnectException" (or something. In this
                        > > scenario, the function fails and either propagates
                        > > the exception or returns something unexpected or
                        > > unwanted. If you wrote a unit test that ensured that
                        > > something was received off the socket, that would fail
                        > > the unit test.
                        >
                        > > - Sean
                        >
                        > I see what you're saying. You're trying to test
                        > exception handling. The easiest way to do this
                        > is to mock it out so that you don't call connect()
                        > directly. You have your own proxy for the
                        > library calls. Then you can set up the mock to throw
                        > an exception on the test for whether the exception
                        > handler works, and to simulate a connection on
                        > the tests where you want to test that function.
                        >
                        > The last time I had a network project, I did that
                        > and it cut my test time way down.
                        >
                        > Where this gets messy is, of course, the question
                        > of whether the mocks are actually modeling the
                        > live environment properly. For that test, I'd put
                        > a simulator on the other end of the wire so I could
                        > force the error conditions I wanted when I wanted
                        > them.
                        >
                        > John Roth

                        Sean,

                        Adding to John's experience ...

                        A couple of years ago, I worked on a project for a music and video
                        distribution warehouse in Indianapolis. The purpose of the project
                        was to utilize a couple of surplus USPS flats sorting machines (FSM-
                        881s) to presort the out-going product, to get a better postage
                        rate. Part of the project was to provide a real-time display of the
                        sorting operation, e.g. throughput, percent complete, audit trail,
                        missing mailpieces, etc.

                        The "dashboard" was a VB6 app, connected to the FSM-881s with a null
                        modem serial cable. When properly configured, the FSM-881 will send
                        a psuedo-xml data stream through the serial line to the listener.
                        All of the physical mailpieces had three barcodes, postnet code,
                        planet code, and order number. As the FSM scanned each package, it
                        would decide what sort bin the package belonged in and send it down
                        the raceway. When the package arrived at the appropriate bin, a big
                        blue finger would knock it off the track and into a collection bin.
                        The FSM would then send a <mailpiece postnet="99999999999"
                        planet="99999999999" /> element to the serial connection. There were
                        many other element types in the data stream, which were mostly
                        ignored. There was no connection protocol, just open the COM port
                        and start parsing.

                        The FSM-881s are 70s technology. They are about 100 feet long by 20
                        feet wide. The on-board control computer runs Q-NIX, all of the
                        control code is written in C. Sometime in the late 80s, the control
                        computer was upgraded to a 286, and remains so today. The machine
                        runs great most of the time, except when it doesn't ...

                        Anyway, I needed a way to test this data stream in my VB6 app. I
                        strung a null modem cable between two PCs, one acting as the control
                        computer and the other acting as the listener. On the control
                        computer, I wrote a small VB6 app that allowed me to select a test
                        case and transmit the data to the listener. There were many test
                        cases that covered the positive path, plus every possible error
                        condition, including dropping the connection, sending invalid
                        mailpiece elements, incomplete mailpiece elements, etc. Most of the
                        test cases simply opened a text file and transmitted the contents,
                        others were more complex.

                        On the listener, I wrote another small VB6 app to consume the data
                        stream and display the parsed data in a large testbox. I did that,
                        so I could learn about collecting data with this mechanism. Using
                        that knowledge, I then added the data consumer code to the dashboard
                        app. Over time, I added code to the test driver on the control
                        computer to read the application's mailpiece database to simulate a
                        complete production run. This allowed me to fully exercise the
                        dashboard display and to do performace and capacity measurements.

                        Gary Brown
                        Salt Valley Software Services
                        Lincoln, NE
                      Your message has been successfully submitted and would be delivered to recipients shortly.