Loading ...
Sorry, an error occurred while loading the content.

Defining a Unit-Testable System/Codebase

Expand Messages
  • Roy Osherove
    I ve come up with what I feel is a reasonable definition of what a unit-testable code base rarely means in terms of code,design quality and unit tests. I d
    Message 1 of 24 , Feb 12, 2006
    • 0 Attachment
      I've come up with what I feel is a reasonable definition of what a
      "unit-testable" code base rarely means in terms of code,design quality and
      unit tests. I'd love to know your thoughts.

      Definition of a Unit Testable System: For each *logical part* in the code
      base, a Unit test or more that verifies that logic can be implemented in a
      *reasonably easy* manner that stands by the PC-COF(I couldn't find a good
      anagram...) rules, defined below:

      1.P- Partial runs of the tests are possible
      2.C- Consistent pass/fail feedback from each test
      3.C-Configuration is not necessary before running any of the tests
      4.O-Order of the test runs does not matter
      5.F- Fast. Tests run Fast.
      #2 is difficult and should still be true while 1 and 4 are true (which is
      really saying "how decoupled is your design?"
      #3 is almost impossible to achieve without proper separation of concerns
      #5 is almost impossible to achieve without separation of Logic and
      dependencies.

      If you can *easily* write tests that adhere to the PC-COF rules, that means
      that your system's design and code base are in a good state, a unit-testable
      state. Of course, TDD leads to these rules automatically. On the other hand,
      *integration Tests* will almost always follow the opposite rules, and are
      more prone to points 4 and 3 with #2 almost impossible to achieve in the
      wrong ordering..

      Thoughts?

      --
      Thanks,

      Roy Osherove
      Principal, Team Agile
      Agile training and mentoring for your dev team
      www.TeamAgile.com
      blog: http://www.iserializable.com


      [Non-text portions of this message have been removed]
    • David H.
      ... What about applications or code base parts that are threaded. I might be misunderstanding something here, but in that particular case you might come across
      Message 2 of 24 , Feb 12, 2006
      • 0 Attachment
        Roy Osherove wrote:
        > I've come up with what I feel is a reasonable definition of what a
        > "unit-testable" code base rarely means in terms of code,design quality and
        > unit tests. I'd love to know your thoughts.
        >
        > Definition of a Unit Testable System: For each *logical part* in the code
        > base, a Unit test or more that verifies that logic can be implemented in a
        > *reasonably easy* manner that stands by the PC-COF(I couldn't find a good
        > anagram...) rules, defined below:
        >
        > 1.P- Partial runs of the tests are possible
        > 2.C- Consistent pass/fail feedback from each test
        > 3.C-Configuration is not necessary before running any of the tests
        > 4.O-Order of the test runs does not matter
        What about applications or code base parts that are threaded. I might be
        misunderstanding something here, but in that particular case you might come
        across a function where it does make a difference in what order the test is
        run, depending on a precondition being met or not, which is then Dependant on
        the state of the threaded app.

        -d
      • Roy Osherove
        The definition was about for each logical part . The logical part would run In a single thread just as well as a multi thread. unless the logic is a thread
        Message 3 of 24 , Feb 12, 2006
        • 0 Attachment
          The definition was about "for each logical part". The logical part would run
          In a single thread just as well as a multi thread. unless the logic is a
          thread manager logic kind of thing.

          IMO using Threads inside the tests is more of an integration test, since
          you're testing locks and system threads as well as your code's logic.
          I would test the *logic* inside a single thread, and use integration tests
          later (system tests too) to make sure that logic runs correctly under heavy
          load.
          If there is a specific logic to the thread handling other than simply
          locking, but maybe queuing and such, I'd see if can write that logic
          test-driven, which would force me to create wrappers around my real thread,
          and make them into logical objects managed by a logical entity which queues
          them, for example. But that's a tough one. Threads have always been tough.
          I mostly stay clear of multi threaded tests. If I actually do them, I
          certainly don't call them unit tests, and I put them in a separate place
          where the longer running integration tests would reside.


          On 2/13/06, David H. <dmalloc@...> wrote:
          >
          > Roy Osherove wrote:
          > > I've come up with what I feel is a reasonable definition of what a
          > > "unit-testable" code base rarely means in terms of code,design quality
          > and
          > > unit tests. I'd love to know your thoughts.
          > >
          > > Definition of a Unit Testable System: For each *logical part* in the
          > code
          > > base, a Unit test or more that verifies that logic can be implemented in
          > a
          > > *reasonably easy* manner that stands by the PC-COF(I couldn't find a
          > good
          > > anagram...) rules, defined below:
          > >
          > > 1.P- Partial runs of the tests are possible
          > > 2.C- Consistent pass/fail feedback from each test
          > > 3.C-Configuration is not necessary before running any of the tests
          > > 4.O-Order of the test runs does not matter
          > What about applications or code base parts that are threaded. I might be
          > misunderstanding something here, but in that particular case you might
          > come
          > across a function where it does make a difference in what order the test
          > is
          > run, depending on a precondition being met or not, which is then Dependant
          > on
          > the state of the threaded app.
          >
          > -d
          >
          >
          > To Post a message, send it to: extremeprogramming@...
          >
          > To Unsubscribe, send a blank message to:
          > extremeprogramming-unsubscribe@...
          >
          > ad-free courtesy of objectmentor.com
          > Yahoo! Groups Links
          >
          >
          >
          >
          >
          >
          >


          --
          Thanks,

          Roy Osherove
          http://www.iserializable.com


          [Non-text portions of this message have been removed]
        • Roy Osherove
          re-reading your email, I have a more focused answer, I think. SO if ordering matters, IMO - it s not a unit test. It s an integration test, or you are just
          Message 4 of 24 , Feb 12, 2006
          • 0 Attachment
            re-reading your email, I have a more focused answer, I think.
            SO if ordering matters, IMO - it's not a unit test. It's an integration
            test, or you are just writing the unit test wrong, and maybe think about
            separating it into two unit tests that share a common function which
            intializes some object's state.. (but it would still not be a unit test, for
            me, but an integration one).


            On 2/13/06, David H. <dmalloc@...> wrote:
            >
            > Roy Osherove wrote:
            > > I've come up with what I feel is a reasonable definition of what a
            > > "unit-testable" code base rarely means in terms of code,design quality
            > and
            > > unit tests. I'd love to know your thoughts.
            > >
            > > Definition of a Unit Testable System: For each *logical part* in the
            > code
            > > base, a Unit test or more that verifies that logic can be implemented in
            > a
            > > *reasonably easy* manner that stands by the PC-COF(I couldn't find a
            > good
            > > anagram...) rules, defined below:
            > >
            > > 1.P- Partial runs of the tests are possible
            > > 2.C- Consistent pass/fail feedback from each test
            > > 3.C-Configuration is not necessary before running any of the tests
            > > 4.O-Order of the test runs does not matter
            > What about applications or code base parts that are threaded. I might be
            > misunderstanding something here, but in that particular case you might
            > come
            > across a function where it does make a difference in what order the test
            > is
            > run, depending on a precondition being met or not, which is then Dependant
            > on
            > the state of the threaded app.
            >
            > -d
            >
            >
            > To Post a message, send it to: extremeprogramming@...
            >
            > To Unsubscribe, send a blank message to:
            > extremeprogramming-unsubscribe@...
            >
            > ad-free courtesy of objectmentor.com
            > Yahoo! Groups Links
            >
            >
            >
            >
            >
            >
            >


            --
            Thanks,

            Roy Osherove
            http://www.iserializable.com


            [Non-text portions of this message have been removed]
          • Steven Gordon
            Roy, What do you mean by consistent in rule 2? Do you mean that if you run the test multiple times without changing the code, you should get the same
            Message 5 of 24 , Feb 12, 2006
            • 0 Attachment
              Roy,

              What do you mean by "consistent" in rule 2? Do you mean that if you run the
              test multiple times without changing the code, you should get the same
              result?

              When it comes to test feedback, I would want each failing test to clearly
              indicate what failed.

              Steven Gordon

              On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
              >
              > I've come up with what I feel is a reasonable definition of what a
              > "unit-testable" code base rarely means in terms of code,design quality and
              > unit tests. I'd love to know your thoughts.
              >
              > Definition of a Unit Testable System: For each *logical part* in the code
              > base, a Unit test or more that verifies that logic can be implemented in a
              > *reasonably easy* manner that stands by the PC-COF(I couldn't find a good
              > anagram...) rules, defined below:
              >
              > 1.P- Partial runs of the tests are possible
              > 2.C- Consistent pass/fail feedback from each test
              > 3.C-Configuration is not necessary before running any of the tests
              > 4.O-Order of the test runs does not matter
              > 5.F- Fast. Tests run Fast.
              > #2 is difficult and should still be true while 1 and 4 are true (which is
              > really saying "how decoupled is your design?"
              > #3 is almost impossible to achieve without proper separation of concerns
              > #5 is almost impossible to achieve without separation of Logic and
              > dependencies.
              >
              > If you can *easily* write tests that adhere to the PC-COF rules, that
              > means
              > that your system's design and code base are in a good state, a
              > unit-testable
              > state. Of course, TDD leads to these rules automatically. On the other
              > hand,
              > *integration Tests* will almost always follow the opposite rules, and are
              > more prone to points 4 and 3 with #2 almost impossible to achieve in the
              > wrong ordering..
              >
              > Thoughts?
              >
              > --
              > Thanks,
              >
              > Roy Osherove
              > Principal, Team Agile
              > Agile training and mentoring for your dev team
              > www.TeamAgile.com
              > blog: http://www.iserializable.com
              >


              [Non-text portions of this message have been removed]
            • Roy Osherove
              Steven - yes. that s what I meant. If it fails, it will keep failing indefinitely until you change the code(e.g break something else :-) ) As for it saying
              Message 6 of 24 , Feb 12, 2006
              • 0 Attachment
                Steven - yes. that's what I meant. If it fails, it will keep failing
                indefinitely until you change the code(e.g break something else :-) )
                As for it saying what failed - that's a unit testing best practice, and
                people doing it or not does not depend on the health of the code base's
                design. But easily writing tests that fill these requirement is a testament
                that the system is indeed "testable" - that is, it lends itself to be tested
                easily using unit tests that fill follow rules. Having a good fail message
                in any of these tests has nothing to do with the system, but with the test
                standards, which are a different story..
                My definition here is that of the system under test, and one would know to
                define a system as easily testable with unit tests or not.

                If we can conclude that these tests being easily writable with a system with
                specific properties, we can perhaps deduce from that many other things which
                may be related, I'm just not sure what they are yet...

                Roy.

                On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                >
                > Roy,
                >
                > What do you mean by "consistent" in rule 2? Do you mean that if you run
                > the
                > test multiple times without changing the code, you should get the same
                > result?
                >
                > When it comes to test feedback, I would want each failing test to clearly
                > indicate what failed.
                >
                > Steven Gordon
                >
                > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                > >
                > > I've come up with what I feel is a reasonable definition of what a
                > > "unit-testable" code base rarely means in terms of code,design quality
                > and
                > > unit tests. I'd love to know your thoughts.
                > >
                > > Definition of a Unit Testable System: For each *logical part* in the
                > code
                > > base, a Unit test or more that verifies that logic can be implemented in
                > a
                > > *reasonably easy* manner that stands by the PC-COF(I couldn't find a
                > good
                > > anagram...) rules, defined below:
                > >
                > > 1.P- Partial runs of the tests are possible
                > > 2.C- Consistent pass/fail feedback from each test
                > > 3.C-Configuration is not necessary before running any of the tests
                > > 4.O-Order of the test runs does not matter
                > > 5.F- Fast. Tests run Fast.
                > > #2 is difficult and should still be true while 1 and 4 are true (which
                > is
                > > really saying "how decoupled is your design?"
                > > #3 is almost impossible to achieve without proper separation of concerns
                > > #5 is almost impossible to achieve without separation of Logic and
                > > dependencies.
                > >
                > > If you can *easily* write tests that adhere to the PC-COF rules, that
                > > means
                > > that your system's design and code base are in a good state, a
                > > unit-testable
                > > state. Of course, TDD leads to these rules automatically. On the other
                > > hand,
                > > *integration Tests* will almost always follow the opposite rules, and
                > are
                > > more prone to points 4 and 3 with #2 almost impossible to achieve in the
                > > wrong ordering..
                > >
                > > Thoughts?
                > >
                > > --
                > > Thanks,
                > >
                > > Roy Osherove
                > > Principal, Team Agile
                > > Agile training and mentoring for your dev team
                > > www.TeamAgile.com
                > > blog: http://www.iserializable.com
                > >
                >
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >
                > To Post a message, send it to: extremeprogramming@...
                >
                > To Unsubscribe, send a blank message to:
                > extremeprogramming-unsubscribe@...
                >
                > ad-free courtesy of objectmentor.com
                > Yahoo! Groups Links
                >
                >
                >
                >
                >
                >
                >
                >


                --
                Thanks,

                Roy Osherove
                http://www.iserializable.com


                [Non-text portions of this message have been removed]
              • Steven Gordon
                Roy, Are you saying that having the same test sometimes fail and sometimes pass (without any code changes) is a property of an untestable system? Could it not
                Message 7 of 24 , Feb 12, 2006
                • 0 Attachment
                  Roy,

                  Are you saying that having the same test sometimes fail and sometimes pass
                  (without any code changes) is a property of an untestable system? Could it
                  not sometimes be due to a poorly written test?

                  Why could it then not also be a property of some untestable systems that a
                  failure of some reasonable unit test cannot be attributed to any single
                  cause?

                  It seems to me either property could be caused by either a poorly written
                  unit test or a system that is too monolithic to be unit-testable.

                  Steve

                  On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                  >
                  > Steven - yes. that's what I meant. If it fails, it will keep failing
                  > indefinitely until you change the code(e.g break something else :-) )
                  > As for it saying what failed - that's a unit testing best practice, and
                  > people doing it or not does not depend on the health of the code base's
                  > design. But easily writing tests that fill these requirement is a
                  > testament
                  > that the system is indeed "testable" - that is, it lends itself to be
                  > tested
                  > easily using unit tests that fill follow rules. Having a good fail message
                  > in any of these tests has nothing to do with the system, but with the test
                  > standards, which are a different story..
                  > My definition here is that of the system under test, and one would know to
                  > define a system as easily testable with unit tests or not.
                  >
                  > If we can conclude that these tests being easily writable with a system
                  > with
                  > specific properties, we can perhaps deduce from that many other things
                  > which
                  > may be related, I'm just not sure what they are yet...
                  >
                  > Roy.
                  >


                  [Non-text portions of this message have been removed]
                • Roy Osherove
                  Steve, Obviously, a poor test that sometimes fails and sometimes does not could be attributed to a bad test, but an inconsistent test result also usually means
                  Message 8 of 24 , Feb 12, 2006
                  • 0 Attachment
                    Steve, Obviously, a poor test that sometimes fails and sometimes does not
                    could be attributed to a bad test, but an inconsistent test result also
                    usually means (at least in my experience) that the state under test is not
                    cleaned up or initialized between the test runs (such as object state not
                    reset, database state not reset). That usually is a sign that the test has
                    to manage external state, and does a poor job of that.
                    You could then perhaps say that the test is poor quality, but that a high
                    quality test for such a system, such that it always returns a fail pass
                    result is hard to write, since one has to manage external state, which goes
                    against the definitions I've laid thus far. You could also say it may not
                    even be a unit test, since it uses external state (touching disk, ordering
                    may matter, database use perhaps, threading)
                    A reverse look may help here.
                    In a well designed system, a *unit test* should be easy to write such that
                    it always returns the same result. The system is easy to initialize in code
                    such that any external state is decoupled enough from the object under test
                    so that the external results of running the tests are easily reversible, or
                    non existant at all (non existent may be the true sign of a*unit test*,
                    since a database related test is an integration test in my mind).

                    so, to conclude,I think that a non consistent test result is a sign that you
                    have to manage some external state, or that your test is not really a unit
                    test. Of course, you can *always* work badly enough so that a test fails
                    just because it did not initialize or tear down an object correctly, but, in
                    my experience, that is is a small fraction of real unit tests behavior, and
                    is *easily* fixable. On the other hand, fixing the same situation when you
                    ave external state not not usually so easy as just dropping a new setup
                    method..

                    Roy.


                    On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                    >
                    > Roy,
                    >
                    > Are you saying that having the same test sometimes fail and sometimes pass
                    > (without any code changes) is a property of an untestable system? Could it
                    > not sometimes be due to a poorly written test?
                    >
                    > Why could it then not also be a property of some untestable systems that a
                    > failure of some reasonable unit test cannot be attributed to any single
                    > cause?
                    >
                    > It seems to me either property could be caused by either a poorly written
                    > unit test or a system that is too monolithic to be unit-testable.
                    >
                    > Steve
                    >
                    > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                    > >
                    > > Steven - yes. that's what I meant. If it fails, it will keep failing
                    > > indefinitely until you change the code(e.g break something else :-) )
                    > > As for it saying what failed - that's a unit testing best practice, and
                    > > people doing it or not does not depend on the health of the code base's
                    > > design. But easily writing tests that fill these requirement is a
                    > > testament
                    > > that the system is indeed "testable" - that is, it lends itself to be
                    > > tested
                    > > easily using unit tests that fill follow rules. Having a good fail
                    > message
                    > > in any of these tests has nothing to do with the system, but with the
                    > test
                    > > standards, which are a different story..
                    > > My definition here is that of the system under test, and one would know
                    > to
                    > > define a system as easily testable with unit tests or not.
                    > >
                    > > If we can conclude that these tests being easily writable with a system
                    > > with
                    > > specific properties, we can perhaps deduce from that many other things
                    > > which
                    > > may be related, I'm just not sure what they are yet...
                    > >
                    > > Roy.
                    > >
                    >
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >
                    > To Post a message, send it to: extremeprogramming@...
                    >
                    > To Unsubscribe, send a blank message to:
                    > extremeprogramming-unsubscribe@...
                    >
                    > ad-free courtesy of objectmentor.com
                    > Yahoo! Groups Links
                    >
                    >
                    >
                    >
                    >
                    >
                    >


                    --
                    Thanks,

                    Roy Osherove
                    http://www.iserializable.com


                    [Non-text portions of this message have been removed]
                  • Charlie Poole
                    Hi Roy, So we should have both high quality system design and high quality tests! ... In my experience, poor tests causing inconsistent results are usually
                    Message 9 of 24 , Feb 13, 2006
                    • 0 Attachment
                      Hi Roy,

                      So we should have both high quality system design and high quality tests!
                      :-)

                      In my experience, poor tests causing inconsistent results are usually
                      suffering from local coding problems - like retained test object state
                      across cases. These are relatively easy to fix.

                      The problems caused by design are broader - and much more interesting.

                      Charlie

                      > -----Original Message-----
                      > From: extremeprogramming@yahoogroups.com
                      > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                      > Sent: Sunday, February 12, 2006 10:51 PM
                      > To: extremeprogramming@yahoogroups.com
                      > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                      >
                      > Steve, Obviously, a poor test that sometimes fails and
                      > sometimes does not could be attributed to a bad test, but an
                      > inconsistent test result also usually means (at least in my
                      > experience) that the state under test is not cleaned up or
                      > initialized between the test runs (such as object state not
                      > reset, database state not reset). That usually is a sign that
                      > the test has to manage external state, and does a poor job of that.
                      > You could then perhaps say that the test is poor quality, but
                      > that a high quality test for such a system, such that it
                      > always returns a fail pass result is hard to write, since one
                      > has to manage external state, which goes against the
                      > definitions I've laid thus far. You could also say it may not
                      > even be a unit test, since it uses external state (touching
                      > disk, ordering may matter, database use perhaps, threading) A
                      > reverse look may help here.
                      > In a well designed system, a *unit test* should be easy to
                      > write such that it always returns the same result. The system
                      > is easy to initialize in code such that any external state is
                      > decoupled enough from the object under test so that the
                      > external results of running the tests are easily reversible,
                      > or non existant at all (non existent may be the true sign of
                      > a*unit test*, since a database related test is an
                      > integration test in my mind).
                      >
                      > so, to conclude,I think that a non consistent test result is
                      > a sign that you have to manage some external state, or that
                      > your test is not really a unit test. Of course, you can
                      > *always* work badly enough so that a test fails just because
                      > it did not initialize or tear down an object correctly, but,
                      > in my experience, that is is a small fraction of real unit
                      > tests behavior, and is *easily* fixable. On the other hand,
                      > fixing the same situation when you ave external state not not
                      > usually so easy as just dropping a new setup method..
                      >
                      > Roy.
                      >
                      >
                      > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                      > >
                      > > Roy,
                      > >
                      > > Are you saying that having the same test sometimes fail and
                      > sometimes
                      > > pass (without any code changes) is a property of an
                      > untestable system?
                      > > Could it not sometimes be due to a poorly written test?
                      > >
                      > > Why could it then not also be a property of some untestable systems
                      > > that a failure of some reasonable unit test cannot be attributed to
                      > > any single cause?
                      > >
                      > > It seems to me either property could be caused by either a poorly
                      > > written unit test or a system that is too monolithic to be
                      > unit-testable.
                      > >
                      > > Steve
                      > >
                      > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                      > > >
                      > > > Steven - yes. that's what I meant. If it fails, it will
                      > keep failing
                      > > > indefinitely until you change the code(e.g break
                      > something else :-)
                      > > > ) As for it saying what failed - that's a unit testing best
                      > > > practice, and people doing it or not does not depend on
                      > the health
                      > > > of the code base's design. But easily writing tests that
                      > fill these
                      > > > requirement is a testament that the system is indeed "testable" -
                      > > > that is, it lends itself to be tested easily using unit
                      > tests that
                      > > > fill follow rules. Having a good fail
                      > > message
                      > > > in any of these tests has nothing to do with the system, but with
                      > > > the
                      > > test
                      > > > standards, which are a different story..
                      > > > My definition here is that of the system under test, and
                      > one would
                      > > > know
                      > > to
                      > > > define a system as easily testable with unit tests or not.
                      > > >
                      > > > If we can conclude that these tests being easily writable with a
                      > > > system with specific properties, we can perhaps deduce from that
                      > > > many other things which may be related, I'm just not sure
                      > what they
                      > > > are yet...
                      > > >
                      > > > Roy.
                      > > >
                      > >
                      > >
                      > > [Non-text portions of this message have been removed]
                      > >
                      > >
                      > >
                      > > To Post a message, send it to: extremeprogramming@...
                      > >
                      > > To Unsubscribe, send a blank message to:
                      > > extremeprogramming-unsubscribe@...
                      > >
                      > > ad-free courtesy of objectmentor.com
                      > > Yahoo! Groups Links
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      >
                      >
                      > --
                      > Thanks,
                      >
                      > Roy Osherove
                      > http://www.iserializable.com
                      >
                      >
                      > [Non-text portions of this message have been removed]
                      >
                      >
                      >
                      > To Post a message, send it to: extremeprogramming@...
                      >
                      > To Unsubscribe, send a blank message to:
                      > extremeprogramming-unsubscribe@...
                      >
                      > ad-free courtesy of objectmentor.com
                      > Yahoo! Groups Links
                      >
                      >
                      >
                      >
                      >
                      >
                      >
                      >
                    • Dale Emery
                      Hi Roy, ... I d say: Test results are repeatable. Dale -- Dale Emery, Consultant Inspiring Leadership for Software People Web: http://www.dhemery.com
                      Message 10 of 24 , Feb 13, 2006
                      • 0 Attachment
                        Hi Roy,

                        > Steven - yes. that's what I meant. If it fails, it will keep
                        > failing indefinitely until you change the code(e.g break
                        > something else :-) )

                        I'd say: Test results are repeatable.

                        Dale

                        --
                        Dale Emery, Consultant
                        Inspiring Leadership for Software People
                        Web: http://www.dhemery.com
                        Weblog: http://www.dhemery.com/cwd
                      • Dale Emery
                        Hi Steve, ... Probably most X-able attributes have that property: The x-ability comes from the relationship between the X-er and the thing being X-ed. So what
                        Message 11 of 24 , Feb 13, 2006
                        • 0 Attachment
                          Hi Steve,

                          > It seems to me either property could be caused by either a
                          > poorly written unit test or a system that is too monolithic to
                          > be unit-testable.

                          Probably most X-able attributes have that property: The
                          x-ability comes from the relationship between the X-er and the
                          thing being X-ed.

                          So what are the attributes of software that make it easier or
                          harder to write tests that are independent, repeatable, and fast?

                          I generally think of testability as having two components:
                          control and visibility. Control over the variables that affect
                          the software's behavior, and visibility into the variables that
                          the software affects. I'm using "variables" not in a programming
                          syntax sense, but in a lay sense. A variable is anything that
                          can vary.

                          So what makes it easier or harder to control and observe the
                          variables I'm interested in? Thinking out loud... the amount of
                          code I have to write in order to ensure that a variable of
                          interest has the value I want at the moment of the test; the
                          amount of code I have to write to observe a variable of interest.
                          And if controlling and observing variables requires manual
                          action, the unit testing score plummets.

                          > Are you saying that having the same test sometimes fail and
                          > sometimes pass (without any code changes) is a property of an
                          > untestable system? Could it not sometimes be due to a poorly
                          > written test?

                          If tests results are not repeatable, then there is some important
                          variable that isn't being controlled.

                          Could be the code; could be the test. So my question (which I
                          haven't yet thought through) is: How much code do I have to
                          write in order to control that variable?

                          Another half-baked question: How come I didn't know about that
                          variable, or that it was important? Again, this oversight could
                          be about the code (it doesn't make explicit the variables to
                          which it responds) or about the tester.

                          > Why could it then not also be a property of some untestable
                          > systems that a failure of some reasonable unit test cannot be
                          > attributed to any single cause?

                          If the test (in the context of other tests) does not clearly and
                          correctly identify the faulty unit, it seems to me that it is not
                          a unit test.

                          So what attributes of software make it easier or harder to write
                          tests that clearly and correctly identify the faulty unit?

                          Dale

                          --
                          Dale Emery, Consultant
                          Inspiring Leadership for Software People
                          Web: http://www.dhemery.com
                          Weblog: http://www.dhemery.com/cwd
                        • Roy Osherove
                          Dale - you re right up there inside my head. The definition states explicitly that the test that posses these rules, can be written *reasonably easily*, which
                          Message 12 of 24 , Feb 13, 2006
                          • 0 Attachment
                            Dale - you're right up there inside my head. The definition states
                            explicitly that the test that posses these rules, can be written *reasonably
                            easily*, which goes right up there with "how much code do I need to write to
                            control that variable". They also need visibility since otherwise they
                            wouldn't be easy to write.
                            I've seen many places that work *really really hard* to make their "unit
                            tests" run while matching these rules, but a true testable system will allow
                            you to adhere to those rules easily.

                            Roy.


                            On 2/13/06, Dale Emery <dale@...> wrote:
                            >
                            > Hi Steve,
                            >
                            > > It seems to me either property could be caused by either a
                            > > poorly written unit test or a system that is too monolithic to
                            > > be unit-testable.
                            >
                            > Probably most X-able attributes have that property: The
                            > x-ability comes from the relationship between the X-er and the
                            > thing being X-ed.
                            >
                            > So what are the attributes of software that make it easier or
                            > harder to write tests that are independent, repeatable, and fast?
                            >
                            > I generally think of testability as having two components:
                            > control and visibility. Control over the variables that affect
                            > the software's behavior, and visibility into the variables that
                            > the software affects. I'm using "variables" not in a programming
                            > syntax sense, but in a lay sense. A variable is anything that
                            > can vary.
                            >
                            > So what makes it easier or harder to control and observe the
                            > variables I'm interested in? Thinking out loud... the amount of
                            > code I have to write in order to ensure that a variable of
                            > interest has the value I want at the moment of the test; the
                            > amount of code I have to write to observe a variable of interest.
                            > And if controlling and observing variables requires manual
                            > action, the unit testing score plummets.
                            >
                            > > Are you saying that having the same test sometimes fail and
                            > > sometimes pass (without any code changes) is a property of an
                            > > untestable system? Could it not sometimes be due to a poorly
                            > > written test?
                            >
                            > If tests results are not repeatable, then there is some important
                            > variable that isn't being controlled.
                            >
                            > Could be the code; could be the test. So my question (which I
                            > haven't yet thought through) is: How much code do I have to
                            > write in order to control that variable?
                            >
                            > Another half-baked question: How come I didn't know about that
                            > variable, or that it was important? Again, this oversight could
                            > be about the code (it doesn't make explicit the variables to
                            > which it responds) or about the tester.
                            >
                            > > Why could it then not also be a property of some untestable
                            > > systems that a failure of some reasonable unit test cannot be
                            > > attributed to any single cause?
                            >
                            > If the test (in the context of other tests) does not clearly and
                            > correctly identify the faulty unit, it seems to me that it is not
                            > a unit test.
                            >
                            > So what attributes of software make it easier or harder to write
                            > tests that clearly and correctly identify the faulty unit?
                            >
                            > Dale
                            >
                            > --
                            > Dale Emery, Consultant
                            > Inspiring Leadership for Software People
                            > Web: http://www.dhemery.com
                            > Weblog: http://www.dhemery.com/cwd
                            >
                            >
                            > To Post a message, send it to: extremeprogramming@...
                            >
                            > To Unsubscribe, send a blank message to:
                            > extremeprogramming-unsubscribe@...
                            >
                            > ad-free courtesy of objectmentor.com
                            > Yahoo! Groups Links
                            >
                            >
                            >
                            >
                            >
                            >
                            >


                            --
                            Thanks,

                            Roy Osherove
                            http://www.iserializable.com


                            [Non-text portions of this message have been removed]
                          • Roy Osherove
                            I m not saying test cod quality is not important, Charlie. I m saying that regardless of the test quality, if one knew how to write good tests, one would find
                            Message 13 of 24 , Feb 13, 2006
                            • 0 Attachment
                              I'm not saying test cod quality is not important, Charlie. I'm saying that
                              regardless of the test quality, if one knew how to write good tests, one
                              would find that writing them while following the PCCOF rules much much
                              easier than in a system full of, legacy code, perhaps.
                              (TDD is out of the topic here since it clearly forces you into the PCCOF
                              rules, IMO)


                              On 2/13/06, Charlie Poole <xp@...> wrote:
                              >
                              > Hi Roy,
                              >
                              > So we should have both high quality system design and high quality tests!
                              > :-)
                              >
                              > In my experience, poor tests causing inconsistent results are usually
                              > suffering from local coding problems - like retained test object state
                              > across cases. These are relatively easy to fix.
                              >
                              > The problems caused by design are broader - and much more interesting.
                              >
                              > Charlie
                              >
                              > > -----Original Message-----
                              > > From: extremeprogramming@yahoogroups.com
                              > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                              > > Sent: Sunday, February 12, 2006 10:51 PM
                              > > To: extremeprogramming@yahoogroups.com
                              > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                              > >
                              > > Steve, Obviously, a poor test that sometimes fails and
                              > > sometimes does not could be attributed to a bad test, but an
                              > > inconsistent test result also usually means (at least in my
                              > > experience) that the state under test is not cleaned up or
                              > > initialized between the test runs (such as object state not
                              > > reset, database state not reset). That usually is a sign that
                              > > the test has to manage external state, and does a poor job of that.
                              > > You could then perhaps say that the test is poor quality, but
                              > > that a high quality test for such a system, such that it
                              > > always returns a fail pass result is hard to write, since one
                              > > has to manage external state, which goes against the
                              > > definitions I've laid thus far. You could also say it may not
                              > > even be a unit test, since it uses external state (touching
                              > > disk, ordering may matter, database use perhaps, threading) A
                              > > reverse look may help here.
                              > > In a well designed system, a *unit test* should be easy to
                              > > write such that it always returns the same result. The system
                              > > is easy to initialize in code such that any external state is
                              > > decoupled enough from the object under test so that the
                              > > external results of running the tests are easily reversible,
                              > > or non existant at all (non existent may be the true sign of
                              > > a*unit test*, since a database related test is an
                              > > integration test in my mind).
                              > >
                              > > so, to conclude,I think that a non consistent test result is
                              > > a sign that you have to manage some external state, or that
                              > > your test is not really a unit test. Of course, you can
                              > > *always* work badly enough so that a test fails just because
                              > > it did not initialize or tear down an object correctly, but,
                              > > in my experience, that is is a small fraction of real unit
                              > > tests behavior, and is *easily* fixable. On the other hand,
                              > > fixing the same situation when you ave external state not not
                              > > usually so easy as just dropping a new setup method..
                              > >
                              > > Roy.
                              > >
                              > >
                              > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                              > > >
                              > > > Roy,
                              > > >
                              > > > Are you saying that having the same test sometimes fail and
                              > > sometimes
                              > > > pass (without any code changes) is a property of an
                              > > untestable system?
                              > > > Could it not sometimes be due to a poorly written test?
                              > > >
                              > > > Why could it then not also be a property of some untestable systems
                              > > > that a failure of some reasonable unit test cannot be attributed to
                              > > > any single cause?
                              > > >
                              > > > It seems to me either property could be caused by either a poorly
                              > > > written unit test or a system that is too monolithic to be
                              > > unit-testable.
                              > > >
                              > > > Steve
                              > > >
                              > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                              > > > >
                              > > > > Steven - yes. that's what I meant. If it fails, it will
                              > > keep failing
                              > > > > indefinitely until you change the code(e.g break
                              > > something else :-)
                              > > > > ) As for it saying what failed - that's a unit testing best
                              > > > > practice, and people doing it or not does not depend on
                              > > the health
                              > > > > of the code base's design. But easily writing tests that
                              > > fill these
                              > > > > requirement is a testament that the system is indeed "testable" -
                              > > > > that is, it lends itself to be tested easily using unit
                              > > tests that
                              > > > > fill follow rules. Having a good fail
                              > > > message
                              > > > > in any of these tests has nothing to do with the system, but with
                              > > > > the
                              > > > test
                              > > > > standards, which are a different story..
                              > > > > My definition here is that of the system under test, and
                              > > one would
                              > > > > know
                              > > > to
                              > > > > define a system as easily testable with unit tests or not.
                              > > > >
                              > > > > If we can conclude that these tests being easily writable with a
                              > > > > system with specific properties, we can perhaps deduce from that
                              > > > > many other things which may be related, I'm just not sure
                              > > what they
                              > > > > are yet...
                              > > > >
                              > > > > Roy.
                              > > > >
                              > > >
                              > > >
                              > > > [Non-text portions of this message have been removed]
                              > > >
                              > > >
                              > > >
                              > > > To Post a message, send it to: extremeprogramming@...
                              > > >
                              > > > To Unsubscribe, send a blank message to:
                              > > > extremeprogramming-unsubscribe@...
                              > > >
                              > > > ad-free courtesy of objectmentor.com
                              > > > Yahoo! Groups Links
                              > > >
                              > > >
                              > > >
                              > > >
                              > > >
                              > > >
                              > > >
                              > >
                              > >
                              > > --
                              > > Thanks,
                              > >
                              > > Roy Osherove
                              > > http://www.iserializable.com
                              > >
                              > >
                              > > [Non-text portions of this message have been removed]
                              > >
                              > >
                              > >
                              > > To Post a message, send it to: extremeprogramming@...
                              > >
                              > > To Unsubscribe, send a blank message to:
                              > > extremeprogramming-unsubscribe@...
                              > >
                              > > ad-free courtesy of objectmentor.com
                              > > Yahoo! Groups Links
                              > >
                              > >
                              > >
                              > >
                              > >
                              > >
                              > >
                              > >
                              >
                              >
                              >
                              >
                              > To Post a message, send it to: extremeprogramming@...
                              >
                              > To Unsubscribe, send a blank message to:
                              > extremeprogramming-unsubscribe@...
                              >
                              > ad-free courtesy of objectmentor.com
                              > Yahoo! Groups Links
                              >
                              >
                              >
                              >
                              >
                              >
                              >


                              --
                              Thanks,

                              Roy Osherove
                              http://www.iserializable.com


                              [Non-text portions of this message have been removed]
                            • Roy Osherove
                              Charlie, What other problems (for the test writer) do you see that I haven t touched? ... -- Thanks, Roy Osherove http://www.iserializable.com [Non-text
                              Message 14 of 24 , Feb 13, 2006
                              • 0 Attachment
                                Charlie, What other problems (for the test writer) do you see that I haven't
                                touched?

                                On 2/13/06, Charlie Poole <xp@...> wrote:
                                >
                                > Hi Roy,
                                >
                                > So we should have both high quality system design and high quality tests!
                                > :-)
                                >
                                > In my experience, poor tests causing inconsistent results are usually
                                > suffering from local coding problems - like retained test object state
                                > across cases. These are relatively easy to fix.
                                >
                                > The problems caused by design are broader - and much more interesting.
                                >
                                > Charlie
                                >
                                > > -----Original Message-----
                                > > From: extremeprogramming@yahoogroups.com
                                > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                                > > Sent: Sunday, February 12, 2006 10:51 PM
                                > > To: extremeprogramming@yahoogroups.com
                                > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                > >
                                > > Steve, Obviously, a poor test that sometimes fails and
                                > > sometimes does not could be attributed to a bad test, but an
                                > > inconsistent test result also usually means (at least in my
                                > > experience) that the state under test is not cleaned up or
                                > > initialized between the test runs (such as object state not
                                > > reset, database state not reset). That usually is a sign that
                                > > the test has to manage external state, and does a poor job of that.
                                > > You could then perhaps say that the test is poor quality, but
                                > > that a high quality test for such a system, such that it
                                > > always returns a fail pass result is hard to write, since one
                                > > has to manage external state, which goes against the
                                > > definitions I've laid thus far. You could also say it may not
                                > > even be a unit test, since it uses external state (touching
                                > > disk, ordering may matter, database use perhaps, threading) A
                                > > reverse look may help here.
                                > > In a well designed system, a *unit test* should be easy to
                                > > write such that it always returns the same result. The system
                                > > is easy to initialize in code such that any external state is
                                > > decoupled enough from the object under test so that the
                                > > external results of running the tests are easily reversible,
                                > > or non existant at all (non existent may be the true sign of
                                > > a*unit test*, since a database related test is an
                                > > integration test in my mind).
                                > >
                                > > so, to conclude,I think that a non consistent test result is
                                > > a sign that you have to manage some external state, or that
                                > > your test is not really a unit test. Of course, you can
                                > > *always* work badly enough so that a test fails just because
                                > > it did not initialize or tear down an object correctly, but,
                                > > in my experience, that is is a small fraction of real unit
                                > > tests behavior, and is *easily* fixable. On the other hand,
                                > > fixing the same situation when you ave external state not not
                                > > usually so easy as just dropping a new setup method..
                                > >
                                > > Roy.
                                > >
                                > >
                                > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                > > >
                                > > > Roy,
                                > > >
                                > > > Are you saying that having the same test sometimes fail and
                                > > sometimes
                                > > > pass (without any code changes) is a property of an
                                > > untestable system?
                                > > > Could it not sometimes be due to a poorly written test?
                                > > >
                                > > > Why could it then not also be a property of some untestable systems
                                > > > that a failure of some reasonable unit test cannot be attributed to
                                > > > any single cause?
                                > > >
                                > > > It seems to me either property could be caused by either a poorly
                                > > > written unit test or a system that is too monolithic to be
                                > > unit-testable.
                                > > >
                                > > > Steve
                                > > >
                                > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                > > > >
                                > > > > Steven - yes. that's what I meant. If it fails, it will
                                > > keep failing
                                > > > > indefinitely until you change the code(e.g break
                                > > something else :-)
                                > > > > ) As for it saying what failed - that's a unit testing best
                                > > > > practice, and people doing it or not does not depend on
                                > > the health
                                > > > > of the code base's design. But easily writing tests that
                                > > fill these
                                > > > > requirement is a testament that the system is indeed "testable" -
                                > > > > that is, it lends itself to be tested easily using unit
                                > > tests that
                                > > > > fill follow rules. Having a good fail
                                > > > message
                                > > > > in any of these tests has nothing to do with the system, but with
                                > > > > the
                                > > > test
                                > > > > standards, which are a different story..
                                > > > > My definition here is that of the system under test, and
                                > > one would
                                > > > > know
                                > > > to
                                > > > > define a system as easily testable with unit tests or not.
                                > > > >
                                > > > > If we can conclude that these tests being easily writable with a
                                > > > > system with specific properties, we can perhaps deduce from that
                                > > > > many other things which may be related, I'm just not sure
                                > > what they
                                > > > > are yet...
                                > > > >
                                > > > > Roy.
                                > > > >
                                > > >
                                > > >
                                > > > [Non-text portions of this message have been removed]
                                > > >
                                > > >
                                > > >
                                > > > To Post a message, send it to: extremeprogramming@...
                                > > >
                                > > > To Unsubscribe, send a blank message to:
                                > > > extremeprogramming-unsubscribe@...
                                > > >
                                > > > ad-free courtesy of objectmentor.com
                                > > > Yahoo! Groups Links
                                > > >
                                > > >
                                > > >
                                > > >
                                > > >
                                > > >
                                > > >
                                > >
                                > >
                                > > --
                                > > Thanks,
                                > >
                                > > Roy Osherove
                                > > http://www.iserializable.com
                                > >
                                > >
                                > > [Non-text portions of this message have been removed]
                                > >
                                > >
                                > >
                                > > To Post a message, send it to: extremeprogramming@...
                                > >
                                > > To Unsubscribe, send a blank message to:
                                > > extremeprogramming-unsubscribe@...
                                > >
                                > > ad-free courtesy of objectmentor.com
                                > > Yahoo! Groups Links
                                > >
                                > >
                                > >
                                > >
                                > >
                                > >
                                > >
                                > >
                                >
                                >
                                >
                                >
                                > To Post a message, send it to: extremeprogramming@...
                                >
                                > To Unsubscribe, send a blank message to:
                                > extremeprogramming-unsubscribe@...
                                >
                                > ad-free courtesy of objectmentor.com
                                > Yahoo! Groups Links
                                >
                                >
                                >
                                >
                                >
                                >
                                >


                                --
                                Thanks,

                                Roy Osherove
                                http://www.iserializable.com


                                [Non-text portions of this message have been removed]
                              • Steven Gordon
                                Roy, Perhaps, I was just not being clear, so I will try again: My understanding is that you are looking for unit-test behaviors that indicate that the system
                                Message 15 of 24 , Feb 13, 2006
                                • 0 Attachment
                                  Roy,

                                  Perhaps, I was just not being clear, so I will try again:

                                  My understanding is that you are looking for unit-test behaviors that
                                  indicate that the system under test is not unit-testable (assuming that the
                                  unit-tests were written reasonably well).

                                  Suppose our system had an object were a particular call could fail in the
                                  exact same way for many different reasons with no indication of the precise
                                  reason. There is no principled way to write the test with a specific
                                  failure message. Just saying "the call X.y() failed" is not good enough.
                                  Any specific message might not be true some of the time.

                                  This seems to me to be a critical property of a unit-testable system.

                                  Steve


                                  > > > -----Original Message-----
                                  > > > From: extremeprogramming@yahoogroups.com
                                  > > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                                  > > > Sent: Sunday, February 12, 2006 10:51 PM
                                  > > > To: extremeprogramming@yahoogroups.com
                                  > > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                  > > >
                                  > > > Steve, Obviously, a poor test that sometimes fails and
                                  > > > sometimes does not could be attributed to a bad test, but an
                                  > > > inconsistent test result also usually means (at least in my
                                  > > > experience) that the state under test is not cleaned up or
                                  > > > initialized between the test runs (such as object state not
                                  > > > reset, database state not reset). That usually is a sign that
                                  > > > the test has to manage external state, and does a poor job of that.
                                  > > > You could then perhaps say that the test is poor quality, but
                                  > > > that a high quality test for such a system, such that it
                                  > > > always returns a fail pass result is hard to write, since one
                                  > > > has to manage external state, which goes against the
                                  > > > definitions I've laid thus far. You could also say it may not
                                  > > > even be a unit test, since it uses external state (touching
                                  > > > disk, ordering may matter, database use perhaps, threading) A
                                  > > > reverse look may help here.
                                  > > > In a well designed system, a *unit test* should be easy to
                                  > > > write such that it always returns the same result. The system
                                  > > > is easy to initialize in code such that any external state is
                                  > > > decoupled enough from the object under test so that the
                                  > > > external results of running the tests are easily reversible,
                                  > > > or non existant at all (non existent may be the true sign of
                                  > > > a*unit test*, since a database related test is an
                                  > > > integration test in my mind).
                                  > > >
                                  > > > so, to conclude,I think that a non consistent test result is
                                  > > > a sign that you have to manage some external state, or that
                                  > > > your test is not really a unit test. Of course, you can
                                  > > > *always* work badly enough so that a test fails just because
                                  > > > it did not initialize or tear down an object correctly, but,
                                  > > > in my experience, that is is a small fraction of real unit
                                  > > > tests behavior, and is *easily* fixable. On the other hand,
                                  > > > fixing the same situation when you ave external state not not
                                  > > > usually so easy as just dropping a new setup method..
                                  > > >
                                  > > > Roy.
                                  > > >
                                  > > >
                                  > > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                  > > > >
                                  > > > > Roy,
                                  > > > >
                                  > > > > Are you saying that having the same test sometimes fail and
                                  > > > sometimes
                                  > > > > pass (without any code changes) is a property of an
                                  > > > untestable system?
                                  > > > > Could it not sometimes be due to a poorly written test?
                                  > > > >
                                  > > > > Why could it then not also be a property of some untestable systems
                                  > > > > that a failure of some reasonable unit test cannot be attributed to
                                  > > > > any single cause?
                                  > > > >
                                  > > > > It seems to me either property could be caused by either a poorly
                                  > > > > written unit test or a system that is too monolithic to be
                                  > > > unit-testable.
                                  > > > >
                                  > > > > Steve
                                  > > > >
                                  > > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                  > > > > >
                                  > > > > > Steven - yes. that's what I meant. If it fails, it will
                                  > > > keep failing
                                  > > > > > indefinitely until you change the code(e.g break
                                  > > > something else :-)
                                  > > > > > ) As for it saying what failed - that's a unit testing best
                                  > > > > > practice, and people doing it or not does not depend on
                                  > > > the health
                                  > > > > > of the code base's design. But easily writing tests that
                                  > > > fill these
                                  > > > > > requirement is a testament that the system is indeed "testable" -
                                  > > > > > that is, it lends itself to be tested easily using unit
                                  > > > tests that
                                  > > > > > fill follow rules. Having a good fail
                                  > > > > message
                                  > > > > > in any of these tests has nothing to do with the system, but with
                                  > > > > > the
                                  > > > > test
                                  > > > > > standards, which are a different story..
                                  > > > > > My definition here is that of the system under test, and
                                  > > > one would
                                  > > > > > know
                                  > > > > to
                                  > > > > > define a system as easily testable with unit tests or not.
                                  > > > > >
                                  > > > > > If we can conclude that these tests being easily writable with a
                                  > > > > > system with specific properties, we can perhaps deduce from that
                                  > > > > > many other things which may be related, I'm just not sure
                                  > > > what they
                                  > > > > > are yet...
                                  > > > > >
                                  > > > > > Roy.
                                  > > > > >
                                  > > > >
                                  > > > >
                                  > > > > [Non-text portions of this message have been removed]
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > > > To Post a message, send it to: extremeprogramming@...
                                  > > > >
                                  > > > > To Unsubscribe, send a blank message to:
                                  > > > > extremeprogramming-unsubscribe@...
                                  > > > >
                                  > > > > ad-free courtesy of objectmentor.com
                                  > > > > Yahoo! Groups Links
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > > >
                                  > > >
                                  > > >
                                  > > > --
                                  > > > Thanks,
                                  > > >
                                  > > > Roy Osherove
                                  > > > http://www.iserializable.com
                                  > > >
                                  > > >
                                  > > > [Non-text portions of this message have been removed]
                                  > > >
                                  > > >
                                  > > >
                                  > > > To Post a message, send it to: extremeprogramming@...
                                  > > >
                                  > > > To Unsubscribe, send a blank message to:
                                  > > > extremeprogramming-unsubscribe@...
                                  > > >
                                  > > > ad-free courtesy of objectmentor.com
                                  > > > Yahoo! Groups Links
                                  > > >
                                  > > >
                                  > > >
                                  > > >
                                  > > >
                                  > > >
                                  > > >
                                  > > >
                                  > >
                                  > >
                                  > >
                                  > >
                                  > > To Post a message, send it to: extremeprogramming@...
                                  > >
                                  > > To Unsubscribe, send a blank message to:
                                  > > extremeprogramming-unsubscribe@...
                                  > >
                                  > > ad-free courtesy of objectmentor.com
                                  > > Yahoo! Groups Links
                                  > >
                                  > >
                                  > >
                                  > >
                                  > >
                                  > >
                                  > >
                                  >
                                  >
                                  > --
                                  > Thanks,
                                  >
                                  > Roy Osherove
                                  > http://www.iserializable.com
                                  >
                                  >
                                  > [Non-text portions of this message have been removed]
                                  >
                                  >
                                  >
                                  > To Post a message, send it to: extremeprogramming@...
                                  >
                                  > To Unsubscribe, send a blank message to:
                                  > extremeprogramming-unsubscribe@...
                                  >
                                  > ad-free courtesy of objectmentor.com
                                  > Yahoo! Groups Links
                                  >
                                  >
                                  >
                                  >
                                  >
                                  >
                                  >


                                  [Non-text portions of this message have been removed]
                                • Charlie Poole
                                  Hi Roy, I was agreeing with you. I ll have to work on that. :-) Charlie
                                  Message 16 of 24 , Feb 13, 2006
                                  • 0 Attachment
                                    Hi Roy,

                                    I was agreeing with you. I'll have to work on that. :-)

                                    Charlie

                                    > -----Original Message-----
                                    > From: extremeprogramming@yahoogroups.com
                                    > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                                    > Sent: Monday, February 13, 2006 2:27 PM
                                    > To: extremeprogramming@yahoogroups.com
                                    > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                    >
                                    > I'm not saying test cod quality is not important, Charlie.
                                    > I'm saying that regardless of the test quality, if one knew
                                    > how to write good tests, one would find that writing them
                                    > while following the PCCOF rules much much easier than in a
                                    > system full of, legacy code, perhaps.
                                    > (TDD is out of the topic here since it clearly forces you
                                    > into the PCCOF rules, IMO)
                                    >
                                    >
                                    > On 2/13/06, Charlie Poole <xp@...> wrote:
                                    > >
                                    > > Hi Roy,
                                    > >
                                    > > So we should have both high quality system design and high
                                    > quality tests!
                                    > > :-)
                                    > >
                                    > > In my experience, poor tests causing inconsistent results
                                    > are usually
                                    > > suffering from local coding problems - like retained test
                                    > object state
                                    > > across cases. These are relatively easy to fix.
                                    > >
                                    > > The problems caused by design are broader - and much more
                                    > interesting.
                                    > >
                                    > > Charlie
                                    > >
                                    > > > -----Original Message-----
                                    > > > From: extremeprogramming@yahoogroups.com
                                    > > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy
                                    > > > Osherove
                                    > > > Sent: Sunday, February 12, 2006 10:51 PM
                                    > > > To: extremeprogramming@yahoogroups.com
                                    > > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                    > > >
                                    > > > Steve, Obviously, a poor test that sometimes fails and sometimes
                                    > > > does not could be attributed to a bad test, but an
                                    > inconsistent test
                                    > > > result also usually means (at least in my
                                    > > > experience) that the state under test is not cleaned up or
                                    > > > initialized between the test runs (such as object state
                                    > not reset,
                                    > > > database state not reset). That usually is a sign that
                                    > the test has
                                    > > > to manage external state, and does a poor job of that.
                                    > > > You could then perhaps say that the test is poor quality,
                                    > but that a
                                    > > > high quality test for such a system, such that it always
                                    > returns a
                                    > > > fail pass result is hard to write, since one has to
                                    > manage external
                                    > > > state, which goes against the definitions I've laid thus far. You
                                    > > > could also say it may not even be a unit test, since it uses
                                    > > > external state (touching disk, ordering may matter, database use
                                    > > > perhaps, threading) A reverse look may help here.
                                    > > > In a well designed system, a *unit test* should be easy to write
                                    > > > such that it always returns the same result. The system
                                    > is easy to
                                    > > > initialize in code such that any external state is
                                    > decoupled enough
                                    > > > from the object under test so that the external results
                                    > of running
                                    > > > the tests are easily reversible, or non existant at all (non
                                    > > > existent may be the true sign of a*unit test*, since a database
                                    > > > related test is an integration test in my mind).
                                    > > >
                                    > > > so, to conclude,I think that a non consistent test result
                                    > is a sign
                                    > > > that you have to manage some external state, or that your test is
                                    > > > not really a unit test. Of course, you can
                                    > > > *always* work badly enough so that a test fails just
                                    > because it did
                                    > > > not initialize or tear down an object correctly, but, in my
                                    > > > experience, that is is a small fraction of real unit
                                    > tests behavior,
                                    > > > and is *easily* fixable. On the other hand, fixing the same
                                    > > > situation when you ave external state not not usually so easy as
                                    > > > just dropping a new setup method..
                                    > > >
                                    > > > Roy.
                                    > > >
                                    > > >
                                    > > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                    > > > >
                                    > > > > Roy,
                                    > > > >
                                    > > > > Are you saying that having the same test sometimes fail and
                                    > > > sometimes
                                    > > > > pass (without any code changes) is a property of an
                                    > > > untestable system?
                                    > > > > Could it not sometimes be due to a poorly written test?
                                    > > > >
                                    > > > > Why could it then not also be a property of some untestable
                                    > > > > systems that a failure of some reasonable unit test cannot be
                                    > > > > attributed to any single cause?
                                    > > > >
                                    > > > > It seems to me either property could be caused by
                                    > either a poorly
                                    > > > > written unit test or a system that is too monolithic to be
                                    > > > unit-testable.
                                    > > > >
                                    > > > > Steve
                                    > > > >
                                    > > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                    > > > > >
                                    > > > > > Steven - yes. that's what I meant. If it fails, it will
                                    > > > keep failing
                                    > > > > > indefinitely until you change the code(e.g break
                                    > > > something else :-)
                                    > > > > > ) As for it saying what failed - that's a unit testing best
                                    > > > > > practice, and people doing it or not does not depend on
                                    > > > the health
                                    > > > > > of the code base's design. But easily writing tests that
                                    > > > fill these
                                    > > > > > requirement is a testament that the system is indeed
                                    > "testable"
                                    > > > > > - that is, it lends itself to be tested easily using unit
                                    > > > tests that
                                    > > > > > fill follow rules. Having a good fail
                                    > > > > message
                                    > > > > > in any of these tests has nothing to do with the system, but
                                    > > > > > with the
                                    > > > > test
                                    > > > > > standards, which are a different story..
                                    > > > > > My definition here is that of the system under test, and
                                    > > > one would
                                    > > > > > know
                                    > > > > to
                                    > > > > > define a system as easily testable with unit tests or not.
                                    > > > > >
                                    > > > > > If we can conclude that these tests being easily
                                    > writable with a
                                    > > > > > system with specific properties, we can perhaps
                                    > deduce from that
                                    > > > > > many other things which may be related, I'm just not sure
                                    > > > what they
                                    > > > > > are yet...
                                    > > > > >
                                    > > > > > Roy.
                                    > > > > >
                                    > > > >
                                    > > > >
                                    > > > > [Non-text portions of this message have been removed]
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > > > To Post a message, send it to: extremeprogramming@...
                                    > > > >
                                    > > > > To Unsubscribe, send a blank message to:
                                    > > > > extremeprogramming-unsubscribe@...
                                    > > > >
                                    > > > > ad-free courtesy of objectmentor.com Yahoo! Groups Links
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > > >
                                    > > >
                                    > > >
                                    > > > --
                                    > > > Thanks,
                                    > > >
                                    > > > Roy Osherove
                                    > > > http://www.iserializable.com
                                    > > >
                                    > > >
                                    > > > [Non-text portions of this message have been removed]
                                    > > >
                                    > > >
                                    > > >
                                    > > > To Post a message, send it to: extremeprogramming@...
                                    > > >
                                    > > > To Unsubscribe, send a blank message to:
                                    > > > extremeprogramming-unsubscribe@...
                                    > > >
                                    > > > ad-free courtesy of objectmentor.com
                                    > > > Yahoo! Groups Links
                                    > > >
                                    > > >
                                    > > >
                                    > > >
                                    > > >
                                    > > >
                                    > > >
                                    > > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > > To Post a message, send it to: extremeprogramming@...
                                    > >
                                    > > To Unsubscribe, send a blank message to:
                                    > > extremeprogramming-unsubscribe@...
                                    > >
                                    > > ad-free courtesy of objectmentor.com
                                    > > Yahoo! Groups Links
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    > >
                                    >
                                    >
                                    > --
                                    > Thanks,
                                    >
                                    > Roy Osherove
                                    > http://www.iserializable.com
                                    >
                                    >
                                    > [Non-text portions of this message have been removed]
                                    >
                                    >
                                    >
                                    > To Post a message, send it to: extremeprogramming@...
                                    >
                                    > To Unsubscribe, send a blank message to:
                                    > extremeprogramming-unsubscribe@...
                                    >
                                    > ad-free courtesy of objectmentor.com
                                    > Yahoo! Groups Links
                                    >
                                    >
                                    >
                                    >
                                    >
                                    >
                                    >
                                  • Charlie Poole
                                    I think you re doing fine. :-)
                                    Message 17 of 24 , Feb 13, 2006
                                    • 0 Attachment
                                      I think you're doing fine. :-)

                                      > -----Original Message-----
                                      > From: extremeprogramming@yahoogroups.com
                                      > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                                      > Sent: Monday, February 13, 2006 2:29 PM
                                      > To: extremeprogramming@yahoogroups.com
                                      > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                      >
                                      > Charlie, What other problems (for the test writer) do you see
                                      > that I haven't touched?
                                      >
                                      > On 2/13/06, Charlie Poole <xp@...> wrote:
                                      > >
                                      > > Hi Roy,
                                      > >
                                      > > So we should have both high quality system design and high
                                      > quality tests!
                                      > > :-)
                                      > >
                                      > > In my experience, poor tests causing inconsistent results
                                      > are usually
                                      > > suffering from local coding problems - like retained test
                                      > object state
                                      > > across cases. These are relatively easy to fix.
                                      > >
                                      > > The problems caused by design are broader - and much more
                                      > interesting.
                                      > >
                                      > > Charlie
                                      > >
                                      > > > -----Original Message-----
                                      > > > From: extremeprogramming@yahoogroups.com
                                      > > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy
                                      > > > Osherove
                                      > > > Sent: Sunday, February 12, 2006 10:51 PM
                                      > > > To: extremeprogramming@yahoogroups.com
                                      > > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                      > > >
                                      > > > Steve, Obviously, a poor test that sometimes fails and sometimes
                                      > > > does not could be attributed to a bad test, but an
                                      > inconsistent test
                                      > > > result also usually means (at least in my
                                      > > > experience) that the state under test is not cleaned up or
                                      > > > initialized between the test runs (such as object state
                                      > not reset,
                                      > > > database state not reset). That usually is a sign that
                                      > the test has
                                      > > > to manage external state, and does a poor job of that.
                                      > > > You could then perhaps say that the test is poor quality,
                                      > but that a
                                      > > > high quality test for such a system, such that it always
                                      > returns a
                                      > > > fail pass result is hard to write, since one has to
                                      > manage external
                                      > > > state, which goes against the definitions I've laid thus far. You
                                      > > > could also say it may not even be a unit test, since it uses
                                      > > > external state (touching disk, ordering may matter, database use
                                      > > > perhaps, threading) A reverse look may help here.
                                      > > > In a well designed system, a *unit test* should be easy to write
                                      > > > such that it always returns the same result. The system
                                      > is easy to
                                      > > > initialize in code such that any external state is
                                      > decoupled enough
                                      > > > from the object under test so that the external results
                                      > of running
                                      > > > the tests are easily reversible, or non existant at all (non
                                      > > > existent may be the true sign of a*unit test*, since a database
                                      > > > related test is an integration test in my mind).
                                      > > >
                                      > > > so, to conclude,I think that a non consistent test result
                                      > is a sign
                                      > > > that you have to manage some external state, or that your test is
                                      > > > not really a unit test. Of course, you can
                                      > > > *always* work badly enough so that a test fails just
                                      > because it did
                                      > > > not initialize or tear down an object correctly, but, in my
                                      > > > experience, that is is a small fraction of real unit
                                      > tests behavior,
                                      > > > and is *easily* fixable. On the other hand, fixing the same
                                      > > > situation when you ave external state not not usually so easy as
                                      > > > just dropping a new setup method..
                                      > > >
                                      > > > Roy.
                                      > > >
                                      > > >
                                      > > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                      > > > >
                                      > > > > Roy,
                                      > > > >
                                      > > > > Are you saying that having the same test sometimes fail and
                                      > > > sometimes
                                      > > > > pass (without any code changes) is a property of an
                                      > > > untestable system?
                                      > > > > Could it not sometimes be due to a poorly written test?
                                      > > > >
                                      > > > > Why could it then not also be a property of some untestable
                                      > > > > systems that a failure of some reasonable unit test cannot be
                                      > > > > attributed to any single cause?
                                      > > > >
                                      > > > > It seems to me either property could be caused by
                                      > either a poorly
                                      > > > > written unit test or a system that is too monolithic to be
                                      > > > unit-testable.
                                      > > > >
                                      > > > > Steve
                                      > > > >
                                      > > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                      > > > > >
                                      > > > > > Steven - yes. that's what I meant. If it fails, it will
                                      > > > keep failing
                                      > > > > > indefinitely until you change the code(e.g break
                                      > > > something else :-)
                                      > > > > > ) As for it saying what failed - that's a unit testing best
                                      > > > > > practice, and people doing it or not does not depend on
                                      > > > the health
                                      > > > > > of the code base's design. But easily writing tests that
                                      > > > fill these
                                      > > > > > requirement is a testament that the system is indeed
                                      > "testable"
                                      > > > > > - that is, it lends itself to be tested easily using unit
                                      > > > tests that
                                      > > > > > fill follow rules. Having a good fail
                                      > > > > message
                                      > > > > > in any of these tests has nothing to do with the system, but
                                      > > > > > with the
                                      > > > > test
                                      > > > > > standards, which are a different story..
                                      > > > > > My definition here is that of the system under test, and
                                      > > > one would
                                      > > > > > know
                                      > > > > to
                                      > > > > > define a system as easily testable with unit tests or not.
                                      > > > > >
                                      > > > > > If we can conclude that these tests being easily
                                      > writable with a
                                      > > > > > system with specific properties, we can perhaps
                                      > deduce from that
                                      > > > > > many other things which may be related, I'm just not sure
                                      > > > what they
                                      > > > > > are yet...
                                      > > > > >
                                      > > > > > Roy.
                                      > > > > >
                                      > > > >
                                      > > > >
                                      > > > > [Non-text portions of this message have been removed]
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > > > To Post a message, send it to: extremeprogramming@...
                                      > > > >
                                      > > > > To Unsubscribe, send a blank message to:
                                      > > > > extremeprogramming-unsubscribe@...
                                      > > > >
                                      > > > > ad-free courtesy of objectmentor.com Yahoo! Groups Links
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > > >
                                      > > >
                                      > > >
                                      > > > --
                                      > > > Thanks,
                                      > > >
                                      > > > Roy Osherove
                                      > > > http://www.iserializable.com
                                      > > >
                                      > > >
                                      > > > [Non-text portions of this message have been removed]
                                      > > >
                                      > > >
                                      > > >
                                      > > > To Post a message, send it to: extremeprogramming@...
                                      > > >
                                      > > > To Unsubscribe, send a blank message to:
                                      > > > extremeprogramming-unsubscribe@...
                                      > > >
                                      > > > ad-free courtesy of objectmentor.com
                                      > > > Yahoo! Groups Links
                                      > > >
                                      > > >
                                      > > >
                                      > > >
                                      > > >
                                      > > >
                                      > > >
                                      > > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > > To Post a message, send it to: extremeprogramming@...
                                      > >
                                      > > To Unsubscribe, send a blank message to:
                                      > > extremeprogramming-unsubscribe@...
                                      > >
                                      > > ad-free courtesy of objectmentor.com
                                      > > Yahoo! Groups Links
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      > >
                                      >
                                      >
                                      > --
                                      > Thanks,
                                      >
                                      > Roy Osherove
                                      > http://www.iserializable.com
                                      >
                                      >
                                      > [Non-text portions of this message have been removed]
                                      >
                                      >
                                      >
                                      > To Post a message, send it to: extremeprogramming@...
                                      >
                                      > To Unsubscribe, send a blank message to:
                                      > extremeprogramming-unsubscribe@...
                                      >
                                      > ad-free courtesy of objectmentor.com
                                      > Yahoo! Groups Links
                                      >
                                      >
                                      >
                                      >
                                      >
                                      >
                                      >
                                    • Roy Osherove
                                      Steve - I m assuming you just described a system that is not unit-testable to a certain degree. If it was, we could reasonably easily write tests for any of
                                      Message 18 of 24 , Feb 13, 2006
                                      • 0 Attachment
                                        Steve - I'm assuming you just described a system that is not unit-testable
                                        to a certain degree.
                                        If it was, we could reasonably easily write tests for any of the reasons
                                        that could fail the system, easily set up the state under test so that the
                                        failure is triggered when appropriate, and make sure that it is indeed
                                        triggered and fails with that same message. But the setup state is different
                                        for each of these tests.
                                        If one could easily write such tests, I'd consider it a unit testable
                                        system.
                                        If one has a hard time writing tests for that system (since they don't know
                                        how to "fail" it at the right time, they have no control over some
                                        dependencies inside the method call) I'd say the system is not designed well
                                        enough to be unit tested easily.

                                        Did I understand you correctly?


                                        On 2/14/06, Steven Gordon <sgordonphd@...> wrote:
                                        >
                                        > Roy,
                                        >
                                        > Perhaps, I was just not being clear, so I will try again:
                                        >
                                        > My understanding is that you are looking for unit-test behaviors that
                                        > indicate that the system under test is not unit-testable (assuming that
                                        > the
                                        > unit-tests were written reasonably well).
                                        >
                                        > Suppose our system had an object were a particular call could fail in the
                                        > exact same way for many different reasons with no indication of the
                                        > precise
                                        > reason. There is no principled way to write the test with a specific
                                        > failure message. Just saying "the call X.y() failed" is not good enough.
                                        > Any specific message might not be true some of the time.
                                        >
                                        > This seems to me to be a critical property of a unit-testable system.
                                        >
                                        > Steve
                                        >
                                        >
                                        > > > > -----Original Message-----
                                        > > > > From: extremeprogramming@yahoogroups.com
                                        > > > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy
                                        > Osherove
                                        > > > > Sent: Sunday, February 12, 2006 10:51 PM
                                        > > > > To: extremeprogramming@yahoogroups.com
                                        > > > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                        > > > >
                                        > > > > Steve, Obviously, a poor test that sometimes fails and
                                        > > > > sometimes does not could be attributed to a bad test, but an
                                        > > > > inconsistent test result also usually means (at least in my
                                        > > > > experience) that the state under test is not cleaned up or
                                        > > > > initialized between the test runs (such as object state not
                                        > > > > reset, database state not reset). That usually is a sign that
                                        > > > > the test has to manage external state, and does a poor job of that.
                                        > > > > You could then perhaps say that the test is poor quality, but
                                        > > > > that a high quality test for such a system, such that it
                                        > > > > always returns a fail pass result is hard to write, since one
                                        > > > > has to manage external state, which goes against the
                                        > > > > definitions I've laid thus far. You could also say it may not
                                        > > > > even be a unit test, since it uses external state (touching
                                        > > > > disk, ordering may matter, database use perhaps, threading) A
                                        > > > > reverse look may help here.
                                        > > > > In a well designed system, a *unit test* should be easy to
                                        > > > > write such that it always returns the same result. The system
                                        > > > > is easy to initialize in code such that any external state is
                                        > > > > decoupled enough from the object under test so that the
                                        > > > > external results of running the tests are easily reversible,
                                        > > > > or non existant at all (non existent may be the true sign of
                                        > > > > a*unit test*, since a database related test is an
                                        > > > > integration test in my mind).
                                        > > > >
                                        > > > > so, to conclude,I think that a non consistent test result is
                                        > > > > a sign that you have to manage some external state, or that
                                        > > > > your test is not really a unit test. Of course, you can
                                        > > > > *always* work badly enough so that a test fails just because
                                        > > > > it did not initialize or tear down an object correctly, but,
                                        > > > > in my experience, that is is a small fraction of real unit
                                        > > > > tests behavior, and is *easily* fixable. On the other hand,
                                        > > > > fixing the same situation when you ave external state not not
                                        > > > > usually so easy as just dropping a new setup method..
                                        > > > >
                                        > > > > Roy.
                                        > > > >
                                        > > > >
                                        > > > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                        > > > > >
                                        > > > > > Roy,
                                        > > > > >
                                        > > > > > Are you saying that having the same test sometimes fail and
                                        > > > > sometimes
                                        > > > > > pass (without any code changes) is a property of an
                                        > > > > untestable system?
                                        > > > > > Could it not sometimes be due to a poorly written test?
                                        > > > > >
                                        > > > > > Why could it then not also be a property of some untestable
                                        > systems
                                        > > > > > that a failure of some reasonable unit test cannot be attributed
                                        > to
                                        > > > > > any single cause?
                                        > > > > >
                                        > > > > > It seems to me either property could be caused by either a poorly
                                        > > > > > written unit test or a system that is too monolithic to be
                                        > > > > unit-testable.
                                        > > > > >
                                        > > > > > Steve
                                        > > > > >
                                        > > > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                        > > > > > >
                                        > > > > > > Steven - yes. that's what I meant. If it fails, it will
                                        > > > > keep failing
                                        > > > > > > indefinitely until you change the code(e.g break
                                        > > > > something else :-)
                                        > > > > > > ) As for it saying what failed - that's a unit testing best
                                        > > > > > > practice, and people doing it or not does not depend on
                                        > > > > the health
                                        > > > > > > of the code base's design. But easily writing tests that
                                        > > > > fill these
                                        > > > > > > requirement is a testament that the system is indeed "testable"
                                        > -
                                        > > > > > > that is, it lends itself to be tested easily using unit
                                        > > > > tests that
                                        > > > > > > fill follow rules. Having a good fail
                                        > > > > > message
                                        > > > > > > in any of these tests has nothing to do with the system, but
                                        > with
                                        > > > > > > the
                                        > > > > > test
                                        > > > > > > standards, which are a different story..
                                        > > > > > > My definition here is that of the system under test, and
                                        > > > > one would
                                        > > > > > > know
                                        > > > > > to
                                        > > > > > > define a system as easily testable with unit tests or not.
                                        > > > > > >
                                        > > > > > > If we can conclude that these tests being easily writable with a
                                        > > > > > > system with specific properties, we can perhaps deduce from that
                                        > > > > > > many other things which may be related, I'm just not sure
                                        > > > > what they
                                        > > > > > > are yet...
                                        > > > > > >
                                        > > > > > > Roy.
                                        > > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > > [Non-text portions of this message have been removed]
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > > To Post a message, send it to: extremeprogramming@...
                                        > > > > >
                                        > > > > > To Unsubscribe, send a blank message to:
                                        > > > > > extremeprogramming-unsubscribe@...
                                        > > > > >
                                        > > > > > ad-free courtesy of objectmentor.com
                                        > > > > > Yahoo! Groups Links
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > > >
                                        > > > >
                                        > > > >
                                        > > > > --
                                        > > > > Thanks,
                                        > > > >
                                        > > > > Roy Osherove
                                        > > > > http://www.iserializable.com
                                        > > > >
                                        > > > >
                                        > > > > [Non-text portions of this message have been removed]
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > > To Post a message, send it to: extremeprogramming@...
                                        > > > >
                                        > > > > To Unsubscribe, send a blank message to:
                                        > > > > extremeprogramming-unsubscribe@...
                                        > > > >
                                        > > > > ad-free courtesy of objectmentor.com
                                        > > > > Yahoo! Groups Links
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > > >
                                        > > >
                                        > > >
                                        > > >
                                        > > >
                                        > > > To Post a message, send it to: extremeprogramming@...
                                        > > >
                                        > > > To Unsubscribe, send a blank message to:
                                        > > > extremeprogramming-unsubscribe@...
                                        > > >
                                        > > > ad-free courtesy of objectmentor.com
                                        > > > Yahoo! Groups Links
                                        > > >
                                        > > >
                                        > > >
                                        > > >
                                        > > >
                                        > > >
                                        > > >
                                        > >
                                        > >
                                        > > --
                                        > > Thanks,
                                        > >
                                        > > Roy Osherove
                                        > > http://www.iserializable.com
                                        > >
                                        > >
                                        > > [Non-text portions of this message have been removed]
                                        > >
                                        > >
                                        > >
                                        > > To Post a message, send it to: extremeprogramming@...
                                        > >
                                        > > To Unsubscribe, send a blank message to:
                                        > > extremeprogramming-unsubscribe@...
                                        > >
                                        > > ad-free courtesy of objectmentor.com
                                        > > Yahoo! Groups Links
                                        > >
                                        > >
                                        > >
                                        > >
                                        > >
                                        > >
                                        > >
                                        >
                                        >
                                        > [Non-text portions of this message have been removed]
                                        >
                                        >
                                        >
                                        > To Post a message, send it to: extremeprogramming@...
                                        >
                                        > To Unsubscribe, send a blank message to:
                                        > extremeprogramming-unsubscribe@...
                                        >
                                        > ad-free courtesy of objectmentor.com
                                        > Yahoo! Groups Links
                                        >
                                        >
                                        >
                                        >
                                        >
                                        >
                                        >


                                        --
                                        Thanks,

                                        Roy Osherove
                                        http://www.iserializable.com


                                        [Non-text portions of this message have been removed]
                                      • Roy Osherove
                                        oops! :-) ... -- Thanks, Roy Osherove http://www.iserializable.com [Non-text portions of this message have been removed]
                                        Message 19 of 24 , Feb 13, 2006
                                        • 0 Attachment
                                          oops! :-)


                                          On 2/14/06, Charlie Poole <xp@...> wrote:
                                          >
                                          > Hi Roy,
                                          >
                                          > I was agreeing with you. I'll have to work on that. :-)
                                          >
                                          > Charlie
                                          >
                                          > > -----Original Message-----
                                          > > From: extremeprogramming@yahoogroups.com
                                          > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy Osherove
                                          > > Sent: Monday, February 13, 2006 2:27 PM
                                          > > To: extremeprogramming@yahoogroups.com
                                          > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                          > >
                                          > > I'm not saying test cod quality is not important, Charlie.
                                          > > I'm saying that regardless of the test quality, if one knew
                                          > > how to write good tests, one would find that writing them
                                          > > while following the PCCOF rules much much easier than in a
                                          > > system full of, legacy code, perhaps.
                                          > > (TDD is out of the topic here since it clearly forces you
                                          > > into the PCCOF rules, IMO)
                                          > >
                                          > >
                                          > > On 2/13/06, Charlie Poole <xp@...> wrote:
                                          > > >
                                          > > > Hi Roy,
                                          > > >
                                          > > > So we should have both high quality system design and high
                                          > > quality tests!
                                          > > > :-)
                                          > > >
                                          > > > In my experience, poor tests causing inconsistent results
                                          > > are usually
                                          > > > suffering from local coding problems - like retained test
                                          > > object state
                                          > > > across cases. These are relatively easy to fix.
                                          > > >
                                          > > > The problems caused by design are broader - and much more
                                          > > interesting.
                                          > > >
                                          > > > Charlie
                                          > > >
                                          > > > > -----Original Message-----
                                          > > > > From: extremeprogramming@yahoogroups.com
                                          > > > > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of Roy
                                          > > > > Osherove
                                          > > > > Sent: Sunday, February 12, 2006 10:51 PM
                                          > > > > To: extremeprogramming@yahoogroups.com
                                          > > > > Subject: Re: [XP] Defining a Unit-Testable System/Codebase
                                          > > > >
                                          > > > > Steve, Obviously, a poor test that sometimes fails and sometimes
                                          > > > > does not could be attributed to a bad test, but an
                                          > > inconsistent test
                                          > > > > result also usually means (at least in my
                                          > > > > experience) that the state under test is not cleaned up or
                                          > > > > initialized between the test runs (such as object state
                                          > > not reset,
                                          > > > > database state not reset). That usually is a sign that
                                          > > the test has
                                          > > > > to manage external state, and does a poor job of that.
                                          > > > > You could then perhaps say that the test is poor quality,
                                          > > but that a
                                          > > > > high quality test for such a system, such that it always
                                          > > returns a
                                          > > > > fail pass result is hard to write, since one has to
                                          > > manage external
                                          > > > > state, which goes against the definitions I've laid thus far. You
                                          > > > > could also say it may not even be a unit test, since it uses
                                          > > > > external state (touching disk, ordering may matter, database use
                                          > > > > perhaps, threading) A reverse look may help here.
                                          > > > > In a well designed system, a *unit test* should be easy to write
                                          > > > > such that it always returns the same result. The system
                                          > > is easy to
                                          > > > > initialize in code such that any external state is
                                          > > decoupled enough
                                          > > > > from the object under test so that the external results
                                          > > of running
                                          > > > > the tests are easily reversible, or non existant at all (non
                                          > > > > existent may be the true sign of a*unit test*, since a database
                                          > > > > related test is an integration test in my mind).
                                          > > > >
                                          > > > > so, to conclude,I think that a non consistent test result
                                          > > is a sign
                                          > > > > that you have to manage some external state, or that your test is
                                          > > > > not really a unit test. Of course, you can
                                          > > > > *always* work badly enough so that a test fails just
                                          > > because it did
                                          > > > > not initialize or tear down an object correctly, but, in my
                                          > > > > experience, that is is a small fraction of real unit
                                          > > tests behavior,
                                          > > > > and is *easily* fixable. On the other hand, fixing the same
                                          > > > > situation when you ave external state not not usually so easy as
                                          > > > > just dropping a new setup method..
                                          > > > >
                                          > > > > Roy.
                                          > > > >
                                          > > > >
                                          > > > > On 2/13/06, Steven Gordon <sgordonphd@...> wrote:
                                          > > > > >
                                          > > > > > Roy,
                                          > > > > >
                                          > > > > > Are you saying that having the same test sometimes fail and
                                          > > > > sometimes
                                          > > > > > pass (without any code changes) is a property of an
                                          > > > > untestable system?
                                          > > > > > Could it not sometimes be due to a poorly written test?
                                          > > > > >
                                          > > > > > Why could it then not also be a property of some untestable
                                          > > > > > systems that a failure of some reasonable unit test cannot be
                                          > > > > > attributed to any single cause?
                                          > > > > >
                                          > > > > > It seems to me either property could be caused by
                                          > > either a poorly
                                          > > > > > written unit test or a system that is too monolithic to be
                                          > > > > unit-testable.
                                          > > > > >
                                          > > > > > Steve
                                          > > > > >
                                          > > > > > On 2/12/06, Roy Osherove <RoyOsherove@...> wrote:
                                          > > > > > >
                                          > > > > > > Steven - yes. that's what I meant. If it fails, it will
                                          > > > > keep failing
                                          > > > > > > indefinitely until you change the code(e.g break
                                          > > > > something else :-)
                                          > > > > > > ) As for it saying what failed - that's a unit testing best
                                          > > > > > > practice, and people doing it or not does not depend on
                                          > > > > the health
                                          > > > > > > of the code base's design. But easily writing tests that
                                          > > > > fill these
                                          > > > > > > requirement is a testament that the system is indeed
                                          > > "testable"
                                          > > > > > > - that is, it lends itself to be tested easily using unit
                                          > > > > tests that
                                          > > > > > > fill follow rules. Having a good fail
                                          > > > > > message
                                          > > > > > > in any of these tests has nothing to do with the system, but
                                          > > > > > > with the
                                          > > > > > test
                                          > > > > > > standards, which are a different story..
                                          > > > > > > My definition here is that of the system under test, and
                                          > > > > one would
                                          > > > > > > know
                                          > > > > > to
                                          > > > > > > define a system as easily testable with unit tests or not.
                                          > > > > > >
                                          > > > > > > If we can conclude that these tests being easily
                                          > > writable with a
                                          > > > > > > system with specific properties, we can perhaps
                                          > > deduce from that
                                          > > > > > > many other things which may be related, I'm just not sure
                                          > > > > what they
                                          > > > > > > are yet...
                                          > > > > > >
                                          > > > > > > Roy.
                                          > > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > > [Non-text portions of this message have been removed]
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > > To Post a message, send it to: extremeprogramming@...
                                          > > > > >
                                          > > > > > To Unsubscribe, send a blank message to:
                                          > > > > > extremeprogramming-unsubscribe@...
                                          > > > > >
                                          > > > > > ad-free courtesy of objectmentor.com Yahoo! Groups Links
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > > >
                                          > > > >
                                          > > > >
                                          > > > > --
                                          > > > > Thanks,
                                          > > > >
                                          > > > > Roy Osherove
                                          > > > > http://www.iserializable.com
                                          > > > >
                                          > > > >
                                          > > > > [Non-text portions of this message have been removed]
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > > To Post a message, send it to: extremeprogramming@...
                                          > > > >
                                          > > > > To Unsubscribe, send a blank message to:
                                          > > > > extremeprogramming-unsubscribe@...
                                          > > > >
                                          > > > > ad-free courtesy of objectmentor.com
                                          > > > > Yahoo! Groups Links
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > > To Post a message, send it to: extremeprogramming@...
                                          > > >
                                          > > > To Unsubscribe, send a blank message to:
                                          > > > extremeprogramming-unsubscribe@...
                                          > > >
                                          > > > ad-free courtesy of objectmentor.com
                                          > > > Yahoo! Groups Links
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > > >
                                          > >
                                          > >
                                          > > --
                                          > > Thanks,
                                          > >
                                          > > Roy Osherove
                                          > > http://www.iserializable.com
                                          > >
                                          > >
                                          > > [Non-text portions of this message have been removed]
                                          > >
                                          > >
                                          > >
                                          > > To Post a message, send it to: extremeprogramming@...
                                          > >
                                          > > To Unsubscribe, send a blank message to:
                                          > > extremeprogramming-unsubscribe@...
                                          > >
                                          > > ad-free courtesy of objectmentor.com
                                          > > Yahoo! Groups Links
                                          > >
                                          > >
                                          > >
                                          > >
                                          > >
                                          > >
                                          > >
                                          >
                                          >
                                          >
                                          >
                                          > To Post a message, send it to: extremeprogramming@...
                                          >
                                          > To Unsubscribe, send a blank message to:
                                          > extremeprogramming-unsubscribe@...
                                          >
                                          > ad-free courtesy of objectmentor.com
                                          > Yahoo! Groups Links
                                          >
                                          >
                                          >
                                          >
                                          >
                                          >
                                          >


                                          --
                                          Thanks,

                                          Roy Osherove
                                          http://www.iserializable.com


                                          [Non-text portions of this message have been removed]
                                        • Steven Gordon
                                          Not quite. My concern is not tests that are expected to fail. They pass when the expected failure occurs. My concern is when a test fails which was expected
                                          Message 20 of 24 , Feb 13, 2006
                                          • 0 Attachment
                                            Not quite.

                                            My concern is not tests that are expected to fail. They pass when the
                                            expected failure occurs.

                                            My concern is when a test fails which was expected to pass if the unit of
                                            code was working correctly. Consider a system is so highly coupled that
                                            when X.y() returns the wrong value, the source of the error might not be in
                                            the design or implementation of the unit of code being tested, but could
                                            well be because of an error in the design or implementation of some other
                                            class in the system. If this is the case, then the test is behaving like an
                                            integration test rather than a unit test.

                                            If the unit test is being written correctly, but cannot localize the cause
                                            of each possible failed assertion in the unit of code being tested, then
                                            this is a sign of a nasty system. This will not be noticed when test
                                            passes, only when the test fails and the bug cannot be localized.

                                            So, this would be a unit test behaviour that indicates a system which cannot
                                            be effectively unit tested, or at least would be very difficult to unit test
                                            effectively.

                                            On 2/13/06, Roy Osherove <RoyOsherove@...> wrote:
                                            >
                                            > Steve - I'm assuming you just described a system that is not unit-testable
                                            > to a certain degree.
                                            > If it was, we could reasonably easily write tests for any of the reasons
                                            > that could fail the system, easily set up the state under test so that the
                                            > failure is triggered when appropriate, and make sure that it is indeed
                                            > triggered and fails with that same message. But the setup state is
                                            > different
                                            > for each of these tests.
                                            > If one could easily write such tests, I'd consider it a unit testable
                                            > system.
                                            > If one has a hard time writing tests for that system (since they don't
                                            > know
                                            > how to "fail" it at the right time, they have no control over some
                                            > dependencies inside the method call) I'd say the system is not designed
                                            > well
                                            > enough to be unit tested easily.
                                            >
                                            > Did I understand you correctly?
                                            >
                                            >


                                            [Non-text portions of this message have been removed]
                                          • Roy Osherove
                                            Interesting. So you re essentially talking about an Integration test in disguise - that is *thought* to be a unit test, right? But then, wouldn t you notice
                                            Message 21 of 24 , Feb 13, 2006
                                            • 0 Attachment
                                              Interesting.
                                              So you're essentially talking about an Integration test in disguise - that
                                              is *thought* to be a unit test, right?
                                              But then, wouldn't you notice that you're writing a unit test that uses more
                                              than one class in its test path by looking at the class code that you're
                                              testing?

                                              nevertheless, you're saying there is one more "rule" that might be missing,
                                              I'm I'm reading you correctly - something like this:

                                              It should be reasonably easy to write a test that clearly tests only a
                                              single unit, without touching any external dependency. If you don't know
                                              it's touching another unit (and find out later) that's a sign of a less than
                                              optimal unit testable system, right?

                                              It's a hard one to find while you're writing the test, I'd think. However,
                                              let's take two views on writing this test and see if we can't flush this
                                              rule out:
                                              1) TDD: You have to make the test *fail* first. You write the test, it
                                              *passes*. You notice something is wrong. It's hard to actually make it fail
                                              first - and voila - you find out that you have a broken rule (should be easy
                                              to test a single unit with expected behavior)
                                              2) test is written after the code is working. Test passes on first go.
                                              Everyone is happy.
                                              Another test for the *opposite* behavior (if it fails correctly, it can also
                                              work correctly, right?) is then written and that one passes too. everyone is
                                              happy. no one notices the "bug", but the odds are high that this is an
                                              integration test rather than a unit test (both of them) since this is
                                              "legacy" code, and not greenfield. Plus, the test writer may find that the
                                              code in the class under test is actually calling out to several
                                              dependencies.

                                              In case one - we can find this easily with a fail-first approach.
                                              In case 2, it's not even a unit test and thus does not fall into the list
                                              of rules I've specified.

                                              Now - what if the test indeed fails, but for the wrong reasons? It should
                                              then pass for the wrong reasons as well, right? If you're doing TDD, you
                                              should be in control of all the failure reasons, so that should not be a
                                              problem. If you're testing legacy code, you're more than likely testing
                                              using an integration test, thus, my theory is out of scope.

                                              I was just thinking out loud here. Hope it makes sense..

                                              Roy.

                                              On 2/14/06, Steven Gordon <sgordonphd@...> wrote:
                                              >
                                              > Not quite.
                                              >
                                              > My concern is not tests that are expected to fail. They pass when the
                                              > expected failure occurs.
                                              >
                                              > My concern is when a test fails which was expected to pass if the unit of
                                              > code was working correctly. Consider a system is so highly coupled that
                                              > when X.y() returns the wrong value, the source of the error might not be
                                              > in
                                              > the design or implementation of the unit of code being tested, but could
                                              > well be because of an error in the design or implementation of some other
                                              > class in the system. If this is the case, then the test is behaving like
                                              > an
                                              > integration test rather than a unit test.
                                              >
                                              > If the unit test is being written correctly, but cannot localize the cause
                                              > of each possible failed assertion in the unit of code being tested, then
                                              > this is a sign of a nasty system. This will not be noticed when test
                                              > passes, only when the test fails and the bug cannot be localized.
                                              >
                                              > So, this would be a unit test behaviour that indicates a system which
                                              > cannot
                                              > be effectively unit tested, or at least would be very difficult to unit
                                              > test
                                              > effectively.
                                              >
                                              > On 2/13/06, Roy Osherove <RoyOsherove@...> wrote:
                                              > >
                                              > > Steve - I'm assuming you just described a system that is not
                                              > unit-testable
                                              > > to a certain degree.
                                              > > If it was, we could reasonably easily write tests for any of the reasons
                                              > > that could fail the system, easily set up the state under test so that
                                              > the
                                              > > failure is triggered when appropriate, and make sure that it is indeed
                                              > > triggered and fails with that same message. But the setup state is
                                              > > different
                                              > > for each of these tests.
                                              > > If one could easily write such tests, I'd consider it a unit testable
                                              > > system.
                                              > > If one has a hard time writing tests for that system (since they don't
                                              > > know
                                              > > how to "fail" it at the right time, they have no control over some
                                              > > dependencies inside the method call) I'd say the system is not designed
                                              > > well
                                              > > enough to be unit tested easily.
                                              > >
                                              > > Did I understand you correctly?
                                              > >
                                              > >
                                              >
                                              >
                                              > [Non-text portions of this message have been removed]
                                              >
                                              >
                                              >
                                              > To Post a message, send it to: extremeprogramming@...
                                              >
                                              > To Unsubscribe, send a blank message to:
                                              > extremeprogramming-unsubscribe@...
                                              >
                                              > ad-free courtesy of objectmentor.com
                                              > Yahoo! Groups Links
                                              >
                                              >
                                              >
                                              >
                                              >
                                              >
                                              >
                                              >


                                              --
                                              Thanks,

                                              Roy Osherove
                                              http://www.iserializable.com


                                              [Non-text portions of this message have been removed]
                                            • Steven Gordon
                                              If the class I am testing is turns out to be a facade for a bunch of highly coupled hidden classes, what looks like a unit test would in fact really be an
                                              Message 22 of 24 , Feb 13, 2006
                                              • 0 Attachment
                                                If the class I am testing is turns out to be a facade for a bunch of highly
                                                coupled hidden classes, what looks like a unit test would in fact really be
                                                an integration test (many people do use Xunit to write integration and
                                                functional tests as well as unit tests). You really could only tell by
                                                looking at the implementation of the class being directly tested (or,
                                                equivalently, analyzing the cause of failed asserts).

                                                Given proper TDD, I believe none of the problems you are noting should
                                                occur. I was assuming not TDD, possibly even legacy code.


                                                On 2/13/06, Roy Osherove <RoyOsherove@...> wrote:
                                                >
                                                > Interesting.
                                                > So you're essentially talking about an Integration test in disguise - that
                                                > is *thought* to be a unit test, right?
                                                > But then, wouldn't you notice that you're writing a unit test that uses
                                                > more
                                                > than one class in its test path by looking at the class code that you're
                                                > testing?
                                                >
                                                > nevertheless, you're saying there is one more "rule" that might be
                                                > missing,
                                                > I'm I'm reading you correctly - something like this:
                                                >
                                                > It should be reasonably easy to write a test that clearly tests only a
                                                > single unit, without touching any external dependency. If you don't know
                                                > it's touching another unit (and find out later) that's a sign of a less
                                                > than
                                                > optimal unit testable system, right?
                                                >
                                                > It's a hard one to find while you're writing the test, I'd think. However,
                                                > let's take two views on writing this test and see if we can't flush this
                                                > rule out:
                                                > 1) TDD: You have to make the test *fail* first. You write the test, it
                                                > *passes*. You notice something is wrong. It's hard to actually make it
                                                > fail
                                                > first - and voila - you find out that you have a broken rule (should be
                                                > easy
                                                > to test a single unit with expected behavior)
                                                > 2) test is written after the code is working. Test passes on first go.
                                                > Everyone is happy.
                                                > Another test for the *opposite* behavior (if it fails correctly, it can
                                                > also
                                                > work correctly, right?) is then written and that one passes too. everyone
                                                > is
                                                > happy. no one notices the "bug", but the odds are high that this is an
                                                > integration test rather than a unit test (both of them) since this is
                                                > "legacy" code, and not greenfield. Plus, the test writer may find that the
                                                > code in the class under test is actually calling out to several
                                                > dependencies.
                                                >
                                                > In case one - we can find this easily with a fail-first approach.
                                                > In case 2, it's not even a unit test and thus does not fall into the list
                                                > of rules I've specified.
                                                >
                                                > Now - what if the test indeed fails, but for the wrong reasons? It should
                                                > then pass for the wrong reasons as well, right? If you're doing TDD, you
                                                > should be in control of all the failure reasons, so that should not be a
                                                > problem. If you're testing legacy code, you're more than likely testing
                                                > using an integration test, thus, my theory is out of scope.
                                                >
                                                > I was just thinking out loud here. Hope it makes sense..
                                                >
                                                > Roy.
                                                >


                                                [Non-text portions of this message have been removed]
                                              • Roy Osherove
                                                turns out is what i have a problem with. If you re writing a unit test per-se, you should know in advance that you re testing one class. If you re
                                                Message 23 of 24 , Feb 14, 2006
                                                • 0 Attachment
                                                  "turns out" is what i have a problem with. If you're writing a unit test
                                                  per-se, you should know in advance that you're testing one class. If you're
                                                  intentionally writing an integration test, knowing fully well that this is
                                                  highly coupled legacy code, that's a different story (an important one, but
                                                  different) and of course, means that the system's design is not that
                                                  testable.
                                                  A unit-testable system might still have a facade layer, but that facade
                                                  layer can be testable, with all its lower level helpers safely decoupled
                                                  using interfaces or virtual method calls or factories, where a unit test can
                                                  easily replace them with fakes and mocks, and testing the facade logic
                                                  separately.

                                                  If the facade has not logic (simply forwards calls with no IFs or switches)
                                                  I'm not sure I'd like to test it. That's why my definition includes "for
                                                  every logical part" in the system.


                                                  On 2/14/06, Steven Gordon <sgordonphd@...> wrote:
                                                  >
                                                  > If the class I am testing is turns out to be a facade for a bunch of
                                                  > highly
                                                  > coupled hidden classes, what looks like a unit test would in fact really
                                                  > be
                                                  > an integration test (many people do use Xunit to write integration and
                                                  > functional tests as well as unit tests). You really could only tell by
                                                  > looking at the implementation of the class being directly tested (or,
                                                  > equivalently, analyzing the cause of failed asserts).
                                                  >
                                                  > Given proper TDD, I believe none of the problems you are noting should
                                                  > occur. I was assuming not TDD, possibly even legacy code.
                                                  >
                                                  >
                                                  > On 2/13/06, Roy Osherove <RoyOsherove@...> wrote:
                                                  > >
                                                  > > Interesting.
                                                  > > So you're essentially talking about an Integration test in disguise -
                                                  > that
                                                  > > is *thought* to be a unit test, right?
                                                  > > But then, wouldn't you notice that you're writing a unit test that uses
                                                  > > more
                                                  > > than one class in its test path by looking at the class code that you're
                                                  > > testing?
                                                  > >
                                                  > > nevertheless, you're saying there is one more "rule" that might be
                                                  > > missing,
                                                  > > I'm I'm reading you correctly - something like this:
                                                  > >
                                                  > > It should be reasonably easy to write a test that clearly tests only a
                                                  > > single unit, without touching any external dependency. If you don't know
                                                  > > it's touching another unit (and find out later) that's a sign of a less
                                                  > > than
                                                  > > optimal unit testable system, right?
                                                  > >
                                                  > > It's a hard one to find while you're writing the test, I'd think.
                                                  > However,
                                                  > > let's take two views on writing this test and see if we can't flush this
                                                  > > rule out:
                                                  > > 1) TDD: You have to make the test *fail* first. You write the test, it
                                                  > > *passes*. You notice something is wrong. It's hard to actually make it
                                                  > > fail
                                                  > > first - and voila - you find out that you have a broken rule (should be
                                                  > > easy
                                                  > > to test a single unit with expected behavior)
                                                  > > 2) test is written after the code is working. Test passes on first go.
                                                  > > Everyone is happy.
                                                  > > Another test for the *opposite* behavior (if it fails correctly, it can
                                                  > > also
                                                  > > work correctly, right?) is then written and that one passes too.
                                                  > everyone
                                                  > > is
                                                  > > happy. no one notices the "bug", but the odds are high that this is an
                                                  > > integration test rather than a unit test (both of them) since this is
                                                  > > "legacy" code, and not greenfield. Plus, the test writer may find that
                                                  > the
                                                  > > code in the class under test is actually calling out to several
                                                  > > dependencies.
                                                  > >
                                                  > > In case one - we can find this easily with a fail-first approach.
                                                  > > In case 2, it's not even a unit test and thus does not fall into the
                                                  > list
                                                  > > of rules I've specified.
                                                  > >
                                                  > > Now - what if the test indeed fails, but for the wrong reasons? It
                                                  > should
                                                  > > then pass for the wrong reasons as well, right? If you're doing TDD, you
                                                  > > should be in control of all the failure reasons, so that should not be a
                                                  > > problem. If you're testing legacy code, you're more than likely testing
                                                  > > using an integration test, thus, my theory is out of scope.
                                                  > >
                                                  > > I was just thinking out loud here. Hope it makes sense..
                                                  > >
                                                  > > Roy.
                                                  > >
                                                  >
                                                  >
                                                  > [Non-text portions of this message have been removed]
                                                  >
                                                  >
                                                  >
                                                  > To Post a message, send it to: extremeprogramming@...
                                                  >
                                                  > To Unsubscribe, send a blank message to:
                                                  > extremeprogramming-unsubscribe@...
                                                  >
                                                  > ad-free courtesy of objectmentor.com
                                                  > Yahoo! Groups Links
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >
                                                  >


                                                  --
                                                  Thanks,

                                                  Roy Osherove
                                                  http://www.iserializable.com


                                                  [Non-text portions of this message have been removed]
                                                • Ron Jeffries
                                                  ... A random observation ... I don t find much value to distinguishing unit test from integration test . When I m doing TDD, I write tests that start out
                                                  Message 24 of 24 , Feb 14, 2006
                                                  • 0 Attachment
                                                    On Tuesday, February 14, 2006, at 4:03:50 AM, Roy Osherove wrote:

                                                    > "turns out" is what i have a problem with. If you're writing a unit test
                                                    > per-se, you should know in advance that you're testing one class. If you're
                                                    > intentionally writing an integration test, knowing fully well that this is
                                                    > highly coupled legacy code, that's a different story (an important one, but
                                                    > different) and of course, means that the system's design is not that
                                                    > testable.
                                                    > A unit-testable system might still have a facade layer, but that facade
                                                    > layer can be testable, with all its lower level helpers safely decoupled
                                                    > using interfaces or virtual method calls or factories, where a unit test can
                                                    > easily replace them with fakes and mocks, and testing the facade logic
                                                    > separately.

                                                    A random observation ...

                                                    I don't find much value to distinguishing "unit test" from
                                                    "integration test". When I'm doing TDD, I write tests that start out
                                                    against one test, but might wind up testing more than one -- though
                                                    usually they don't. When I'm testing code that I've written, the
                                                    tests are more likely to test more than one class at a time, but I
                                                    write them the same way and require them to work the same way.

                                                    I don't do much isolation when I'm building the code up with TDD --
                                                    it is rare for me to write a fake or mock, though sometimes I do.
                                                    More commonly, I write objects that are loosely enough coupled that
                                                    it is easy to instantiate one and give it the values I want from a
                                                    string or simple object; my objects don't generally have intimate
                                                    relations with databases or other such objects of the night.

                                                    I do often wish, with legacy code, that I could instantiate
                                                    individual classes or clusters. The obstacle is usually that these
                                                    objects do consort too intimately with too many other objects. In
                                                    that case, I'd like to resort to fakes or mocks, but it is of course
                                                    a lot of trouble to do so. I'm not sure yet whether, instead of
                                                    surrounding such objects with a fake database, it might be better to
                                                    extract in the other direction, producing an object that, in place,
                                                    is just one step away from the real database, but in use is fed with
                                                    simpler objects.


                                                    I feel that that last needs a bit more explanation. I'm not sure I'm
                                                    up to it, but I'll try.

                                                    If we are given an objectUnderTest whose connection to the database
                                                    is getting in the way of testing, we might extract its call for a
                                                    database connection, then instantiate it with a fake connection.
                                                    Then for testing, we'd make our fake connection feed whatever test
                                                    data we wanted. In severe cases, we might have to test what kind of
                                                    database calls or date was sent back to our fake.

                                                    Another way to go might be to create an "inner object" that did most
                                                    of the logic of our objectUnderTest, given simpler or more
                                                    accessible parameters, such as strings or nice little record
                                                    objects. We could quite likely create such a thing by extraction, or
                                                    perhaps by TDD. Either way, this object could be testable in
                                                    conventional "unit test" style, by creating it, sending it messages,
                                                    checking the answers. Then, finally, we'd gut our objectUnderTest,
                                                    making it use our new "inner object".

                                                    The second approach could save us creating a fake or mock database
                                                    kind of thing. I'm not sure when the second approach is better and
                                                    when it's not.

                                                    Ron Jeffries
                                                    www.XProgramming.com
                                                    Get over it. -- The Eagles
                                                  Your message has been successfully submitted and would be delivered to recipients shortly.