Loading ...
Sorry, an error occurred while loading the content.

Adventures in C#: How Do We Do a Test That We Can't Do?

Expand Messages
  • Ron Jeffries
    Adventures in C#: How Do We Do a Test That We Can t Do? Ron Jeffries 10/31/2002 We encountered a real bug in the system that wasn t caught by our tests. The
    Message 1 of 12 , Nov 1, 2002
    • 0 Attachment
      Adventures in C#: How Do We Do a Test That We Can't Do?
      Ron Jeffries
      10/31/2002

      We encountered a real bug in the system that wasn't caught by our
      tests. The feature worked but the GUI wasn't hooked to it. We don't
      know how to test that, but our rules of operation say that we must.
      What can we do?

      http://www.xprogramming.com/xpmag/acsGUItesting.htm

      Ron Jeffries
      www.XProgramming.com
      The practices are not XP. They are a path to XP.
    • Ron Jeffries
      Around Friday, November 1, 2002, 7:01:05 AM, Ron Jeffries wrote: Wait! Wait, wait, wait! ... It s cool writing a book where if you screw up it s just part of
      Message 2 of 12 , Nov 1, 2002
      • 0 Attachment
        Around Friday, November 1, 2002, 7:01:05 AM, Ron Jeffries wrote:

        Wait! Wait, wait, wait!

        > http://www.xprogramming.com/xpmag/acsGUItesting.htm#N332

        It's cool writing a book where if you screw up it's just part of the
        story ...

        Ron Jeffries
        www.XProgramming.com
        I could be wrong, but I'm not. --Eagles, Victim of Love
      • Daniel Berlinger
        ... Ron, Would it be wrong to write some notes that explain your thinking of the moment (you re on a role, you know just how to do it). Or is that almost as
        Message 3 of 12 , Nov 1, 2002
        • 0 Attachment
          > Wait! Wait, wait, wait!
          >
          > > http://www.xprogramming.com/xpmag/acsGUItesting.htm#N332
          >
          Ron,

          Would it be wrong to write some notes that explain your thinking "of the moment" (you're on a role, you know just how to do it). Or is that almost as bad as coding with the fantasy of the future?

          Thanks,

          d.

          PS I use the terms "bad" and "wrong" in the context of the lesson.
        • Ron Jeffries
          ... I guess you re asking if as a developer I might write something down to be sure that I remember how to do it? I might well do that. On the other hand, I
          Message 4 of 12 , Nov 1, 2002
          • 0 Attachment
            On Friday, November 1, 2002, at 11:10:01 AM, Daniel Berlinger wrote:

            >> Wait! Wait, wait, wait!
            >>
            >> > http://www.xprogramming.com/xpmag/acsGUItesting.htm#N332
            >>
            > Ron,

            > Would it be wrong to write some notes that explain your thinking "of the moment" (you're on a role, you know
            > just how to do it). Or is that almost as bad as coding with the fantasy of the future?

            I guess you're asking if as a developer I might write something down
            to be sure that I remember how to do it? I might well do that.

            On the other hand, I like to live on the edge, and I enjoy learning
            new things. So I don't mind forgetting stuff, because it's so exciting
            to discover it later. ;->

            Ron Jeffries
            www.XProgramming.com
            "How do I get to XP?" "Practice, man, practice."
          • Charlie Poole
            Ron, I want to think a bit about what you re doing here, but in the spirit of just telling it as it happens, here s what s in my head right now. What is Ron
            Message 5 of 12 , Nov 1, 2002
            • 0 Attachment
              Ron,

              I want to think a bit about what you're doing here, but in the spirit
              of just telling it as it happens, here's what's in my head right now.

              What is Ron testing? He's already tested that the model does what
              it should do when it's methods are called correctly, so what's left?

              Here's a first cut.
              * Test that altS causes the model's correct method to be called correctly.

              Hmmm... That certainly covers it all in one shot, but it seemed hard
              to test. What could possibly go wrong?

              Candidates:
              1. The Windows operating system might ignore the accelerator
              2. The Accelerator might not be on the menu item
              3. The menu Item might not be on the form and it's main menu
              4. The menu Item might not generate a click event when clicked
              5. The click event might not be connected to the proper handler
              6. The handler might not call the proper model method.

              I put items 1 and 4 in as things that probably don't need to be
              tested. Of course, in my own code, it sometimes seems as if one
              of these things has happened, but it usually turns out that
              I've done something wrong myself.

              So how hard is it to test for the other items?

              Items 2 and 3 can be looked at by instantiating the form and
              examining the menu and menu item. Something like the following
              untested code:

              MyForm form = new MyForm(...);
              Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
              Assert( form.someMenuItem.Mnemonic == 'S' );
              etc.

              Items 5 and 6 could be tested by a mock model or using the
              actual model as you've done. It strikes me that a mock model
              that did nothing but report on how it had been called and what
              it's arguments were would make isolated testing of the ui a
              lot easier. So...

              MyModel model = new MyModel(...);
              MyForm form = new MyForm( model );

              // Init form and model as needed

              form.MenuItem.PerformClick();
              Assert( model.SomeMethodWasCalled );
              AssertEquals( "somestring", model.SomeMethodParametersPassed );

              Of course, the same thing can be done with the real model by checking
              that the state has been appropriately modified.

              Like I said, just an initial set of thoughts...

              Charlie Poole
              cpoole@...
              www.pooleconsulting.com
              www.charliepoole.org



              > -----Original Message-----
              > From: Ron Jeffries [mailto:ronjeffries@...]
              > Sent: Friday, November 01, 2002 6:46 AM
              > To: extremeprogramming@yahoogroups.com
              > Subject: Re: [XP] Adventures in C#: How Do We Do a Test That We Can't
              > Do?
              >
              >
              > Around Friday, November 1, 2002, 7:01:05 AM, Ron Jeffries wrote:
              >
              > Wait! Wait, wait, wait!
              >
              > > http://www.xprogramming.com/xpmag/acsGUItesting.htm#N332
              >
              > It's cool writing a book where if you screw up it's just part of the
              > story ...
              >
              > Ron Jeffries
              > www.XProgramming.com
              > I could be wrong, but I'm not. --Eagles, Victim of Love
              >
              >
              > To Post a message, send it to: extremeprogramming@...
              >
              > To Unsubscribe, send a blank message to:
              > extremeprogramming-unsubscribe@...
              >
              > ad-free courtesy of objectmentor.com
              >
              > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
              >
              >
              >
              >
            • Kevin Lawrence
              From: Charlie Poole ... I was going to say something similar to Charlie but I got distracted with an essay about estimating. So
              Message 6 of 12 , Nov 1, 2002
              • 0 Attachment
                From: "Charlie Poole" <cpoole@...>
                >
                > Items 2 and 3 can be looked at by instantiating the form and
                > examining the menu and menu item. Something like the following
                > untested code:
                >
                > MyForm form = new MyForm(...);
                > Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                > Assert( form.someMenuItem.Mnemonic == 'S' );
                > etc.
                >


                I was going to say something similar to Charlie but I got distracted with an
                essay about estimating. So I'll delete my reply and I'll add to Charlie's.

                Here's what I do.

                I assert that the shortcut is correct
                AssertEquals(Shortcut.CtrlP, menu.Shortcut); - or whatever

                I assert that the label is right
                AssertEquals("Insert &Section", menu.Text);

                I assert that the menu does what it is supposed to when I click it
                menu.PerformClick();
                AssertEquals("<p></p>", textbox.Text);
                AssertEquals(3, textbox.SelectionStart);

                I assume that Bill Gates & Co have made the little ampersand thingie and the
                shortcut key work right so I don't test that.

                If I wanted to implement your script-driven acceptance tests, I might be
                tempted to write them in terms of menu labels rather than shortcut keys (and
                assume that they are equivalent).

                *input
                <p>whatever</p>
                *Edit|Insert Section
                *output
                etc etc etc

                That would make it supremely easy to match script commands with menu
                objects.


                > Items 5 and 6 could be tested by a mock model or using the
                > actual model as you've done. It strikes me that a mock model
                > that did nothing but report on how it had been called and what
                > it's arguments were would make isolated testing of the ui a
                > lot easier. So...
                >
                > MyModel model = new MyModel(...);
                > MyForm form = new MyForm( model );
                >
                > // Init form and model as needed
                >
                > form.MenuItem.PerformClick();
                > Assert( model.SomeMethodWasCalled );
                > AssertEquals( "somestring", model.SomeMethodParametersPassed );
                >
                > Of course, the same thing can be done with the real model by checking
                > that the state has been appropriately modified.
                >

                In my product, I mocked the model like this very early on and - now that
                things are a little more complex - I really wish I hadn't. I have had a few
                little bugs slip down the crack between testing the model and testing the UI
                and - now that my real model has a lot of internal magic going on, I find
                myself having to fake that magic in the mock or in my tests. I am
                duplicating logic between the model and the mock model when it would have
                been much simpler to use the real model in the first place.

                Lesson learned : Don't use a mock when it's easy enough to use the real
                thing. I think somebody else said that already - but I had to learn it for
                myself :-(


                Kevin

                Diamond Sky Software
                www.diamond-sky.com
              • Ron Jeffries
                ... I haven t replied to charlie, so I ll reply to you. ;- ... I m doing this one ... ... part of this one ... ... and all of this one, with the one test that
                Message 7 of 12 , Nov 1, 2002
                • 0 Attachment
                  On Friday, November 1, 2002, at 3:22:32 PM, Kevin Lawrence wrote:

                  > From: "Charlie Poole" <cpoole@...>
                  >>
                  >> Items 2 and 3 can be looked at by instantiating the form and
                  >> examining the menu and menu item. Something like the following
                  >> untested code:
                  >>
                  >> MyForm form = new MyForm(...);
                  >> Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                  >> Assert( form.someMenuItem.Mnemonic == 'S' );
                  >> etc.
                  >>


                  > I was going to say something similar to Charlie but I got distracted with an
                  > essay about estimating. So I'll delete my reply and I'll add to Charlie's.

                  I haven't replied to charlie, so I'll reply to you. ;->

                  > Here's what I do.

                  > I assert that the shortcut is correct
                  > AssertEquals(Shortcut.CtrlP, menu.Shortcut); - or whatever

                  I'm doing this one ...

                  > I assert that the label is right
                  > AssertEquals("Insert &Section", menu.Text);

                  part of this one ...

                  > I assert that the menu does what it is supposed to when I click it
                  > menu.PerformClick();
                  > AssertEquals("<p></p>", textbox.Text);
                  > AssertEquals(3, textbox.SelectionStart);

                  and all of this one, with the one test that runs &P. It just turned
                  out that way. I'm not inclined to test the menu labels but maybe I'll
                  learn that I should.

                  > I assume that Bill Gates & Co have made the little ampersand thingie and the
                  > shortcut key work right so I don't test that.

                  Yes, me too.

                  > If I wanted to implement your script-driven acceptance tests, I might be
                  > tempted to write them in terms of menu labels rather than shortcut keys (and
                  > assume that they are equivalent).

                  > *input
                  > <p>whatever</p>
                  > *Edit|Insert Section
                  > *output
                  > etc etc etc

                  > That would make it supremely easy to match script commands with menu
                  > objects.

                  Yes. I predict that it will turn out rather close to that way. The
                  menu command will be something like *menu any string at all, and will
                  be implemented by looking for a menu with that string in it, and
                  calling it.


                  ...
                  >>
                  >> Of course, the same thing can be done with the real model by checking
                  >> that the state has been appropriately modified.
                  >>

                  > In my product, I mocked the model like this very early on and - now that
                  > things are a little more complex - I really wish I hadn't. I have had a few
                  > little bugs slip down the crack between testing the model and testing the UI
                  > and - now that my real model has a lot of internal magic going on, I find
                  > myself having to fake that magic in the mock or in my tests. I am
                  > duplicating logic between the model and the mock model when it would have
                  > been much simpler to use the real model in the first place.

                  > Lesson learned : Don't use a mock when it's easy enough to use the real
                  > thing. I think somebody else said that already - but I had to learn it for
                  > myself :-(

                  That's kind of where I'm at, though I can't say why ...

                  Thanks,

                  Ron Jeffries
                  www.XProgramming.com
                  First they ignore you, then they laugh at you, then they fight you, then you win.
                  - Gandhi
                • Ron Jeffries
                  ... Agreed ... ... This kind of test doesn t serve me, somehow. Partly, it s because it s tieing me down to someMenuItem as a name. Partly, there s just
                  Message 8 of 12 , Nov 1, 2002
                  • 0 Attachment
                    On Friday, November 1, 2002, at 2:54:07 PM, Charlie Poole wrote:

                    > Here's a first cut.
                    > * Test that altS causes the model's correct method to be called correctly.

                    > Hmmm... That certainly covers it all in one shot, but it seemed hard
                    > to test. What could possibly go wrong?

                    > Candidates:
                    > 1. The Windows operating system might ignore the accelerator
                    > 2. The Accelerator might not be on the menu item
                    > 3. The menu Item might not be on the form and it's main menu
                    > 4. The menu Item might not generate a click event when clicked
                    > 5. The click event might not be connected to the proper handler
                    > 6. The handler might not call the proper model method.

                    > I put items 1 and 4 in as things that probably don't need to be
                    > tested. Of course, in my own code, it sometimes seems as if one
                    > of these things has happened, but it usually turns out that
                    > I've done something wrong myself.

                    Agreed ...

                    > So how hard is it to test for the other items?

                    > Items 2 and 3 can be looked at by instantiating the form and
                    > examining the menu and menu item. Something like the following
                    > untested code:

                    > MyForm form = new MyForm(...);
                    > Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                    > Assert( form.someMenuItem.Mnemonic == 'S' );
                    > etc.

                    This kind of test doesn't serve me, somehow. Partly, it's because it's
                    tieing me down to "someMenuItem" as a name. Partly, there's just
                    something to rote or obvious about it ... ah. It doesn't drive me to
                    build new functionality in the system, just to put in a menu item. It
                    doesn't drive design either. Something like that ...

                    > Items 5 and 6 could be tested by a mock model or using the
                    > actual model as you've done. It strikes me that a mock model
                    > that did nothing but report on how it had been called and what
                    > it's arguments were would make isolated testing of the ui a
                    > lot easier. So...

                    > MyModel model = new MyModel(...);
                    > MyForm form = new MyForm( model );

                    > // Init form and model as needed

                    > form.MenuItem.PerformClick();
                    > Assert( model.SomeMethodWasCalled );
                    > AssertEquals( "somestring", model.SomeMethodParametersPassed );

                    > Of course, the same thing can be done with the real model by checking
                    > that the state has been appropriately modified.

                    Yes. And it seems to me that the form can be out of sync with the
                    mock. For example, the handlers might have their names reversed and
                    exist but do exactly the wrong thing ...

                    > Like I said, just an initial set of thoughts...

                    And interesting ones at that. Thanks,

                    Ron Jeffries
                    www.XProgramming.com
                    The rules are ways of thinking, not ways to avoid thinking.
                  • Kevin Lawrence
                    From: Ron Jeffries ... You made me question why I do this in my own tests... I decided that it s somewhat analogous to your previous
                    Message 9 of 12 , Nov 1, 2002
                    • 0 Attachment
                      From: "Ron Jeffries" <ronjeffries@...>
                      >
                      > > MyForm form = new MyForm(...);
                      > > Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                      > > Assert( form.someMenuItem.Mnemonic == 'S' );
                      > > etc.
                      >
                      > This kind of test doesn't serve me, somehow. Partly, it's because it's
                      > tieing me down to "someMenuItem" as a name. Partly, there's just
                      > something to rote or obvious about it ... ah. It doesn't drive me to
                      > build new functionality in the system, just to put in a menu item. It
                      > doesn't drive design either. Something like that ...
                      >

                      You made me question why I do this in my own tests...

                      I decided that it's somewhat analogous to your previous dilemma about adding
                      the line of code that makes the thing scroll correctly without tests.

                      The test doesn't drive the design and it doesn't drive you to put new
                      functionality in ... but it does stop you from accidently deleting that line
                      during a later refactoring. You wouldn't want to be in a situation where the
                      story worked for everyone but the users ;-)

                      Sometimes a test is just a test.

                      Kevin
                    • Ron Jeffries
                      ... Yes. I prefer a test that does both. I m not sure that I have them now, and I surely don t always get them. But that s the aesthetic that s driving me. Ron
                      Message 10 of 12 , Nov 1, 2002
                      • 0 Attachment
                        On Friday, November 1, 2002, at 5:52:45 PM, Kevin Lawrence wrote:

                        > The test doesn't drive the design and it doesn't drive you to put new
                        > functionality in ... but it does stop you from accidently deleting that line
                        > during a later refactoring. You wouldn't want to be in a situation where the
                        > story worked for everyone but the users ;-)

                        > Sometimes a test is just a test.

                        Yes. I prefer a test that does both. I'm not sure that I have them
                        now, and I surely don't always get them. But that's the aesthetic
                        that's driving me.

                        Ron Jeffries
                        www.XProgramming.com
                        Bang, bang, Jeffries' silver hammer came down upon their heads ...
                      • Alex Chaffee / Purple Technology
                        ... Oh, that s nice. I ve been trying lately to make the leap from mostly-test-first to fully-test-first and I find it s difficult to go back to the test code
                        Message 11 of 12 , Nov 3, 2002
                        • 0 Attachment
                          On Fri, Nov 01, 2002 at 02:52:45PM -0800, Kevin Lawrence wrote:
                          > From: "Ron Jeffries" <ronjeffries@...>
                          > >
                          > > > MyForm form = new MyForm(...);
                          > > > Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                          > > > Assert( form.someMenuItem.Mnemonic == 'S' );
                          > > > etc.
                          > >
                          > > This kind of test doesn't serve me, somehow.
                          > > ... ah. It doesn't drive me to
                          > > build new functionality in the system, just to put in a menu item. It
                          > > doesn't drive design either. Something like that ...
                          > >
                          >
                          > You made me question why I do this in my own tests...
                          >
                          > I decided that it's somewhat analogous to your previous dilemma about adding
                          > the line of code that makes the thing scroll correctly without tests.
                          >
                          > The test doesn't drive the design and it doesn't drive you to put new
                          > functionality in ... but it does stop you from accidently deleting that line
                          > during a later refactoring. You wouldn't want to be in a situation where the
                          > story worked for everyone but the users ;-)
                          >
                          > Sometimes a test is just a test.

                          Oh, that's nice. I've been trying lately to make the leap from
                          mostly-test-first to fully-test-first and I find it's difficult to go
                          back to the test code in order to add a few lines of test code when I
                          already *know* what the implementation code is going to be -- and it's
                          often going to have fewer lines of code than the test).

                          "Sometimes a test is just a test" captures the spirit of how I can
                          convince myself to go back and add the brief test code for writing
                          literally every line of code test-first. It's not because I need a
                          test to drive the design. It's because I need a test!

                          (Still not sure if it's worth doing all the tim... but at least I'm
                          trying to learn how to do it before I make up my mind about that.)

                          - A



                          --
                          Alex Chaffee mailto:alex@...
                          jGuru - Java News and FAQs http://www.jguru.com/alex/
                          Creator of Gamelan http://www.gamelan.com/
                          Founder of Purple Technology http://www.purpletech.com/
                          Curator of Stinky Art Collective http://www.stinky.com/
                        • Ron Jeffries
                          ... This is good. I ve been pushing as hard as I can on the practices since I got into XP and I have found that it teaches me things, builds good habits, and
                          Message 12 of 12 , Nov 4, 2002
                          • 0 Attachment
                            On Monday, November 4, 2002, at 12:16:43 AM, Alex Chaffee / Purple Technology wrote:

                            > On Fri, Nov 01, 2002 at 02:52:45PM -0800, Kevin Lawrence wrote:
                            >> From: "Ron Jeffries" <ronjeffries@...>
                            >> >
                            >> > > MyForm form = new MyForm(...);
                            >> > > Assert( form.Menu.MenuItems.Contains( form.someMenuItem ) );
                            >> > > Assert( form.someMenuItem.Mnemonic == 'S' );
                            >> > > etc.
                            >> >
                            >> > This kind of test doesn't serve me, somehow.
                            >> > ... ah. It doesn't drive me to
                            >> > build new functionality in the system, just to put in a menu item. It
                            >> > doesn't drive design either. Something like that ...
                            >> >
                            >>
                            >> You made me question why I do this in my own tests...
                            >>
                            >> I decided that it's somewhat analogous to your previous dilemma about adding
                            >> the line of code that makes the thing scroll correctly without tests.
                            >>
                            >> The test doesn't drive the design and it doesn't drive you to put new
                            >> functionality in ... but it does stop you from accidently deleting that line
                            >> during a later refactoring. You wouldn't want to be in a situation where the
                            >> story worked for everyone but the users ;-)
                            >>
                            >> Sometimes a test is just a test.

                            > Oh, that's nice. I've been trying lately to make the leap from
                            > mostly-test-first to fully-test-first and I find it's difficult to go
                            > back to the test code in order to add a few lines of test code when I
                            > already *know* what the implementation code is going to be -- and it's
                            > often going to have fewer lines of code than the test).

                            > "Sometimes a test is just a test" captures the spirit of how I can
                            > convince myself to go back and add the brief test code for writing
                            > literally every line of code test-first. It's not because I need a
                            > test to drive the design. It's because I need a test!

                            > (Still not sure if it's worth doing all the tim... but at least I'm
                            > trying to learn how to do it before I make up my mind about that.)

                            This is good. I've been pushing as hard as I can on the practices
                            since I got into XP and I have found that it teaches me things, builds
                            good habits, and presents interesting and fun discoveries.

                            I wanted to be clear about my comments above about this kind of test.
                            Like you and Kevin, I'll do them if I feel I need them and can't think
                            of anything better. A safety test is better than no test at all.

                            The kind of test that I /prefer/ is the kind I describe above, where
                            the test drives me to do something, and to learn something. Those are
                            the tests, for me, where the riches lie.

                            Ron Jeffries
                            www.XProgramming.com
                            Sorry about your cow ... I didn't know she was sacred.
                          Your message has been successfully submitted and would be delivered to recipients shortly.