Loading ...
Sorry, an error occurred while loading the content.

How do you unit test web interfaces?

Expand Messages
  • Desilets, Alain
    Hi folks, I m about to embark on two projects that will involve a lot of web development. I thought I would take this opportunity to try and upgrade my web
    Message 1 of 22 , Apr 23, 2007
    • 0 Attachment
      Hi folks,

      I'm about to embark on two projects that will involve a lot of web development. I thought I would take this opportunity to try and upgrade my web testing bag of tricks by asking people on this list how they do it.

      While many Xpers believe that the presentation layer of an application should not be unit tested (cause they think it's too hard and/or it provides little added value over testing the model and the business logic that underlies the UI), I am of the firm opinion that every line of code should be unit tested, and that includes UI code.

      In the past, I have had good success testing the UI of non-web apps, by writing tests that programmatically control the various widgets directly (ex: someButton.click(), where someButton is the variable holding the button widget) instead of through black box scripting
      (ex: clickOnButtonWithCaption("Go"), where we hope the UI is currently displaying a button with that caption).

      But I couldn't find a way to make that kind of approach work for web apps, because I don't know how to get my hands on the variables that hold the widgets.

      So I resort to an approach where I issue CGI requests to the application (actually, I simulate issuing such requests, so I don't have to go through an actual web server), capture the output of the CGI script, then compare that to the expected HTML.

      I find this approach presents two major problems:

      Firstly, it only tests for single CGI requests. I can't easily test a chain of requests like, enter a key word in the search field, click on Go, then click on the first hits in the list.

      Secondly, it's too tightly coupled with the details of the presentation. For example, if I change the HTML template used throughout the app for generating HTML (i.e. the HTML that describes the common "container" inside which specific content is inserted), then all of my tests fail.

      I'm curious to see how people here do it and am looking forward to learning new tricks.


      ----
      Alain Désilets, MASc
      Agent de recherches/Research Officer
      Institut de technologie de l'information du CNRC /
      NRC Institute for Information Technology

      alain.desilets@...
      Tél/Tel (613) 990-2813
      Facsimile/télécopieur: (613) 952-7151

      Conseil national de recherches Canada, M50, 1200 chemin Montréal,
      Ottawa (Ontario) K1A 0R6
      National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
      K1A 0R6

      Gouvernement du Canada | Government of Canada


      ----
      Alain Désilets, MASc
      Agent de recherches/Research Officer
      Institut de technologie de l'information du CNRC /
      NRC Institute for Information Technology

      alain.desilets@...
      Tél/Tel (613) 990-2813
      Facsimile/télécopieur: (613) 952-7151

      Conseil national de recherches Canada, M50, 1200 chemin Montréal,
      Ottawa (Ontario) K1A 0R6
      National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
      K1A 0R6

      Gouvernement du Canada | Government of Canada
    • Bruce Rennie
      Hi Alain, One word: Watir. http://wtr.rubyforge.org/ Watir is Web application testing in Ruby . It allows you to create tests, programmed in Ruby, that
      Message 2 of 22 , Apr 23, 2007
      • 0 Attachment

        Hi Alain,

         

        One word: Watir.

         

        http://wtr.rubyforge.org/

         

        Watir is “Web application testing in Ruby”. It allows you to create tests, programmed in Ruby, that exercise your application through the browser. It uses the DOM to access interface artifacts which can protect your tests from insignificant (moving a control) to semi-significant (changing the visual name of a control) changes to the interface.  On the down side, you need to know Ruby and it only works with IE (though I hear there’s a version under development for FF as well).

         

        There’s also a version called Watin that we’re trying out in our .Net environment.

         

        Cheers,

        /bruce

         

        From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Desilets, Alain
        Sent: Monday, April 23, 2007 3:35 PM
        To: agile-usability@yahoogroups.com
        Subject: [agile-usability] How do you unit test web interfaces?

         

        Hi folks,

        I'm about to embark on two projects that will involve a lot of web development. I thought I would take this opportunity to try and upgrade my web testing bag of tricks by asking people on this list how they do it.

        While many Xpers believe that the presentation layer of an application should not be unit tested (cause they think it's too hard and/or it provides little added value over testing the model and the business logic that underlies the UI), I am of the firm opinion that every line of code should be unit tested, and that includes UI code.

        In the past, I have had good success testing the UI of non-web apps, by writing tests that programmatically control the various widgets directly (ex: someButton.click(), where someButton is the variable holding the button widget) instead of through black box scripting
        (ex: clickOnButtonWithCaption("Go"), where we hope the UI is currently displaying a button with that caption).

        But I couldn't find a way to make that kind of approach work for web apps, because I don't know how to get my hands on the variables that hold the widgets.

        So I resort to an approach where I issue CGI requests to the application (actually, I simulate issuing such requests, so I don't have to go through an actual web server), capture the output of the CGI script, then compare that to the expected HTML.

        I find this approach presents two major problems:

        Firstly, it only tests for single CGI requests. I can't easily test a chain of requests like, enter a key word in the search field, click on Go, then click on the first hits in the list.

        Secondly, it's too tightly coupled with the details of the presentation. For example, if I change the HTML template used throughout the app for generating HTML (i.e. the HTML that describes the common "container" inside which specific content is inserted), then all of my tests fail.

        I'm curious to see how people here do it and am looking forward to learning new tricks.

        ----
        Alain Désilets, MASc
        Agent de recherches/Research Officer
        Institut de technologie de l'information du CNRC /
        NRC Institute for Information Technology

        alain.desilets@...
        Tél/Tel (613) 990-2813
        Facsimile/télécopieur: (613) 952-7151

        Conseil national de recherches Canada, M50, 1200 chemin Montréal,
        Ottawa (Ontario) K1A 0R6
        National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
        K1A 0R6

        Gouvernement du Canada | Government of Canada


        ----
        Alain Désilets, MASc
        Agent de recherches/Research Officer
        Institut de technologie de l'information du CNRC /
        NRC Institute for Information Technology

        alain.desilets@...
        Tél/Tel (613) 990-2813
        Facsimile/télécopieur: (613) 952-7151

        Conseil national de recherches Canada, M50, 1200 chemin Montréal,
        Ottawa (Ontario) K1A 0R6
        National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
        K1A 0R6

        Gouvernement du Canada | Government of Canada

      • Landes Eric (RBNA/CIT2.2)
        If you are using asp.net, there is an nunitasp out there (http://nunitasp.sourceforge.net/) that extends nunit to test web pages in much the manner you
        Message 3 of 22 , Apr 23, 2007
        • 0 Attachment
          If you are using asp.net, there is an nunitasp out there (http://nunitasp.sourceforge.net/) that extends nunit to test web pages in much the manner you describe.  I'm not sure about other options for CGI. 
           

          Eric Landes

          Microsoft MVP

          Editor Crystal Alliance (http://aspalliance.com/crystal)

          http://www.aspadvice.com/blogs/elandes (My Blog)

           


          From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Desilets, Alain
          Sent: Monday, April 23, 2007 3:35 PM
          To: agile-usability@yahoogroups.com
          Subject: [agile-usability] How do you unit test web interfaces?

          Hi folks,

          I'm about to embark on two projects that will involve a lot of web development. I thought I would take this opportunity to try and upgrade my web testing bag of tricks by asking people on this list how they do it.

          While many Xpers believe that the presentation layer of an application should not be unit tested (cause they think it's too hard and/or it provides little added value over testing the model and the business logic that underlies the UI), I am of the firm opinion that every line of code should be unit tested, and that includes UI code.

          In the past, I have had good success testing the UI of non-web apps, by writing tests that programmatically control the various widgets directly (ex: someButton.click( ), where someButton is the variable holding the button widget) instead of through black box scripting
          (ex: clickOnButtonWithCa ption("Go" ), where we hope the UI is currently displaying a button with that caption).

          But I couldn't find a way to make that kind of approach work for web apps, because I don't know how to get my hands on the variables that hold the widgets.

          So I resort to an approach where I issue CGI requests to the application (actually, I simulate issuing such requests, so I don't have to go through an actual web server), capture the output of the CGI script, then compare that to the expected HTML.

          I find this approach presents two major problems:

          Firstly, it only tests for single CGI requests. I can't easily test a chain of requests like, enter a key word in the search field, click on Go, then click on the first hits in the list.

          Secondly, it's too tightly coupled with the details of the presentation. For example, if I change the HTML template used throughout the app for generating HTML (i.e. the HTML that describes the common "container" inside which specific content is inserted), then all of my tests fail.

          I'm curious to see how people here do it and am looking forward to learning new tricks.

          ----
          Alain Désilets, MASc
          Agent de recherches/Research Officer
          Institut de technologie de l'information du CNRC /
          NRC Institute for Information Technology

          alain.desilets@ nrc-cnrc. gc.ca
          Tél/Tel (613) 990-2813
          Facsimile/télé copieur: (613) 952-7151

          Conseil national de recherches Canada, M50, 1200 chemin Montréal,
          Ottawa (Ontario) K1A 0R6
          National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
          K1A 0R6

          Gouvernement du Canada | Government of Canada


          ----
          Alain Désilets, MASc
          Agent de recherches/Research Officer
          Institut de technologie de l'information du CNRC /
          NRC Institute for Information Technology

          alain.desilets@ nrc-cnrc. gc.ca
          Tél/Tel (613) 990-2813
          Facsimile/télé copieur: (613) 952-7151

          Conseil national de recherches Canada, M50, 1200 chemin Montréal,
          Ottawa (Ontario) K1A 0R6
          National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
          K1A 0R6

          Gouvernement du Canada | Government of Canada

        • Pascal Roy
          Alain, You can use Selenium for Unit Testing your UI. The nice thing about it is that you can also use Selenium to do acceptance testing. For unit testing, you
          Message 4 of 22 , Apr 23, 2007
          • 0 Attachment
            Alain,

            You can use Selenium for Unit Testing your UI. The nice thing about it is that you can also use Selenium to do acceptance testing.

            For unit testing, you would use Selenium RC (for Remote Control)). Basically, Selenum RC which includes a server component gives you programmatic control of  the browser accessing the application under test (actually, Selenium Core which is an embedded javascript engine) using your favorite language (java, C#, VB#.NET,...). It supports a few languages but  it's fairly easy to implement the necessary front end if yours is not in the list.
            For acceptance testing, we typically write the tests in Selanese through Selenium IDE which is an IDE for writing acceptance tests in Selanese. It provides interesting facilities including recording your actions in selanese so you can run your test scripts back later on.

            It all works pretty well, and it's from the guys at ThoughtWorks...  Oh yes, it also supports Ajax if you are into that sort of thing...

            Pascal Roy, ing./P.Eng., PMP
            Vice-Président/Vice President
            Elapse Technologies inc.

            [url]        www.elapsetech.com
            [email]]  pascal.roy@...
            [cell]       514-862-6836


            Le 07-04-23 à 15:34, Desilets, Alain a écrit :

            Hi folks,

            I'm about to embark on two projects that will involve a lot of web development. I thought I would take this opportunity to try and upgrade my web testing bag of tricks by asking people on this list how they do it.

            While many Xpers believe that the presentation layer of an application should not be unit tested (cause they think it's too hard and/or it provides little added value over testing the model and the business logic that underlies the UI), I am of the firm opinion that every line of code should be unit tested, and that includes UI code.

            In the past, I have had good success testing the UI of non-web apps, by writing tests that programmatically control the various widgets directly (ex: someButton.click(), where someButton is the variable holding the button widget) instead of through black box scripting
            (ex: clickOnButtonWithCaption("Go"), where we hope the UI is currently displaying a button with that caption).

            But I couldn't find a way to make that kind of approach work for web apps, because I don't know how to get my hands on the variables that hold the widgets.

            So I resort to an approach where I issue CGI requests to the application (actually, I simulate issuing such requests, so I don't have to go through an actual web server), capture the output of the CGI script, then compare that to the expected HTML.

            I find this approach presents two major problems:

            Firstly, it only tests for single CGI requests. I can't easily test a chain of requests like, enter a key word in the search field, click on Go, then click on the first hits in the list.

            Secondly, it's too tightly coupled with the details of the presentation. For example, if I change the HTML template used throughout the app for generating HTML (i.e. the HTML that describes the common "container" inside which specific content is inserted), then all of my tests fail.

            I'm curious to see how people here do it and am looking forward to learning new tricks.

            ----
            Alain Désilets, MASc
            Agent de recherches/Research Officer
            Institut de technologie de l'information du CNRC /
            NRC Institute for Information Technology

            alain.desilets@nrc-cnrc.gc.ca
            Tél/Tel (613) 990-2813
            Facsimile/télécopieur: (613) 952-7151

            Conseil national de recherches Canada, M50, 1200 chemin Montréal,
            Ottawa (Ontario) K1A 0R6
            National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
            K1A 0R6

            Gouvernement du Canada | Government of Canada


            ----
            Alain Désilets, MASc
            Agent de recherches/Research Officer
            Institut de technologie de l'information du CNRC /
            NRC Institute for Information Technology

            alain.desilets@nrc-cnrc.gc.ca
            Tél/Tel (613) 990-2813
            Facsimile/télécopieur: (613) 952-7151

            Conseil national de recherches Canada, M50, 1200 chemin Montréal,
            Ottawa (Ontario) K1A 0R6
            National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
            K1A 0R6

            Gouvernement du Canada | Government of Canada


          • Phlip
            ... That is a Customer Test system. The best unit tests leave out extraneous units, such as web servers and browsers. The best possible way to learn unit
            Message 5 of 22 , Apr 23, 2007
            • 0 Attachment
              Bruce Rennie wrote:

              > Hi Alain,

              > One word: Watir.

              That is a Customer Test system. The best unit tests leave out
              extraneous units, such as web servers and browsers.

              The best possible way to learn unit testing for any web project is to
              study how Ruby on Rails does it.

              Warning: Once you learn Rails, going back to Brand X will be very hard.

              --
              Phlip
              http://www.oreilly.com/catalog/9780596510657/
              "Test Driven Ajax (on Rails)
              assert_xpath, assert_javascript, & assert_ajax
            • George Dinwiddie
              ... As Bruce points out, there s WATIR (and WATIJ and some others) for allowing your tests to drive the browser. There HttpUnit, JwebUnit, and such for
              Message 6 of 22 , Apr 23, 2007
              • 0 Attachment
                Desilets, Alain wrote:
                > But I couldn't find a way to make that kind of approach work for web
                > apps, because I don't know how to get my hands on the variables that
                > hold the widgets.

                As Bruce points out, there's WATIR (and WATIJ and some others) for
                allowing your tests to drive the browser. There HttpUnit, JwebUnit, and
                such for simulating a browser and interacting with your application.

                > Secondly, it's too tightly coupled with the details of the
                > presentation. For example, if I change the HTML template used
                > throughout the app for generating HTML (i.e. the HTML that describes
                > the common "container" inside which specific content is inserted),
                > then all of my tests fail.

                If you're going to check particular things in the HTML, I recommend
                using XPath expressions to reduce the fragility of your assertions. You
                can take a look at http://idiacomputing.com/moin/HtmlTestingUsingXpath
                to see how I used XPath expressions with HttpUnit tests.

                - George

                --
                ----------------------------------------------------------------------
                * George Dinwiddie * http://blog.gdinwiddie.com
                Software Development http://www.idiacomputing.com
                Consultant and Coach http://www.agilemaryland.org
                ----------------------------------------------------------------------
              • Todd Moy
                I actually asked this very same question to one of my friends a few months back. He steered me toward Selenium too. From my light use of it, Selenium seems to
                Message 7 of 22 , Apr 23, 2007
                • 0 Attachment
                  I actually asked this very same question to one of my friends a few months back. He steered me toward Selenium too. From my light use of it, Selenium seems to be a very approachable and easy way to test UIs. You can record actions, or program the test directly if you are so inclined.

                  Another cool thing is that it can integrate with Fitnesse, a wiki-based testing framework. The nice part about this is that you have a more direct relationship to the documentation / requirements and the tests. Of course there's also a corollary benefit of being able to expose your tests readily through the wiki.

                  -TM


                  On 4/23/07, Phlip <phlip2005@...> wrote:

                  Bruce Rennie wrote:

                  > Hi Alain,

                  > One word: Watir.

                  That is a Customer Test system. The best unit tests leave out
                  extraneous units, such as web servers and browsers.

                  The best possible way to learn unit testing for any web project is to
                  study how Ruby on Rails does it.

                  Warning: Once you learn Rails, going back to Brand X will be very hard.

                  --
                  Phlip
                  http://www.oreilly.com/catalog/9780596510657/
                  "Test Driven Ajax (on Rails)
                  assert_xpath, assert_javascript, & assert_ajax




                  --
                  ____________________________
                  oombrella | User Experience Design
                  http://www.oombrella.com
                  oombrella /a/ gmail.com
                • Phlip
                  ... ... None of these are for _unit_ tests. Please write as many tests as possible to the HTML layer, or lower, for best results. Don t be fooled by pretty
                  Message 8 of 22 , Apr 23, 2007
                  • 0 Attachment
                    Todd Moy wrote:

                    > I actually asked this very same question to one of my friends a few months back. He steered me toward Selenium too. From my light use of it, Selenium seems to be a very approachable and easy way to test UIs. You can record actions, or program the test directly if you are so inclined.
                    >
                    > Another cool thing is that it can integrate with Fitnesse
                    ...

                    None of these are for _unit_ tests. Please write as many tests as
                    possible to the HTML layer, or lower, for best results. Don't be
                    fooled by pretty special effects.

                    --
                    Phlip
                    http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!
                  • Håkan Reis
                    Talking about Watir... for asp.net there is also WatiN. http://watin.sourceforge.net/ I havent tested it in a real project but it looks simple enough. --
                    Message 9 of 22 , Apr 23, 2007
                    • 0 Attachment
                      Talking about Watir... for asp.net there is also WatiN. http://watin.sourceforge.net/
                      I havent tested it in a real project but it looks simple enough.

                      --
                      Håkan Reis
                      Dotway AB

                      http://blog.reis.se
                    • Desilets, Alain
                      BTW, thanks for all the excellent leads folks. You ve given me a couple of days of investigation to do at least (hum... Not sure I should be thankful after all
                      Message 10 of 22 , Apr 24, 2007
                      • 0 Attachment
                        BTW, thanks for all the excellent leads folks. You've given me a couple
                        of days of investigation to do at least (hum... Not sure I should be
                        thankful after all ;-), just kiddig of course).

                        > > I actually asked this very same question to one of my
                        > friends a few months back. He steered me toward Selenium too.
                        > From my light use of it, Selenium seems to be a very
                        > approachable and easy way to test UIs. You can record
                        > actions, or program the test directly if you are so inclined.
                        > >
                        > > Another cool thing is that it can integrate with Fitnesse
                        > ...
                        >
                        > None of these are for _unit_ tests. Please write as many
                        > tests as possible to the HTML layer, or lower, for best
                        > results. Don't be fooled by pretty special effects.

                        Phlip, can you explain more precisely what you mean by "writing a test
                        to the HTML layer or lower". I think I know what you mean, but want to
                        be sure.

                        Alain
                      • Phlip
                        ... Convince your HTML generating modules to generate HTML as if a web browser had called them. Then use the XPath query technique to reach into the HTML and
                        Message 11 of 22 , Apr 24, 2007
                        • 0 Attachment
                          > > None of these are for _unit_ tests. Please write as many
                          > > tests as possible to the HTML layer, or lower, for best
                          > > results. Don't be fooled by pretty special effects.
                          >
                          > Phlip, can you explain more precisely what you mean by "writing a test
                          > to the HTML layer or lower". I think I know what you mean, but want to
                          > be sure.

                          Convince your HTML generating modules to generate HTML as if a web
                          browser had called them. Then use the XPath query technique to reach
                          into the HTML and look for details. This probably covers that:

                          http://idiacomputing.com/moin/HtmlTestingUsingXpath

                          --
                          Phlip
                          http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!
                        • Phlip
                          ... server ... ! -- Phlip http://c2.com/cgi/wiki?ZeekLand
                          Message 12 of 22 , Apr 24, 2007
                          • 0 Attachment
                            > Convince your HTML generating modules to generate HTML as if a web

                            server

                            > had called them. Then use the XPath query technique to reach
                            > into the HTML and look for details. This probably covers that:
                            >
                            > http://idiacomputing.com/moin/HtmlTestingUsingXpath

                            !

                            --
                            Phlip
                            http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!
                          • Brian Marick
                            ... I have something of an example here, using the Atomic Object s variant of whatever-Martin-Fowler-calls-Model/View/Presenter-these-days:
                            Message 13 of 22 , Apr 24, 2007
                            • 0 Attachment
                              On Apr 24, 2007, at 8:29 AM, Desilets, Alain wrote:

                              > Phlip, can you explain more precisely what you mean by "writing a test
                              > to the HTML layer or lower". I think I know what you mean, but want to
                              > be sure.

                              I have something of an example here, using the Atomic Object's
                              variant of whatever-Martin-Fowler-calls-Model/View/Presenter-these-days:

                              <http://www.testing.com/cgi-bin/blog/2007/01/05#wireframe2>
                              <http://www.testing.com/cgi-bin/blog/2007/01/11#wireframe3>
                              <http://www.testing.com/cgi-bin/blog/2007/01/18#wireframe4>

                              This example assumes a non-HTML app, but I plan to repurpose it to
                              HTML to show that's tractable. (Haven't started that yet.) The idea
                              of MVP (note: different than MVC) is to make the view very thin and
                              put all the view smarts ("when text is entered in this text box, make
                              that button visible") into a layer below it. The first link is to a
                              movie that explains the Atomic Object style, which uses the Observer
                              pattern heavily.

                              See also Mike Feather's "Humble Dialog Box":
                              <http://www.objectmentor.com/resources/articles/TheHumbleDialogBox.pdf>

                              The Atomic Object style uses mocks heavily. I was a little surprised
                              to find myself deviating from that. What happened is that I found
                              that business-facing tests written as annotated wireframes (and used
                              Fit as an execution engine) seemed like just as good a way to test-
                              drive the presenter layer, and less work besides.

                              I need to get back to that project. So many opportunities to
                              experiment, so little time...

                              -----
                              Brian Marick, independent consultant
                              Mostly on agile methods with a testing slant
                              www.exampler.com, www.exampler.com/blog
                            • Desilets, Alain
                              OK, thx. That was the next step up I was thinking of taking, i.e. instead of just diffing the HTML against a known past output, just assert that the HTML
                              Message 14 of 22 , Apr 24, 2007
                              • 0 Attachment
                                OK, thx. That was the next step up I was thinking of taking, i.e.
                                instead of just diffing the HTML against a known past output, just
                                assert that the HTML contains specific things I am expecting to find.

                                This would certainly address my first problem (i.e. all my tests failing
                                as soon as the HTML container is changed).

                                But what about the other issue I was raising, i.e. the fact that you are
                                not testing pathes through various HTML pages.

                                In the past, I have been burned badly by this. For example, I would have
                                one HTML page whose URL was:

                                http://www.somewhere.com/cgi-bin/a-script.cgi

                                that was supposed to have a link to a page with URL:

                                http://www.somewhere.com/cgi-bin/another-script.cgi?some-argument=1

                                I would test that both the a-script.cgi and
                                another-script.cgi?some-argument=1 pages produced the expected HTML.

                                Then, I would refactor the name of the argument to another-script, so
                                that its URL would be:

                                http://www.somewhere.com/cgi-bin/another-script.cgi?refactored-argument-
                                name=1

                                My tests would all run fine, but when I would test interactively
                                (something I found myself doing far too often), I would realize that the
                                first HTML now had a dead link that pointed nowhere known.

                                I guess with the Xpath approach, this is less likely to happen, because
                                I can define some variable which all tests can access, which says
                                something like this:

                                my $argument_name = 'some-argument';

                                And use that in my assertions.

                                Are there other common pitfalls and ways to avoid them?

                                Alain

                                > -----Original Message-----
                                > From: agile-usability@yahoogroups.com
                                > [mailto:agile-usability@yahoogroups.com] On Behalf Of Phlip
                                > Sent: April 24, 2007 11:17 AM
                                > To: agile-usability@yahoogroups.com
                                > Subject: Re: [agile-usability] How do you unit test web interfaces?
                                >
                                > > > None of these are for _unit_ tests. Please write as many
                                > > tests
                                > > as possible to the HTML layer, or lower, for best >
                                > results. Don't be
                                > > fooled by pretty special effects.
                                > >
                                > > Phlip, can you explain more precisely what you mean by "writing a
                                > > test to the HTML layer or lower". I think I know what you
                                > mean, but
                                > > want to be sure.
                                >
                                > Convince your HTML generating modules to generate HTML as if
                                > a web browser had called them. Then use the XPath query
                                > technique to reach into the HTML and look for details. This
                                > probably covers that:
                                >
                                > http://idiacomputing.com/moin/HtmlTestingUsingXpath
                                >
                                > --
                                > Phlip
                                > http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!
                                >
                                >
                                >
                                > Yahoo! Groups Links
                                >
                                >
                                >
                                >
                              • George Dinwiddie
                                Alain, What is the focus of your testing? Are you trying to determine if a given query produces the correct output? Or are you trying to test a Use Case
                                Message 15 of 22 , Apr 24, 2007
                                • 0 Attachment
                                  Alain,

                                  What is the focus of your testing? Are you trying to determine if a
                                  given query produces the correct output? Or are you trying to test a
                                  Use Case through a series of interactions with the system?

                                  - George

                                  Desilets, Alain wrote:
                                  > OK, thx. That was the next step up I was thinking of taking, i.e.
                                  > instead of just diffing the HTML against a known past output, just
                                  > assert that the HTML contains specific things I am expecting to find.
                                  >
                                  > This would certainly address my first problem (i.e. all my tests failing
                                  > as soon as the HTML container is changed).
                                  >
                                  > But what about the other issue I was raising, i.e. the fact that you are
                                  > not testing pathes through various HTML pages.
                                  >
                                  > In the past, I have been burned badly by this. For example, I would have
                                  > one HTML page whose URL was:
                                  >
                                  > http://www.somewhere.com/cgi-bin/a-script.cgi
                                  >
                                  > that was supposed to have a link to a page with URL:
                                  >
                                  > http://www.somewhere.com/cgi-bin/another-script.cgi?some-argument=1
                                  >
                                  > I would test that both the a-script.cgi and
                                  > another-script.cgi?some-argument=1 pages produced the expected HTML.
                                  >
                                  > Then, I would refactor the name of the argument to another-script, so
                                  > that its URL would be:
                                  >
                                  > http://www.somewhere.com/cgi-bin/another-script.cgi?refactored-argument-
                                  > name=1
                                  >
                                  > My tests would all run fine, but when I would test interactively
                                  > (something I found myself doing far too often), I would realize that the
                                  > first HTML now had a dead link that pointed nowhere known.
                                  >
                                  > I guess with the Xpath approach, this is less likely to happen, because
                                  > I can define some variable which all tests can access, which says
                                  > something like this:
                                  >
                                  > my $argument_name = 'some-argument';
                                  >
                                  > And use that in my assertions.
                                  >
                                  > Are there other common pitfalls and ways to avoid them?
                                  >
                                  > Alain
                                  >
                                  >> -----Original Message-----
                                  >> From: agile-usability@yahoogroups.com
                                  >> [mailto:agile-usability@yahoogroups.com] On Behalf Of Phlip
                                  >> Sent: April 24, 2007 11:17 AM
                                  >> To: agile-usability@yahoogroups.com
                                  >> Subject: Re: [agile-usability] How do you unit test web interfaces?
                                  >>
                                  >>> > None of these are for _unit_ tests. Please write as many
                                  >> > tests
                                  >>> as possible to the HTML layer, or lower, for best >
                                  >> results. Don't be
                                  >>> fooled by pretty special effects.
                                  >>>
                                  >>> Phlip, can you explain more precisely what you mean by "writing a
                                  >>> test to the HTML layer or lower". I think I know what you
                                  >> mean, but
                                  >>> want to be sure.
                                  >> Convince your HTML generating modules to generate HTML as if
                                  >> a web browser had called them. Then use the XPath query
                                  >> technique to reach into the HTML and look for details. This
                                  >> probably covers that:
                                  >>
                                  >> http://idiacomputing.com/moin/HtmlTestingUsingXpath
                                  >>
                                  >> --
                                  >> Phlip
                                  >> http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!
                                  >>
                                  >>
                                  >>
                                  >> Yahoo! Groups Links
                                  >>
                                  >>
                                  >>
                                  >>
                                  >
                                  >
                                  >
                                  > Yahoo! Groups Links
                                  >
                                  >
                                  >
                                  >


                                  --
                                  ----------------------------------------------------------------------
                                  * George Dinwiddie * http://blog.gdinwiddie.com
                                  Software Development http://www.idiacomputing.com
                                  Consultant and Coach http://www.agilemaryland.org
                                  ----------------------------------------------------------------------
                                • Desilets, Alain
                                  ... Both. In some case, I may even need to test that the javascript embedded in the HTML is working properly. Alain
                                  Message 16 of 22 , Apr 24, 2007
                                  • 0 Attachment
                                    > Alain,
                                    >
                                    > What is the focus of your testing? Are you trying to
                                    > determine if a given query produces the correct output? Or
                                    > are you trying to test a Use Case through a series of
                                    > interactions with the system?

                                    Both.

                                    In some case, I may even need to test that the javascript embedded in
                                    the HTML is working properly.

                                    Alain
                                  • George Dinwiddie
                                    ... This all sounds rather far from the concept of unit testing as defined in Test Driven Development. It sounds more like acceptance or customer testing, to
                                    Message 17 of 22 , Apr 24, 2007
                                    • 0 Attachment
                                      Desilets, Alain wrote:
                                      >> Alain,
                                      >>
                                      >> What is the focus of your testing? Are you trying to
                                      >> determine if a given query produces the correct output? Or
                                      >> are you trying to test a Use Case through a series of
                                      >> interactions with the system?
                                      >
                                      > Both.
                                      >
                                      > In some case, I may even need to test that the javascript embedded in
                                      > the HTML is working properly.

                                      This all sounds rather far from the concept of unit testing as defined
                                      in Test Driven Development. It sounds more like acceptance or customer
                                      testing, to me. For that, one of the tools to run a real browser (like
                                      WATIR) is probably the way to go. I'd ask on the agile-testing list. I
                                      think that you'll get better results, though, if you think through what
                                      you're trying to accomplish. "Testing everything" at the GUI is not
                                      often a very productive strategy.

                                      If you're trying to TDD the gui, then you'll find that a variety of
                                      tools are needed, as there is no SwissArmyUnit(tm) tool that does
                                      everything. For TDDing javascript, I've found jsUnit useful.

                                      - George

                                      --
                                      ----------------------------------------------------------------------
                                      * George Dinwiddie * http://blog.gdinwiddie.com
                                      Software Development http://www.idiacomputing.com
                                      Consultant and Coach http://www.agilemaryland.org
                                      ----------------------------------------------------------------------
                                    • Desilets, Alain
                                      ... I agree that testing pathes through several pages is acceptance testing, not unit testing. But I think testing the behaviour of a single web page
                                      Message 18 of 22 , Apr 25, 2007
                                      • 0 Attachment
                                        > > In some case, I may even need to test that the javascript
                                        > embedded in
                                        > > the HTML is working properly.
                                        >
                                        > This all sounds rather far from the concept of unit testing
                                        > as defined in Test Driven Development.
                                        > It sounds more like
                                        > acceptance or customer testing, to me.


                                        I agree that testing pathes through several pages is acceptance testing,
                                        not unit testing.

                                        But I think testing the behaviour of a single web page (including the
                                        behaviour of it's embedded JavaScript code) fits the definition of unit
                                        testing.

                                        In any case, I have never worried too much about whether I am doing unit
                                        or acceptance testing. I always do both, usually at the same time, and
                                        often in the same TestCase files (the later being practice I am trying
                                        to move away from).

                                        > For that, one of the
                                        > tools to run a real browser (like
                                        > WATIR) is probably the way to go. I'd ask on the
                                        > agile-testing list. I think that you'll get better results,
                                        > though, if you think through what you're trying to
                                        > accomplish. "Testing everything" at the GUI is not often a
                                        > very productive strategy.

                                        Thanks for the agile testing hint. I'll post something there (I didn't
                                        know that list existed).

                                        I'm not thinking about testing everything at the GUI level. I always
                                        have more internal tests that check the internals independantly of the
                                        GUI.

                                        But the Gui is code too, and it therefore needs to be tested. I'm
                                        looking for a better way to test it than I have been doing in the past.

                                        > If you're trying to TDD the gui, then you'll find that a
                                        > variety of tools are needed, as there is no SwissArmyUnit(tm)
                                        > tool that does everything. For TDDing javascript, I've found
                                        > jsUnit useful.

                                        Yes, I know. Some collegues and I are currently developing a library of
                                        "manipulators" for wxPython, to make it easier to programmatically
                                        manipulate widgets in a wxPython application for the purpose of unit
                                        testing. I'm surprised we had to write that ourselves.


                                        Alain
                                      • George Dinwiddie
                                        ... It fits *some* definitions of unit testing, but not the one I generally use. I generally use the TDD concept of unit testing, rather than the ones
                                        Message 19 of 22 , Apr 25, 2007
                                        • 0 Attachment
                                          Desilets, Alain wrote:
                                          > But I think testing the behaviour of a single web page (including the
                                          > behaviour of it's embedded JavaScript code) fits the definition of unit
                                          > testing.

                                          It fits *some* definitions of unit testing, but not the one I generally
                                          use. I generally use the TDD concept of unit testing, rather than the
                                          ones generally accepted by the testing community. I don't call anything
                                          "unit testing" if it requires deployment to run.

                                          The overloading of the term "unit testing" has lead me to prefer
                                          "programmer testing" instead.

                                          > Thanks for the agile testing hint. I'll post something there (I didn't
                                          > know that list existed).
                                          >
                                          > I'm not thinking about testing everything at the GUI level. I always
                                          > have more internal tests that check the internals independantly of the
                                          > GUI.
                                          >
                                          > But the Gui is code too, and it therefore needs to be tested. I'm
                                          > looking for a better way to test it than I have been doing in the past.

                                          Yes, and you'll find a good community of testing professionals over
                                          there who will give you good testing advice. There's also a few of us
                                          developers there, who tend to look at things from the "TDD as a design
                                          tool" point of view.

                                          - George

                                          --
                                          ----------------------------------------------------------------------
                                          * George Dinwiddie * http://blog.gdinwiddie.com
                                          Software Development http://www.idiacomputing.com
                                          Consultant and Coach http://www.agilemaryland.org
                                          ----------------------------------------------------------------------
                                        • Desilets, Alain
                                          ... I don t really want to argue about the definition of unit vs acceptance testing. I agree with you that testing which requires deployment sucks, especially
                                          Message 20 of 22 , Apr 25, 2007
                                          • 0 Attachment
                                            > Desilets, Alain wrote:
                                            > > But I think testing the behaviour of a single web page
                                            > (including the
                                            > > behaviour of it's embedded JavaScript code) fits the definition of
                                            > > unit testing.
                                            >
                                            > It fits *some* definitions of unit testing, but not the one I
                                            > generally use. I generally use the TDD concept of unit
                                            > testing, rather than the ones generally accepted by the
                                            > testing community. I don't call anything "unit testing" if
                                            > it requires deployment to run.

                                            I don't really want to argue about the definition of unit vs acceptance testing.

                                            I agree with you that testing which requires deployment sucks, especially if it's manual deployment. I'm looking for tools and techniques that will allow me to do automated testing (whether it be unit or acceptance) of the GUI without requiring too much in the way of deployment. It seems to me that testing the embedded javascript should be possible without full deployment. For example, invoke the script that generates the HTML containing that javascript, then load that HTML into some kind of browser emulator that has a JavaScript interpretor, etc... Dunno if that exists or not.


                                            ----
                                            Alain Désilets, MASc
                                            Agent de recherches/Research Officer
                                            Institut de technologie de l'information du CNRC /
                                            NRC Institute for Information Technology

                                            alain.desilets@...
                                            Tél/Tel (613) 990-2813
                                            Facsimile/télécopieur: (613) 952-7151

                                            Conseil national de recherches Canada, M50, 1200 chemin Montréal,
                                            Ottawa (Ontario) K1A 0R6
                                            National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
                                            K1A 0R6

                                            Gouvernement du Canada | Government of Canada
                                          • Desilets, Alain
                                            ... Thx Brian. I ll have a look. ... Right. When I test non-web GUIs, that s the approach I use. Except that contrarily to Mike, I don t use a MockView. I find
                                            Message 21 of 22 , Apr 25, 2007
                                            • 0 Attachment
                                              > I have something of an example here, using the Atomic
                                              > Object's variant of
                                              > whatever-Martin-Fowler-calls-Model/View/Presenter-these-days:
                                              >
                                              > <http://www.testing.com/cgi-bin/blog/2007/01/05#wireframe2>
                                              > <http://www.testing.com/cgi-bin/blog/2007/01/11#wireframe3>
                                              > <http://www.testing.com/cgi-bin/blog/2007/01/18#wireframe4>
                                              >
                                              > This example assumes a non-HTML app, but I plan to repurpose
                                              > it to HTML to show that's tractable. (Haven't started that
                                              > yet.) The idea of MVP (note: different than MVC) is to make
                                              > the view very thin and put all the view smarts ("when text is
                                              > entered in this text box, make that button visible") into a
                                              > layer below it. The first link is to a movie that explains
                                              > the Atomic Object style, which uses the Observer pattern heavily.

                                              Thx Brian. I'll have a look.

                                              >
                                              > See also Mike Feather's "Humble Dialog Box":
                                              > <http://www.objectmentor.com/resources/articles/TheHumbleDialo
                                              > gBox.pdf>

                                              Right. When I test non-web GUIs, that's the approach I use. Except that contrarily to Mike, I don't use a MockView. I find it's just as easy to test using the actual graphical view. And doing it that way catches all sort of silly little bugs in the view. Things like a button not being properly wired to the model method it's supposed to invoke.

                                              I haven't figured out how to do this in a web context though, hence my post.


                                              ----
                                              Alain Désilets, MASc
                                              Agent de recherches/Research Officer
                                              Institut de technologie de l'information du CNRC /
                                              NRC Institute for Information Technology

                                              alain.desilets@...
                                              Tél/Tel (613) 990-2813
                                              Facsimile/télécopieur: (613) 952-7151

                                              Conseil national de recherches Canada, M50, 1200 chemin Montréal,
                                              Ottawa (Ontario) K1A 0R6
                                              National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
                                              K1A 0R6

                                              Gouvernement du Canada | Government of Canada
                                            • George Dinwiddie
                                              ... If you want to test just the javascript, put it in a .js file and test it with jsUnit. -- ... * George Dinwiddie *
                                              Message 22 of 22 , Apr 25, 2007
                                              • 0 Attachment
                                                Desilets, Alain wrote:
                                                > I agree with you that testing which requires deployment sucks,
                                                > especially if it's manual deployment. I'm looking for tools and
                                                > techniques that will allow me to do automated testing (whether it be
                                                > unit or acceptance) of the GUI without requiring too much in the way
                                                > of deployment. It seems to me that testing the embedded javascript
                                                > should be possible without full deployment. For example, invoke the
                                                > script that generates the HTML containing that javascript, then load
                                                > that HTML into some kind of browser emulator that has a JavaScript
                                                > interpretor, etc... Dunno if that exists or not.

                                                If you want to test just the javascript, put it in a .js file and test
                                                it with jsUnit.

                                                --
                                                ----------------------------------------------------------------------
                                                * George Dinwiddie * http://blog.gdinwiddie.com
                                                Software Development http://www.idiacomputing.com
                                                Consultant and Coach http://www.agilemaryland.org
                                                ----------------------------------------------------------------------
                                              Your message has been successfully submitted and would be delivered to recipients shortly.