Loading ...
Sorry, an error occurred while loading the content.

Q: Writing Acceptance Tests (long)

Expand Messages
  • Paul Michali
    Hi, As a learning exercise, I ve been trying some of the aspects of XP on a sample sample program that I m creating. I m doing XPx1 with test-first (I try -
    Message 1 of 2 , Feb 1, 2001
    View Source
    • 0 Attachment
      Hi,

      As a learning exercise, I've been trying some of the aspects of XP
      on a sample sample program that I'm creating. I'm doing XPx1
      with test-first (I try - but it's tough), unit-tests, cards for stories
      and
      tasks, refactoring, and a metaphor (which also was hard).

      One thing I have not tried is acceptance tests. Partly, because this
      is such a small program and partly because I'm really trying to get
      the test-first thing down.

      Well, now I realized that I should have spent some time trying to
      figure out the acceptance test part as I've hit a problem with the
      app (Doh!). Here's the scoop...

      I've been merrily chugging along, writing tests, getting them to fail,
      getting them to pass, refactoring, etc. Everything seems cool.

      The app, just so you understand what is going on, looks at a
      web page, parses it, grabs off some info, check it, and if it passes
      some criteria, it e-mails the data to one or more users.

      Now, I have unit tests that verify that mail is sent (I use Dixie &
      send mail to a dummy user using this server). I have tests that
      verify that I can read a live web page. I have tests that take
      files that contain snapshots of web pages and I parse them to
      make sure I can find the data and process it correctly. I have
      test that make sure I can read the configuration parameters for
      the program from a properties file. Seems like I have it covered
      pretty well (anything missed?)

      Although I didn't do acceptance tests, I did hard code in a live
      URL and would try it out occasionally, from debug mode in the IDE.

      Before I get to the problem... What kinds of acceptance tests
      should I consider? Should they consider the system as a black
      box or white box? Should I use some of the unit test code to
      make the acceptance test code?

      I was thinking I could setup a web server with a static page and
      make sure that it found the data and sent an e-mail to me (Not
      sure how I could automate this, other than using Dixie and writing
      a program that reads the mail to verify the data, like what I do
      in one test case).

      Any other ideas? Any that would not be so involved (especially for
      this small of a program)?

      So, what happened? Well, I made a property file that pointed to
      an external web site and ran the program as an AT job on my PC. I
      manually viewed the web page in my browser and saw the data so
      I knew to expect an e-mail and what should be in it.

      Well, I never received the e-mail. Next, I just ran the program from
      the command line, instead of as an AT job. Same result. Finally,
      I brought up the IDE, and ran the debugger and... it works!?!

      There are some suspects as to what could cause me not to get
      an e-mail. The properties file has the web page, email addresses,
      data patterns to search, etc. Maybe the code is looking at the
      wrong properties file (one in the JAR is not the same as the
      one the IDE sees). Another is that for some reason it could
      not read the web page, which it treats as "no data found" and
      doesn't generate an error currently. Probably other suspects
      that I haven't thought of yet.

      So faced with this error, I have some issues to figure out.
      First, I think I need to write some unit tests to try to highlight the
      problem better. I'm anxious to see where the unit tests have holes.

      Second, would acceptance tests have caught this (probably)?
      Would they have pointed me enough to find the error?

      Third, what should my debug approach be at this point
      in time? In the past, I would have added (or already had in the
      code) debug print statements and I could have used them to see
      what happens, when not running under the IDE. It's difficult as
      it works fine under the IDE and there are times when the live
      site has data such that no e-mail should be sent.

      (I can hear you already - set up a dummy web server with data...:^)

      Lastly, besides the fact that I didn't do any acceptance tests, and
      didn't try running outside the IDE till I had everything done, did I
      screw up in other places?

      Thanks in advance.


      PCM (Paul Michali)

      Carrier Voice Gateway Business Unit (CVGBU)
      Cisco Systems, Inc.
      250 Apollo Drive
      Chelmsford, MA 01824

      Phone : (800) 572-6771 x 45817 (978) 244-5817 [direct]
      Paging: (800) 365-4578 [voice] pcm@... [email page]
    • Adam Judson
      Okay, dumb question, but do your unit tests all run correctly outside of the IDE as well? It seems an obvious thing to do, but it wasn t completely clear from
      Message 2 of 2 , Feb 2, 2001
      View Source
      • 0 Attachment
        Okay, dumb question, but do your unit tests all run correctly
        outside of the IDE as well? It seems an obvious thing to do,
        but it wasn't completely clear from your note, and I've been
        burned before by forgetting little things like this...

        Adam.

        --- In extremeprogramming@y..., Paul Michali <pcm@c...> wrote:
        >
        > Now, I have unit tests that verify that mail is sent (I use Dixie &
        > send mail to a dummy user using this server). I have tests that
        > verify that I can read a live web page. I have tests that take
        > files that contain snapshots of web pages and I parse them to
        > make sure I can find the data and process it correctly. I have
        > test that make sure I can read the configuration parameters for
        > the program from a properties file. Seems like I have it covered
        > pretty well (anything missed?)
        >
        > Although I didn't do acceptance tests, I did hard code in a live
        > URL and would try it out occasionally, from debug mode in the IDE.
        >
        .
        .
        .
        >
        > Third, what should my debug approach be at this point
        > in time? In the past, I would have added (or already had in the
        > code) debug print statements and I could have used them to see
        > what happens, when not running under the IDE. It's difficult as
        > it works fine under the IDE and there are times when the live
        > site has data such that no e-mail should be sent.
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.