Loading ...
Sorry, an error occurred while loading the content.

Re: [junit] A test that will always fail

Expand Messages
  • Jason Rogers
    On Mon, 2004-03-01 at 17:38, Jonathan Oddy wrote: [snip] I have worked in situations in the past like this (the fix is dependent on another team, internally or
    Message 1 of 8 , Mar 1 5:45 AM
    • 0 Attachment
      On Mon, 2004-03-01 at 17:38, Jonathan Oddy wrote:
      [snip]

      I have worked in situations in the past like this (the fix is dependent
      on another team, internally or externally)... in order to document the
      problem we wrote the test. Then, in order to get green bar we wrapped
      the test in a time bomb block:

      if (currentDate.isAfter(expirationDate)) {
      ... run test ...
      }

      The approach is feasible in the cases when you know a fix is supposed(!)
      to come at some appointed time. One problem with this approach however,
      is that the test is not run unless you have passed the expiration
      point. At this time the system may have evolved to a point that the
      test will need major recoding in order to work.

      Another solution I have used is to wrap the test in a FailableTest
      framework so that it always gets run but is reported elsewhere. The
      easiest way to do this is to override runTest() and catch throwable.
      You then need to come up with a way to report that the test failed.

      Having more than one feedback point can obviously lead to headaches, as
      it did in the team I was in, so we finally evolved that solution to
      extending the UI framework and adding a warning type of failure that
      turned the bar yellow. That may require a lot more work than what you
      are up for if you are running in Eclipse. The team I was on at the time
      scoped it out at about 20 - 30 man hours, b/c no one knew the Eclipse
      plugin environment. The implementation in the vanilla-JUnit framework
      was not very hard, so once you get past the SWT gotchas and the plugin
      framework of Eclipse, it may not be that tedious.

      --
      Jason Rogers

      "Where there is no vision, the people perish..."
      Bible, Proverbs 29:18
    • Jonathan Oddy
      Dear All, In my current project I am responsible for writing a UI client to respond to a number of different events sent to it from some server. Earlier today
      Message 2 of 8 , Mar 1 9:38 AM
      • 0 Attachment
        Dear All,

        In my current project I am responsible for writing a UI client to respond to
        a number of different events sent to it from some server. Earlier today a
        developer noticed a UI bug where by a selected row within one of my reports
        was not maintaining its selection state whilst processing events. After
        investigation I wrote a number of unit tests to highlight the fault and as a
        result fix the bug. However on testing in a live environment I noticed my
        application still exhibited the same erroneous behaviour, this I have
        tracked down to an error condition within the server - in certain
        circumstances the server sends an additional erroneous event to the client.
        The client *must* assume each event received is both valid and correct, and
        acts accordingly, unfortunately what the user sees is the same visual bug.

        Whilst investigating this I wrote a further test that demonstrates that the
        client will always fail given a particular series of events. I like this
        test because it documents that there is a problem within our system - which
        is useful when QA/support come over some time in the future to query the
        behaviour the failing test reminds me where the problem is and what needs to
        be done to fix it. However being pragmatic I can see the server guys not
        fixing this for some time (in fact as we move towards a major release of the
        server on a different branch it is a moot point as to whether it will ever
        be fixed on this branch). All of this means that I probably now have a
        red-bar test for the rest of this branch's life - and this doesn't sit well
        with me.

        So I am asking whether anyone else has had a similar experience and what
        they did to resolve it? The only thing I can think of at the moement is to
        generate a couple of suites - one with all my tests in it, the other with
        all but my failing test. I don't like that really because I like the simple
        click and go approach that Eclipse gives me so I would still always get a
        red-bar, and secondly who says I'd ever remember to run this special suite?

        Peoples thoughts most welcome.

        Jonathan



        --
        Jonathan Oddy
        SERVICEPower Business Solutions Ltd
        Petersgate House - St Petersgate - Stockport - SK1 1HE - ENGLAND
        Tel: +44 (0)161 476 2277
        Email: j.oddy@... <mailto:j.oddy@...>
      • Kevin Klinemeier
        Hi Jonathan, I remember when this same sort of question (as I ve distilled it below) came up a few months ago. I m sure the archives have more, but here s one
        Message 3 of 8 , Mar 1 10:06 AM
        • 0 Attachment
          Hi Jonathan,

          I remember when this same sort of question (as I've distilled it below)
          came up a few months ago. I'm sure the archives have more, but here's
          one of the suggestions I found interesting:

          Turn the sense of the test around. Instead of a test for the bug that
          always fails, turn it into a test for the erroneous behavior. This
          test will pass as long as the bug is in your code. (which can make
          sense if you think of it as testing your assumptions about your
          library)

          Once the "server guys" get around to fixing it, your test will fail
          again, and hopefully your test documentation will explain what it is
          you need to do.

          Of course, if the nature of this bug is that the server-side fix will
          just fix it without changes on your end, I'd be tempted to just send
          the test to the server-side guys and forget about it.

          -Kevin

          --- Jonathan Oddy <j.oddy@...> wrote:
          > Dear All,
          >
          > a developer noticed a UI bug [...] in certain circumstances the
          > server sends an additional erroneous event to the client.



          __________________________________
          Do you Yahoo!?
          Get better spam protection with Yahoo! Mail.
          http://antispam.yahoo.com/tools
        • J. B. Rainsberger
          Jonathan Oddy wrote: ... One possibility is to build an alternative production implementation that compensates for the defect, then use /that/ in
          Message 4 of 8 , Mar 1 11:52 AM
          • 0 Attachment
            Jonathan Oddy wrote:
            <snip />
            > So I am asking whether anyone else has had a similar experience and what
            > they did to resolve it?

            One possibility is to build an alternative production implementation
            that compensates for the defect, then use /that/ in production. If your
            client has to work with their server, then your client /has/ to respect
            the server's protocol, even if it doesn't make sense. Use method/class
            names or comments liberally and have some kind of automated notification
            mechanism /somewhere/ (a calendar tool or something?) to remind you that
            a future release of the server /might/ fix the problem.

            If you make the client match the (bad) server behavior, but keep
            documentation to that effect, then later, when they fix the server, your
            test will fail and you'll know that fixing the server caused the
            failure. At that point, fix the client.
            --
            J. B. Rainsberger,
            Diaspar Software Services
            http://www.diasparsoftware.com :: +1 416 791-8603
            Let's write software that people understand
          • Vladimir Ritz Bossicard
            If you don t mind using another (beta) runner, you can check the JUnit-addons runner (junit-addons @ sf). You can omit a test by appending _ignored to its
            Message 5 of 8 , Mar 1 1:39 PM
            • 0 Attachment
              If you don't mind using another (beta) runner, you can check the JUnit-addons
              runner (junit-addons @ sf). You can omit a test by appending '_ignored' to its
              name. The runner will not execute the test but print out the name into the
              standard output (as a reminder).

              -Vladimir

              --
              Vladimir Ritz Bossicard
              www.bossicard.com

              ----------------------------------------------------------------
              This message was sent using IMP, the Internet Messaging Program.
            • Jason Rogers
              ... I liked Kevin s idea. Clear, succint, and it should break when the server guys fix their bug (or did I misunderstand you there). -- Jason Rogers Where
              Message 6 of 8 , Mar 2 1:08 AM
              • 0 Attachment
                On Tue, 2004-03-02 at 11:16, Jonathan Oddy wrote:
                > Dear All,
                >
                > Thanks for your, as always, informed responses.
                >
                > Kevin hit the nail on the head with his observation that the server-side fix
                > doesn't/won't require any client-side changes. Which is my problem - there
                > isn't a client solution to this bug - it's an asynchronous data stream that
                > is actually a valid sequence of data, it's just that in *this* scenario the
                > sequence is erroneous but the actual client code has no way of knowing that,
                > so unfortunately, JB, it's not possible to provide an alternate client
                > implementation. I like the green-red-yellow bar solution, if only I had the
                > time...:)
                >
                > Thanks again

                I liked Kevin's idea. Clear, succint, and it should break when the
                server guys fix their bug (or did I misunderstand you there).

                --
                Jason Rogers

                "Where there is no vision, the people perish..."
                Bible, Proverbs 29:18
              • Jonathan Oddy
                Dear All, Thanks for your, as always, informed responses. Kevin hit the nail on the head with his observation that the server-side fix doesn t/won t require
                Message 7 of 8 , Mar 2 3:16 AM
                • 0 Attachment
                  Dear All,

                  Thanks for your, as always, informed responses.

                  Kevin hit the nail on the head with his observation that the server-side fix
                  doesn't/won't require any client-side changes. Which is my problem - there
                  isn't a client solution to this bug - it's an asynchronous data stream that
                  is actually a valid sequence of data, it's just that in *this* scenario the
                  sequence is erroneous but the actual client code has no way of knowing that,
                  so unfortunately, JB, it's not possible to provide an alternate client
                  implementation. I like the green-red-yellow bar solution, if only I had the
                  time...:)

                  Thanks again

                  Jonathan

                  -----Original Message-----
                  From: Kevin Klinemeier [mailto:zipwow@...]
                  Sent: 01 March 2004 18:06
                  To: junit@yahoogroups.com
                  Subject: Re: [junit] A test that will always fail



                  Hi Jonathan,

                  I remember when this same sort of question (as I've distilled it below)
                  came up a few months ago. I'm sure the archives have more, but here's
                  one of the suggestions I found interesting:

                  Turn the sense of the test around. Instead of a test for the bug that
                  always fails, turn it into a test for the erroneous behavior. This
                  test will pass as long as the bug is in your code. (which can make
                  sense if you think of it as testing your assumptions about your
                  library)

                  Once the "server guys" get around to fixing it, your test will fail
                  again, and hopefully your test documentation will explain what it is
                  you need to do.

                  Of course, if the nature of this bug is that the server-side fix will
                  just fix it without changes on your end, I'd be tempted to just send
                  the test to the server-side guys and forget about it.

                  -Kevin

                  --- Jonathan Oddy <j.oddy@...> wrote:
                  > Dear All,
                  >
                  > a developer noticed a UI bug [...] in certain circumstances the
                  > server sends an additional erroneous event to the client.



                  __________________________________
                  Do you Yahoo!?
                  Get better spam protection with Yahoo! Mail.
                  http://antispam.yahoo.com/tools <http://antispam.yahoo.com/tools>



                  _____

                  Yahoo! Groups Links


                  * To visit your group on the web, go to:
                  http://groups.yahoo.com/group/junit/ <http://groups.yahoo.com/group/junit/>


                  * To unsubscribe from this group, send an email to:
                  junit-unsubscribe@yahoogroups.com
                  <mailto:junit-unsubscribe@yahoogroups.com?subject=Unsubscribe>


                  * Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service
                  <http://docs.yahoo.com/info/terms/> .




                  [Non-text portions of this message have been removed]
                • J. B. Rainsberger
                  ... In that case, just code the client to match the server s weird behavior, but include documentation with the tests/production code that makes that clear.
                  Message 8 of 8 , Mar 2 7:47 AM
                  • 0 Attachment
                    Jonathan Oddy wrote:

                    > Dear All,
                    >
                    > Thanks for your, as always, informed responses.
                    >
                    > Kevin hit the nail on the head with his observation that the server-side fix
                    > doesn't/won't require any client-side changes. Which is my problem - there
                    > isn't a client solution to this bug - it's an asynchronous data stream that
                    > is actually a valid sequence of data, it's just that in *this* scenario the
                    > sequence is erroneous but the actual client code has no way of knowing that,
                    > so unfortunately, JB, it's not possible to provide an alternate client
                    > implementation. I like the green-red-yellow bar solution, if only I had the
                    > time...:)

                    In that case, just code the client to match the server's weird behavior,
                    but include documentation with the tests/production code that makes that
                    clear. Some day, the server will be fixed and you'll want enough
                    information in the code to remind you that you need to fix the client
                    accordingly.

                    Take care.
                    --
                    J. B. Rainsberger,
                    Diaspar Software Services
                    http://www.diasparsoftware.com :: +1 416 791-8603
                    Let's write software that people understand
                  Your message has been successfully submitted and would be delivered to recipients shortly.