Loading ...
Sorry, an error occurred while loading the content.

TDD Quandary

Expand Messages
  • bigbaseballer
    Well guys, I am really confused about this and I think I need to sleep on it. But before I nod off, I figured I would ask here ;) The issue is this: For much
    Message 1 of 10 , Oct 1, 2004
    • 0 Attachment
      Well guys, I am really confused about this and I think I need to sleep
      on it. But before I nod off, I figured I would ask here ;)

      The issue is this: For much of my project, TDD works out just fine.
      However, there are certain cases where it just becomes one ugly mess.

      How in the world do I test drive graphics (DirectX) code. That is,
      code that really does very little data manipulation at all, but
      basically just works with an API to get things done. And by done, I
      mean drawing objects on the screen. Not something I can really "test"
      in the sense of having inputs and outputs and verifying them, it is
      just so complex that the closest thing to traditional unit testing I
      could do is verify if certain pixels were certain colors ... and I
      assure you that would be a worthless test.

      The idea of mock objects comes up. Except, that idea has two
      implications. First of all, it means that I need to mock the DirectX
      API ... which I could do. It would be a bit annoying, but I could do
      it. But my BIGGEST issue with this approach is this: I have
      historically test driven a public interface, and then refactored some
      internal stuff. But basically, the test shows how the code is supposed
      to work from an external viewpoint. I get complete code coverage this
      way, and from talking to some people it sounds like I am on the right
      track. But what about this? My external interface is not supposed to
      really tell anything about the DirectX api. How do mocks come into
      play when I can't actually put them into use without making the
      interface aware of DirectX?

      Or should I do that? I mean, it seems like it is my only option for
      TDD in this scenario. But I feel really "icky" doing that, because
      from just a coding standpoint, I don't want my public code to reflect
      all these nitty gritty details, the sort of thing I would need to test
      drive.

      I hope this hasn't been too confusing for you ... I really need some
      sleep. This thing is eating away at me. Its quite possible I am just
      being dense, and that TDD will yield a good design regardless of my
      gut instinct. But before I make a decision I really would be
      interested in your input.
    • William Pietri
      Hi, Erik. It sounds like you re thrashing a little, so I m going to throw out some answers in hopes of helping you make some progress. If I misunderstand your
      Message 2 of 10 , Oct 2, 2004
      • 0 Attachment
        Hi, Erik. It sounds like you're thrashing a little, so I'm going to
        throw out some answers in hopes of helping you make some progress. If I
        misunderstand your situation, don't hesitate to ask further.

        On Fri, 2004-10-01 at 22:43, bigbaseballer wrote:
        > How in the world do I test drive graphics (DirectX) code. That is,
        > code that really does very little data manipulation at all, but
        > basically just works with an API to get things done.

        I know zero about DirectX, so you may need to take this with a grain of
        salt. But I did write a program to do web albums from my digital camera
        photos; its output was large globs of (rotated, scaled, mutated) images
        and HTML.

        For the low-level objects, I tested to make sure they did the actual
        work. For images, this involved me comparing output bitmaps to
        known-good bitmaps that I visually inspected before putting in test
        cases. (In doing so, we even caught and reported a JDK bug.) For HTML,
        my assertions were more readable. I can't remember what I did at the
        time, but these days I ususally use XPath assertions.

        But for some tests, testing the final result was slow or messy, at which
        point I mocked things out.

        > The idea of mock objects comes up. Except, that idea has two
        > implications. First of all, it means that I need to mock the DirectX
        > API ... which I could do. It would be a bit annoying, but I could do
        > it.

        Yeah. Here tools can help you. Several different times I've ended up
        mocking out the Servlet API. IntelliJ's IDEA has nice tools for
        generating null implementations of interfaces. By default, I have every
        method throw an UnimplementedException, and I just implement fake
        versions of the ones I actually touch. If you tools don't do this, a
        quick perl script might.

        > But my BIGGEST issue with this approach is this: [...]
        > My external interface is not supposed to
        > really tell anything about the DirectX api. How do mocks come into
        > play when I can't actually put them into use without making the
        > interface aware of DirectX?

        I guess I'm not understanding what the problem is. It's only the lowest
        layer that needs to deal with the external doodads. Where possible, you
        should just let that interaction happen. But where it's inconvenient for
        testing, you just need a way to make sure that there's a way to tell
        your middle layer to use a fake lowest layer rather than the real stuff.

        There are a lot of solutions to this. In this article, Alex Chaffee and
        I describe some approaches:

        http://www-106.ibm.com/developerworks/library/j-mocktest.html

        You also may want to create a neutral, testable interface on top of
        whatever you're writing against. The Java objects for files and
        directories, for example, are pretty ugly and hard to mock. So in the
        past I've created a wrapper around that interface. Then I've created two
        implementations. One talks to the real filesystem, and one works only in
        RAM. Most of my code talks only to my interface. In the production, I
        use the real implementation; in tests, I use the fake one, which is made
        to be fast and easy to write asserts against.

        And I'm sure others here have their own pet techniques.


        > But I feel really "icky" doing that, because
        > from just a coding standpoint, I don't want my public code to reflect
        > all these nitty gritty details, the sort of thing I would need to test
        > drive.

        No, your instinct to hide the details is correct. Learning where to test
        in isolation and where to test the whole shebang takes a while, so don't
        get discouraged.

        If it's any consolation, this is an area where I don't find theory so
        helpful. My decisions about how to test which lumps are almost entirely
        pragmatic. So if one style of testing something becomes slow or awkward,
        don't be afraid to experiment with other techniques.

        William
      • Laurent Bossavit
        Erik, ... When such code is buggy, in exactly what ways does it tend to be broken ? How can you tell ? Cheers, -[Laurent]- It is tempting to suppose that
        Message 3 of 10 , Oct 2, 2004
        • 0 Attachment
          Erik,

          > How in the world do I test drive graphics (DirectX) code.

          When such code is buggy, in exactly what ways does it tend to be
          broken ? How can you tell ?

          Cheers,

          -[Laurent]-
          It is tempting to suppose that everyone might fully realize
          his powers and that some at least can become complete exemplars
          of humanity. But this is impossible. It is a feature of human
          sociability that we are by ourselves but parts of what we might
          be. We must look to others to attain the excellences that we
          must leave aside, or lack altogether. -- John Rawls
        • Ken Boucher
          ... What strikes me as painful about this is tt seems that the tests involved are mostly testing Direct X, not your code, and as far as that goes, they re only
          Message 4 of 10 , Oct 2, 2004
          • 0 Attachment
            > How in the world do I test drive graphics (DirectX) code. That is,
            > code that really does very little data manipulation at all, but
            > basically just works with an API to get things done. And by done, I
            > mean drawing objects on the screen.

            What strikes me as painful about this is tt seems that the tests
            involved are mostly testing Direct X, not your code, and as far as
            that goes, they're only testing how the version of Direct-X you have
            installed on that machine actually works. It makes me wonder if TDD
            is the right tool for that particular problem or if you want a different
            tool for that layer.

            Software tools that may help:
            http://www.parasoft.com/jsp/products/article.jsp?articleId=1500 for C++
            http://www.mvps.org/directx/smalltalk/articles/startmtdx8.htm for smalltalk
          • Ron Jeffries
            ... Good questions, Laurent [and Erik]. This will help us decide which tests are necessary. TDD, as I do it, is for testing the logic of my program. What s
            Message 5 of 10 , Oct 2, 2004
            • 0 Attachment
              On Saturday, October 2, 2004, at 6:41:48 AM, Laurent Bossavit wrote:

              >> How in the world do I test drive graphics (DirectX) code.

              > When such code is buggy, in exactly what ways does it tend to be
              > broken ? How can you tell ?

              Good questions, Laurent [and Erik]. This will help us decide which tests
              are necessary.

              TDD, as I do it, is for testing the "logic" of my program. What's the logic
              of the programs you are writing, Erik?

              Ron Jeffries
              www.XProgramming.com
              This is how I develop software.
              Take the parts that make sense to you.
              Ignore the rest.
            • Phlip
              ... Write a temporary test function called reveal() . It will activate the MS Windows event queue, and allow a testee to paint on the screen, and stay up
              Message 6 of 10 , Oct 2, 2004
              • 0 Attachment
                bigbaseballer wrote:

                > How in the world do I test drive graphics (DirectX)
                > code. That is,
                > code that really does very little data manipulation
                > at all, but
                > basically just works with an API to get things done.
                > And by done, I
                > mean drawing objects on the screen. Not something I
                > can really "test"
                > in the sense of having inputs and outputs and
                > verifying them, it is
                > just so complex that the closest thing to
                > traditional unit testing I
                > could do is verify if certain pixels were certain
                > colors ... and I
                > assure you that would be a worthless test.

                Write a temporary test function called 'reveal()'. It
                will activate the MS Windows event queue, and allow a
                testee to paint on the screen, and stay up until you
                close it.

                Call it at the bottom of your test case. The case will
                set up a scenario, then each time you hit the test
                button a window appears, and you can visually inspect
                the results. When you like the appearance, comment out
                the 'reveal()'.

                Sadly, this technique boosts early velocity, and
                without real assertions in your code the technique
                will lead to source cannot refactor without high risk.

                > The idea of mock objects comes up. Except, that idea
                > has two
                > implications. First of all, it means that I need to
                > mock the DirectX
                > API ... which I could do. It would be a bit
                > annoying, but I could do
                > it.

                The style is called "Mock Graphics", and one should
                use it as a last resort.

                I recently wrote an OpenGL application using Mock
                Graphics directly at the OpenGL boundary, based on the
                glTrace package by Phil Frisbie, Jr. (When Kent Beck
                finishes reviewing this effort, I might get to publish
                it...)

                The unwisely curious can get the sample code early, in
                the CVS repository at http://sf.net/projects/flea

                > But my BIGGEST issue with this approach is
                > this: I have
                > historically test driven a public interface, and
                > then refactored some
                > internal stuff. But basically, the test shows how
                > the code is supposed
                > to work from an external viewpoint. I get complete
                > code coverage this
                > way, and from talking to some people it sounds like
                > I am on the right
                > track. But what about this? My external interface is
                > not supposed to
                > really tell anything about the DirectX api. How do
                > mocks come into
                > play when I can't actually put them into use without
                > making the
                > interface aware of DirectX?

                Here is a "does this help?" answer: Copy your code
                into a scratch sandbox project, outside your version
                control system, and then do Exploratory Testing and
                Spike Solutions in it. Those are big words for "noodle
                around". Make the code do what you want, and neglect
                the tests. Then write beautiful tests that look like
                they somehow came before the tested code.

                Then copy the tests, line by line, back into the main
                project, and use them to truly test-first the design
                your scratch project arrived at.


                =====
                Phlip
                http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces



                _______________________________
                Do you Yahoo!?
                Declare Yourself - Register online to vote today!
                http://vote.yahoo.com
              • bigbaseballer
                ... the logic ... Ok, to the first question, buggy would be either a) it crashes, or b) there are visual artifacts that we don t like. That second one is sort
                Message 7 of 10 , Oct 2, 2004
                • 0 Attachment
                  --- In extremeprogramming@yahoogroups.com, Ron Jeffries
                  <ronjeffries@X...> wrote:
                  > On Saturday, October 2, 2004, at 6:41:48 AM, Laurent Bossavit wrote:
                  >
                  > >> How in the world do I test drive graphics (DirectX) code.
                  >
                  > > When such code is buggy, in exactly what ways does it tend to be
                  > > broken ? How can you tell ?
                  >
                  > Good questions, Laurent [and Erik]. This will help us decide which tests
                  > are necessary.
                  >
                  > TDD, as I do it, is for testing the "logic" of my program. What's
                  the logic
                  > of the programs you are writing, Erik?

                  Ok, to the first question, buggy would be either a) it crashes, or b)
                  there are visual artifacts that we don't like. That second one is sort
                  of broad ... but really that kind of comes down to acceptance testing
                  rather than unit testing.

                  And there is logic in this part of the code, I need to manage the
                  DirectX state, and how I actually draw certain things. But other than
                  managing DirectX, most of the actual logic is in performance
                  optimizations, and I am looking to stay away from that (for now). But
                  when I start, I want to have something to fallback on.

                  I think I will do what Philip said. It is actually a really good idea,
                  under the circumstances ... I really think I could write decent tests
                  if I knew exactly what I was doing, versus just knowing what I want
                  the end result to be.
                • Laurent Bossavit
                  Erik, ... I would think unit-testing-at-all would be sufficient to catch most of these ? Or is there something else I should know about the kind of DirectX
                  Message 8 of 10 , Oct 2, 2004
                  • 0 Attachment
                    Erik,

                    > Ok, to the first question, buggy would be either a) it crashes,

                    I would think unit-testing-at-all would be sufficient to catch most
                    of these ? Or is there something else I should know about the kind of
                    DirectX coding errors, specifically, that make it crash ?

                    > b) there are visual artifacts that we don't like.

                    Can you recall a specific occasion where you had written code that
                    produced a visual artifact, and the code should have been written
                    another way ?

                    Cheers,

                    -[Laurent]-
                    The greatest obstacle to transforming the world is that we lack the
                    clarity and imagination to conceive that it could be different.
                    Roberto Mangabeira Unger
                  • Amir Kolsky
                    Whenever we come across an external public interface that we don t own - usually packaged as a DLL we have a four stage process: 1. We write a set of tests
                    Message 9 of 10 , Oct 2, 2004
                    • 0 Attachment
                      Whenever we come across an external public interface that we don't own -
                      usually packaged as a DLL we have a four stage process:

                      1. We write a set of tests that exercise the interface to achieve what we
                      want it to do, in isolate. This is often called a spike, with the difference
                      that we keep it around rather than throw it away. The code in the spike
                      serves to prove that we can do what we want as well as provide a set of
                      examples on how to use the external interface.

                      3. Based on the spike we write a mock. The stage is to have the mock pass
                      all the spike tests. This is sorta like test driving the mock :-). But,
                      before we get around to this we must ensure that we have

                      2. An interface that both the mock and the access to the dll will go
                      through. If the external interface is good enough, we use it. Usually,
                      though, we roll our own interface with just the functionality we need.

                      3. Now we have a working mock which we can use to TDD the application which
                      uses our interface.

                      Amir Kolsky
                      XP& Software


                      ]-----Original Message-----
                      ]From: bigbaseballer [mailto:erik.davis@...]
                      ]Sent: Saturday, October 02, 2004 7:43 AM
                      ]To: extremeprogramming@yahoogroups.com
                      ]Subject: [XP] TDD Quandary
                      ]
                      ]
                      ]Well guys, I am really confused about this and I think I need
                      ]to sleep on it. But before I nod off, I figured I would ask here ;)
                      ]
                      ]The issue is this: For much of my project, TDD works out just fine.
                      ]However, there are certain cases where it just becomes one ugly mess.
                      ]
                      ]How in the world do I test drive graphics (DirectX) code. That
                      ]is, code that really does very little data manipulation at
                      ]all, but basically just works with an API to get things done.
                      ]And by done, I mean drawing objects on the screen. Not
                      ]something I can really "test"
                      ]in the sense of having inputs and outputs and verifying them,
                      ]it is just so complex that the closest thing to traditional
                      ]unit testing I could do is verify if certain pixels were
                      ]certain colors ... and I assure you that would be a worthless test.
                      ]
                      ]The idea of mock objects comes up. Except, that idea has two
                      ]implications. First of all, it means that I need to mock the
                      ]DirectX API ... which I could do. It would be a bit annoying,
                      ]but I could do it. But my BIGGEST issue with this approach is
                      ]this: I have historically test driven a public interface, and
                      ]then refactored some internal stuff. But basically, the test
                      ]shows how the code is supposed to work from an external
                      ]viewpoint. I get complete code coverage this way, and from
                      ]talking to some people it sounds like I am on the right track.
                      ]But what about this? My external interface is not supposed to
                      ]really tell anything about the DirectX api. How do mocks come
                      ]into play when I can't actually put them into use without
                      ]making the interface aware of DirectX?
                      ]
                      ]Or should I do that? I mean, it seems like it is my only
                      ]option for TDD in this scenario. But I feel really "icky"
                      ]doing that, because from just a coding standpoint, I don't
                      ]want my public code to reflect all these nitty gritty details,
                      ]the sort of thing I would need to test drive.
                      ]
                      ]I hope this hasn't been too confusing for you ... I really
                      ]need some sleep. This thing is eating away at me. Its quite
                      ]possible I am just being dense, and that TDD will yield a good
                      ]design regardless of my gut instinct. But before I make a
                      ]decision I really would be interested in your input.
                      ]
                      ]
                      ]
                      ]To Post a message, send it to: extremeprogramming@...
                      ]
                      ]To Unsubscribe, send a blank message to:
                      ]extremeprogramming-unsubscribe@...
                      ]
                      ]ad-free courtesy of objectmentor.com
                      ]Yahoo! Groups Links
                      ]
                      ]
                      ]
                      ]
                      ]
                      ]
                      ]
                      ]
                    • J. B. Rainsberger
                      ... I might use the Gold Master technique here. Draw the screen and check it by hand once. Now capture it and check future draws against the output you
                      Message 10 of 10 , Oct 2, 2004
                      • 0 Attachment
                        bigbaseballer wrote:

                        > How in the world do I test drive graphics (DirectX) code. That is,
                        > code that really does very little data manipulation at all, but
                        > basically just works with an API to get things done. And by done, I
                        > mean drawing objects on the screen. Not something I can really "test"
                        > in the sense of having inputs and outputs and verifying them, it is
                        > just so complex that the closest thing to traditional unit testing I
                        > could do is verify if certain pixels were certain colors ... and I
                        > assure you that would be a worthless test.

                        I might use the Gold Master technique here. Draw the screen and check it
                        by hand once. Now capture it and check future draws against the output
                        you verified by hand. If a future draw is different from the Gold
                        Master, the test fails. Time to investigate. It's not TDD, but it
                        provides a refactoring safety net.

                        Another technique is to think of driving the DirectX API as generating a
                        sequence of events that a DirectXEventListener catches and sends to the
                        API. The techniques for testing event sources and event listeners are
                        well documented. (See _JUnit Recipes_, chapter 14.) Now the goal is to
                        verify that your code generates the proper set (or sequence) of events,
                        the result of which would be a correctly-drawn screen. The set of events
                        could be quite big. If it is, then I highly recommend testing this with
                        FIT/FitNesse, rather than xUnit.

                        I hope this helps.
                        --
                        J. B. (Joe) Rainsberger
                        Diaspar Software Services
                        http://www.diasparsoftware.com :: +1 416 791-8603
                        Predictable, repeatable, quality delivery
                      Your message has been successfully submitted and would be delivered to recipients shortly.