Loading ...
Sorry, an error occurred while loading the content.

RE: [agile-usability] test-driven iterative user centered design

Expand Messages
  • Desilets, Alain
    (WARNING: Longish post up ahead ;-)) Geat insight Jeff! I never thought of it that way, but you are right, User Goals are to the UI what Tests are to code.
    Message 1 of 9 , Jan 9, 2006
    • 0 Attachment
      (WARNING: Longish post up ahead ;-))

      Geat insight Jeff! I never thought of it that way, but you are right, User Goals are to the UI what Tests are to code.

      Note however that it does not automatically follow from this that all User Goals need to be spelled out before any screen is drawn or any code is written. Or at least, that's not how it turns out to be for writing code. When we write code Test-First, we do not spend months cranking out the tests for the whole system, then spend months cranking out the code that passes them all. Instead, we write one test, then write the changes that make that test pass, and so on. Of course, as Larry pointed out you have to be careful about making parallels between the coding and the UI worlds, but it COULD be (not saying that it is) that the most efficient way to validate UI design is to spell out or clarify a small set of User Goals, then design or modify the design of part of the UI to meet those new or clarified goals, and so on. Has anyone tried this sort of approach at all?

      Also, it does not necessarily follow from your insight that Design Inspection techniques are better than Testing techniques at finding usability problems. It just means that the User Goals are what you will be inspecting or testing against, and that therefore the User Goals relevant to the part of the UI you are inspecting or testing need to be defined first. But it COULD still be that testing a paper mockup of that part of the UI is more efficient than inspecting a design of that part of the UI. Again, I am not knowlegeable in that area to say... I'm merely pointing out that it's not a foregone conclusion from your insight.

      I spent a good part of this weekend thinking about what has been said in this thread, and trying to make sense of it based on my extensive programming experience, and my limited experience in U*.

      In this thread, we talked about a lot of different things. Mostly, the discussion focused around the relative merits of Inspection versus Testing techniques for finding usability problems in UI.

      But hidden in there is also the usual question of whether techniques that can be done upfront (Design Inspection, Paper Prototyping) are more effective that techniques that can be done at coding-time (Testing a Running Prototype).

      To make things even more interesting, Larry also introduced an analogy between finding usability problems in Uis and finding bugs in code, which brings up the issue of whether the answer to the first two questions is the same for both of these situations.

      Let me post some claims which I welcome you to poke shots at to see if they hold up.

      1) For the purpose of finding bugs in code, TDD (a testing technique) is more efficient than any inspection technique.

      2) For the purpose of finding bugs in code, TDD (a coding-time technique) is more efficient than any upfront technique.

      3) For the purpose of finding usability problems with a UI, upfront techniques (ex: UI Design Inspection, Paper Prototype Testing) are more efficient than techniques coding-time techniques (ex: Testing a Running Prototype).

      4) For the purpose of finding usability problems in UI, I venture no claim as to the relative efficiency of upfront inspection (ex: UI Design Inspection) versus upfront testing (ex: Paper Mockup Testing) techniques

      5) For both the purpose of finding bugs in code and finding usability problems in UIs, you need to apply a variety of techniques beside the most efficient one, because even less efficient techniques will still catch different types of bug/issue.

      Let me support those claims with some arguments, starting with points 1) and 2) with which I am most familiar. Below is an ordered list of techniques I know for finding bugs in code:

      Architecture Design Inspection -- Cognitive Walktrhoughs of PseudoCode -- Formal Code Inspection -- Pair-Programming -- TDD

      Techniques on the left tend to be more upfront and inspection-based, whereas techniques on the left tend to be more coding-time, and testing-based.

      For the purpose of finding bugs in code, TDD beats Formal Code Inspection for reasons I already explained. Unit tests can be automated so that they can be run continuously and exhaustively, and they find future bugs as well as past ones. Formal Code Inspections can only be done infrequently, and must focus on a particular part of the code, and they can only find past bugs, not ones that will happen in the future.

      For the purpose of finding bugs in code, TDD also beats Pair-Programming (PP), because it is more exhaustive. As Larry pointed out, PP is somewhat like a continuous code inspection, which takes away the TDD advantage of being frequent. However, PP can only focus on particular parts of the code at a time, whereas TDD tests ALL of the code ALL of the time. Note that I still believe in PP, because it has many other benefits (better structured code, dissimenation of knowledge and expertise inside the team). Even for the purpose of finding bugs, PP complements TDD well in that it tends to find other kinds of bugs than TDD (for example, security flaws). But I think in terms of number of bugs found per hour, TDD is better than PP.

      For the purpose of finding bugs in code, TDD is better than Cognitive Walkthroughs of PseudoCode (a somewhat upfront testing technique) because PseudoCode is only marginally easier to write than actual code. In fact, I would say that actual code is easier to write than PseudoCode, because there are lots of tools to support code writing and none to support PseudoCode writing. Moreover, you can validate actual code by running it, whereas you can't do that with PseudoCode.

      Finally, for the purpose of finding bugs in code, TDD beats Architecture Design Inspection, because Architectural Design documents do not capture sufficent details about the system (if they did, they would be as large and as costly to write as the code itself), and bugs occur mostly at the level of details, not at the architectural level. But again, Architectural Design Inspections are a good idea (as long as you don't spend months writing the Architectural Design document), because they may point out other kinds of issues like: performance issues, security issues, extensability, interoperability with other systems.

      So to me, claims 1) and 2) are pretty clear.

      So what about "Finding usability issues in UI"? Below is another ordered list of techniques I know of for doing this (smaller because I don't know as much about U* as I do about development):

      Design Inspection -- Paper Prototyping -- Testing running prototype

      Again, left means upfront and inspection-based, right means coding-time and testing-based.

      Contrarily to the bug finding situation, I believe that for the purpose of finding usability issues with a UI, upfront techniques are more efficient than coding-time techniques. That's because I believe it is possible to capture most of the relevant details about user interaction through relatively small artifacts that can be created upfront (ex: Design Documents or Paper Prototypes). This is not the case with finding bugs in code, where any upfront artefact that contains sufficient details to be meaningful for that task would turn out to be almost as big and almost as expensive to produce as the code itself. Note that I still think you should not delay too much before getting code out and testing it, because there are many kinds of usability issues (ex: slow response time) which can only be identified by testing a running prototype. It just means that upfront techniques are your first line of defence because they are much cheaper, yet will identify a lot of issues early.

      I don't have a good sense of the relative efficiency of upfront inspection (ex: Design Document Inspection) vs upfront testing (ex: Testing Paper Prototype) techniques for finding usability issues. But I suspect that both techniques are good, necessary and within the same range as far as efficiency goes. Inspection has the advantage of being somewhat cheaper (but not that much I bet), but I suspect testing with paper prototypes will catch a different set of issues which are more "dynamic" (i.e. have to do with the sequence in which things are done by the user) and which are hard to see when reviewing a "static" design document.

      So, I think point 3) is pretty clear to me now, but I don't know about point 4).

      As to point 5) (need to use several techniques beside the most efficient one), I have already discussed it above, both for finding bugs in code and finding usability issues in UIs.

      Well, that's it. If you have made it this far in reading this post, you might as well react to it ;-).


      ----
      Alain Désilets, MASc
      Agent de recherches/Research Officer
      Institut de technologie de l'information du CNRC /
      NRC Institute for Information Technology

      alain.desilets@...
      Tél/Tel (613) 990-2813
      Facsimile/télécopieur: (613) 952-7151

      Conseil national de recherches Canada, M50, 1200 chemin Montréal,
      Ottawa (Ontario) K1A 0R6
      National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
      K1A 0R6

      Gouvernement du Canada | Government of Canada
    • Jared M. Spool
      ... In my opinion, the reason this is unclear is that it is *very* dependent on the practitioners involved. Previous experience, exposure to research data,
      Message 2 of 9 , Jan 9, 2006
      • 0 Attachment
        At 02:58 PM 1/9/2006, Desilets, Alain wrote:
        >I don't have a good sense of the relative efficiency of upfront inspection
        >(ex: Design Document Inspection) vs upfront testing (ex: Testing Paper
        >Prototype) techniques for finding usability issues. But I suspect that
        >both techniques are good, necessary and within the same range as far as
        >efficiency goes.

        In my opinion, the reason this is unclear is that it is *very* dependent on
        the practitioners involved. Previous experience, exposure to research data,
        innate skill, and a variety of other factors will affect both the results
        of an inspection and the analysis of testing.

        Something the usability/user experience community isn't addressing is how
        much this is a craft, where individual prowess trumps collective wisdom and
        process. We keep wanting to pretend what we do is an engineering
        discipline, but unfortunately, it's a craft where the skill of the
        craftsperson dictates success. This is the big elephant in the room that
        no-one is talking about.

        In research we conducted a while back, we traced where in the development
        process usability problems were introduced into the design. Our major
        finding was that they almost always were introduced at the same point:
        someone on the design team needed to make a decision for which they didn't
        have all the information necessary.

        From this finding, I took away that the more important end result from any
        type of usability activity is an informed decision.

        Design Document Inspection can work if the inspection team arrives at
        informed decisions in the process. That's going to depend on the existing
        knowledge, skills, and deductive insights of the team. Paper Prototype
        Testing arrives at informed decisions in a different manner. It's going to
        depend on the test team's skills at creating tests, recruiting users, and
        making proper inferences from the observations. Both techniques are as
        likely to steer a team wrong as they are to steer a team right.

        In other words, a poorly-executed usability activity will result in poor
        decisions. There are no guarantees.

        I think if we want to make usability into more of an engineering practice,
        we need to control some of these variables a lot better.

        Just my thoughts late this evening...

        Jared


        Jared M. Spool, Founding Principal, User Interface Engineering
        4 Lookout Lane, Unit 4d, Middleton, MA 01949
        978 777-9123 jspool@... http://www.uie.com
        Blog: http://www.uie.com/brainsparks
      • Jon Meads
        Jared Spool writes: In research we conducted a while back, we traced where in the development process usability problems were introduced into the design. Our
        Message 3 of 9 , Jan 9, 2006
        • 0 Attachment
          Jared Spool writes:
          In research we conducted a while back, we traced where in the development process usability problems were introduced into the design. Our major finding was that they almost always were introduced at the same point: someone on the design team needed to make a decision for which they didn't have all the information necessary.
          So true. The problem is that there are variables and things that we just won't know until we test. As unfortunately stated, "There are things that we don't know that we don't know." This is not justification for skipping user research (Sun Tzu had the right idea about learning the lay of the land and the nature of the enemy). It is simply recognition of an empirical aspect to UI design. While it is possible that a UI designer might create a great UI design without user research for one project, there is no guarantee that they'll be able to do so on the next project.
           
          Cheers,
          jon
           

                     Jon Meads
                     Usability Architects, Inc.
                     PO Box 3222
                     Kirkland, WA 98083-3222
           
              Voice: 425-827-9296
               Cell: 206-409-7548
                Fax: 425-827-6692
              Email: jon@...
           
                    Designing the User Experience
                 http://www.usability-architects.com/


        • Larry Constantine
          Warning: Short reaction to a long post. ... User Goals are to ... this thread, and ... and my limited ... they hold up. ... might as well ... And I say amen.
          Message 4 of 9 , Jan 10, 2006
          • 0 Attachment
            Warning: Short reaction to a long post.

            Alain wrote:
            > Great insight Jeff! I never thought of it that way, but you are right,
            User Goals are to
            > the UI what Tests are to code....
            > I spent a good part of this weekend thinking about what has been said in
            this thread, and
            > trying to make sense of it based on my extensive programming experience,
            and my limited
            > experience in U*....
            > Let me post some claims which I welcome you to poke shots at to see if
            they hold up.
            > ...
            > Well, that's it. If you have made it this far in reading this post, you
            might as well
            > react to it ;-).

            And I say amen. Very thorough analysis and absolutely on the mark, Alain.

            --Larry Constantine, IDSA
          • Tobias Mayer
            ... Thanks for raising that. It inspired me to talk about that elephant. My thoughts are not usability-specific, so not repeated here. If you are interested
            Message 5 of 9 , Jan 10, 2006
            • 0 Attachment
              Jared wrote:
              > Something the usability/user experience community isn't addressing is how much this is a craft, where individual prowess trumps collective wisdom and process. We keep wanting to pretend what we do is an engineering discipline, but unfortunately, it's a craft where the skill of the craftsperson dictates success. This is the big elephant in the room that no-one is talking about.
               
              Thanks for raising that.  It inspired me to talk about that elephant.  My thoughts are not usability-specific, so not repeated here.  If you are interested see: http://agilethinking.net/artisans.html
               
              Regards,
              Tobias
              (not an engineer)


              "Jared M. Spool" <jspool@...> wrote:
              At 02:58 PM 1/9/2006, Desilets, Alain wrote:
              >I don't have a good sense of the relative efficiency of upfront inspection
              >(ex: Design Document Inspection) vs upfront testing (ex: Testing Paper
              >Prototype) techniques for finding usability issues. But I suspect that
              >both techniques are good, necessary and within the same range as far as
              >efficiency goes.

              In my opinion, the reason this is unclear is that it is *very* dependent on
              the practitioners involved. Previous experience, exposure to research data,
              innate skill, and a variety of other factors will affect both the results
              of an inspection and the analysis of testing.

              Something the usability/user experience community isn't addressing is how
              much this is a craft, where individual prowess trumps collective wisdom and
              process. We keep wanting to pretend what we do is an engineering
              discipline, but unfortunately, it's a craft where the skill of the
              craftsperson dictates success. This is the big elephant in the room that
              no-one is talking about.

              In research we conducted a while back, we traced where in the development
              process usability problems were introduced into the design. Our major
              finding was that they almost always were introduced at the same point:
              someone on the design team needed to make a decision for which they didn't
              have all the information necessary.

              From this finding, I took away that the more important end result from any
              type of usability activity is an informed decision.

              Design Document Inspection can work if the inspection team arrives at
              informed decisions in the process. That's going to depend on the existing
              knowledge, skills, and deductive insights of the team. Paper Prototype
              Testing arrives at informed decisions in a different manner. It's going to
              depend on the test team's skills at creating tests, recruiting users, and
              making proper inferences from the observations. Both techniques are as
              likely to steer a team wrong as they are to steer a team right.

              In other words, a poorly-executed usability activity will result in poor
              decisions. There are no guarantees.

              I think if we want to make usability into more of an engineering practice,
              we need to control some of these variables a lot better.

              Just my thoughts late this evening...

              Jared


              Jared M. Spool, Founding Principal, User Interface Engineering
              4 Lookout Lane, Unit 4d, Middleton, MA 01949
              978 777-9123   jspool@...  http://www.uie.com
              Blog: http://www.uie.com/brainsparks



            • Desilets, Alain
              Unfortunately, in my world, it s VERY difficult to know ahead of time what I m going to face until we re into the project awhile. And I have to quote a price
              Message 6 of 9 , Jan 19, 2006
              • 0 Attachment
                Unfortunately, in my world, it's VERY difficult to know ahead of time
                what I'm going to face until we're into the project awhile. And I have
                to quote a price for a project before I have enough data to know whether
                I can get away with little outside usability help or need a lot (or need
                a new project sponsor).

                So, I have two questions:

                1. How do you work around this problem realistically? (I'd prefer input
                from people who have tried things and been successful... I've gotten
                pretty good at assessing new clients/projects with very little
                introduction... I'm not talking about hypothetical situations, I'm
                talking real ones).

                -- Alain:
                I guess this is a problem that ALL software consultants face, not just
                agile ones right?

                I asked that question once to Joshua Kerievsky (author of "Refactoring
                to Patterns") and his answer was: "Don't do fixed cost contracts.
                Instead, negociate contracts based on access to resources and control
                over them."

                In other words, instead of selling deliverabls to the customer, you
                should sell something like: X hours of such and such team per month, and
                the privilege of steering them in such and such way.

                Makes a lot of sense to me. I know that's how a lot of professionals
                work. When you hire a lawyer for example, you agree on an hourly rate,
                not on the final product (i.e. a decision of the judge in your favour).

                The problem with this approach is of course that most clients want to
                cling to the illusion of control provided by a cost estimate. Is there
                anyone out there that actually tried this approach and was successful?

                I have only once negociated a contract like that and the client actually
                preferred this kind of arrangement to a fixed cost contract. He was a
                small startup entrepreneur with a vision for a new and highly innovative
                product. Because the product was so innovative, he had no way of knowing
                what the market was exactly, nor which parts of his vision were
                realisable or not, nor which particular variant of his vision would
                catch on. So to him, the idea of being able to pour his limited capital
                one month at a time, and pull the plug whenever he wanted was very
                attractive.

                Note that this doesn't really count as "real-life" experience because
                the contract was never finalised. The reason was not that he did not
                like the format, but rather that he was unable to raise sufficient
                venture capital to start the company (this was 2 years after the
                internet bubble burst, so VC was hard to come by at that time).
                ----
              • kauerrolemodel
                ... Actually, that s roughly the approach we use and it is the best I ve ever used for all involved. Fixed price, variable scope . That gives everyone who
                Message 7 of 9 , Jan 19, 2006
                • 0 Attachment
                  --- In agile-usability@yahoogroups.com, "Desilets, Alain"
                  <alain.desilets@n...> wrote:
                  > In other words, instead of selling deliverabls to the customer, you
                  > should sell something like: X hours of such and such team per month, and
                  > the privilege of steering them in such and such way.
                  > [snip]
                  > The problem with this approach is of course that most clients want to
                  > cling to the illusion of control provided by a cost estimate. Is there
                  > anyone out there that actually tried this approach and was successful?

                  Actually, that's roughly the approach we use and it is the best I've
                  ever used for all involved. "Fixed price, variable scope". That
                  gives everyone who wants to know the budget a number and we virtually
                  always deliver on budget. That makes people happy and life easier for
                  all involved.

                  --- In agile-usability@yahoogroups.com, "Peter Boersma" <peter@p...>
                  wrote:
                  > Ken wrote:
                  > > Unfortunately, in my world, it's VERY difficult to know ahead of time
                  > > what I'm going to face until we're into the project awhile. And I
                  have
                  > > to quote a price for a project before I have enough data to know
                  > > whether I can get away with little outside usability help or need
                  a lot
                  > > (or need a new project sponsor).
                  >
                  > Isn't this true for both the usability work and the software development
                  > costs? Or do you need an expert at estimating the usability
                  complexities of
                  > the project, just like you estimate the complexity of the software
                  required?

                  I think the answer to that is yes.

                  I just helped scope out another project today and I asked a few
                  questions that suggested strongly we would need some more usability
                  help. I'll be putting that in the quote. Now, we just need to land
                  the contract and get someone in to help. I think the first one is
                  high probability. Anyone out there care to help on the latter?

                  I do have a usability expert I have used in the past (and have been
                  very happy with). He won't be available much until May (and I'm
                  already negotiating with him for his time then). I'm just finding
                  myself in more situations that both need more help and have sponsors
                  that may be willing to pay for it more than some other recent clients.

                  --- In agile-usability@yahoogroups.com, "Larry Constantine"
                  <lconstantine@f...> wrote:
                  > A perennial problem for all of us. That is why I only work on a
                  > time-and-travel basis.

                  And that's the way I pay people. But I have to do it within a budget
                  which, in my business, comes from negotiating a budget with my
                  clients. Of course this brings me back to the original question which
                  I think Peter answered as well as anyone (and should have been
                  intuitively obvious when I asked it, I guess)... I just need to get
                  better at estimating that part as I have gotten better at estimating
                  the software part.

                  By the way, by negotiating a budget with my clients and managing that
                  budget myself, I have a better chance of delivering a solid product.
                  When I find out "uh-oh, I should have had more usability help on this
                  one", I at least have some options other than going back to the client
                  with my hat in my hand begging for some more budget. The trick is
                  negotiating enough buffer in the budget that gives me those options
                  but doesn't inflate the project cost. That's the one I'm needing help
                  on...

                  By the way, for those of you who are full-time employees, you can
                  sometimes spend a little more time determining the scope of the
                  project, because you are getting paid as you are determining it. In
                  my business, very few people want to pay me to spend time determining
                  the scope. They call it the "cost of sales" (and so do I). I've
                  found that I've been forced to come up with a number quickly and get a
                  contract rolling so we can get the project rolling. The bad news is
                  we don't have as much time to think about the cost. The good news is
                  because we negotiate agile projects, we are negotiating scope all the
                  time based on more feedback than you can get just talking and pushing
                  paper around. (And, by the way, the negotiation I do is basically
                  what your managers do at "budget time". You just don't ever see it
                  from the front lines).

                  So, what are the questions a usability expert would ask up front to
                  estimate how much usability help a project "needs"?

                  Ken
                • Desilets, Alain
                  Actually, that s roughly the approach we use and it is the best I ve ever used for all involved. Fixed price, variable scope . That gives everyone who wants
                  Message 8 of 9 , Jan 20, 2006
                  • 0 Attachment
                    Actually, that's roughly the approach we use and it is the best I've
                    ever used for all involved. "Fixed price, variable scope". That gives
                    everyone who wants to know the budget a number and we virtually always
                    deliver on budget. That makes people happy and life easier for all
                    involved.

                    -- Alain:
                    Do you find there are problems of trust at sales time or during the
                    project? In other words, do you find your clients are afraid that you
                    will slack off (since payment is not tied to delivery of a final
                    product)?

                    The one time I negociated a fixed price, variable scope contract, I
                    found I could easily address that fear by explaining the transparency of
                    XP to the client, i.e. things like:

                    - single bull-pen area for the whole project team
                    - possibility for the client to have a representative in that bull-pen
                    area at all time (actually, this is not only possible, but highly
                    recommended)
                    - weekly or semi-weekly delivery of working prototypes

                    Do people use any other tricks besides that to address this issue of
                    trust?
                    ----
                  • kauerrolemodel
                    ... I don t find that there is a lack of trust based on what you have said. I ve got three kinds of clients... a) Those I ve done work for before. For those
                    Message 9 of 9 , Jan 25, 2006
                    • 0 Attachment
                      --- In agile-usability@yahoogroups.com, "Desilets, Alain"
                      <alain.desilets@n...> wrote:
                      >
                      >> Actually, that's roughly the approach we use and it is the best I've
                      >> ever used for all involved. "Fixed price, variable scope". That gives
                      >> everyone who wants to know the budget a number and we virtually always
                      >> deliver on budget. That makes people happy and life easier for all
                      >> involved.
                      >
                      > -- Alain:
                      > Do you find there are problems of trust at sales time or during the
                      > project? In other words, do you find your clients are afraid that you
                      > will slack off (since payment is not tied to delivery of a final
                      > product)?
                      >
                      > The one time I negociated a fixed price, variable scope contract, I
                      > found I could easily address that fear by explaining the transparency of
                      > XP to the client, i.e. things like:
                      >
                      > - single bull-pen area for the whole project team
                      > - possibility for the client to have a representative in that bull-pen
                      > area at all time (actually, this is not only possible, but highly
                      > recommended)
                      > - weekly or semi-weekly delivery of working prototypes

                      I don't find that there is a lack of trust based on what you have
                      said. I've got three kinds of clients...

                      a) Those I've done work for before. For those people, it is all about
                      what they have the budget to do. They seem to believe we will deliver
                      (based on past experience) and that the process will keep us on track
                      for at least some of the reasons you mentioned.

                      b) Those I haven't done work for before and who found me because they
                      were looking for someone to outsource to who would do agile
                      development or, more specifically XP. For these people, the big issue
                      is both budget and the idea of trying to figure out what they'll have
                      at the end of the budget... "Let's see, if I can get a budget for 6
                      iterations and we get something that roughly delivers the stories
                      we've got laid out, will I get in trouble for not having more stories
                      done for that budget or will I have trouble getting the next chunk of
                      budget we'll need to get the next stories people will want."

                      c) Those I haven't done work for before and who don't know anything
                      about XP or agile. These people are all over the map. They almost
                      never expect XP or the implications of it. So, we spend a lot of time
                      trying to convince them that this is either a good or the best way of
                      approaching there project and, if they buy the general idea, trying to
                      help them figure out how to get the people they need on the business
                      side to devote the time they'll need (and of course, the budget). Of
                      course, we are also competing against whatever other alternatives they
                      think they have (which runs the gamut from big players, small local
                      players, internal players, buy vs. build, etc.). One of their
                      objections MAY be what you mentioned, but I've found that people with
                      that objection usually have a lot of other issues with respect to
                      preparedness or accepting reality that are much bigger issues and this
                      objection is usually just a symptom of a bigger underlying problem.
                      I'd rather find the bigger underlying problem earlier rather than
                      later. Sometimes we can help them work through it, other times they
                      go away. Either way, it's OK.

                      Ken
                    Your message has been successfully submitted and would be delivered to recipients shortly.