Loading ...
Sorry, an error occurred while loading the content.

RE: [agile-usability] test-driven iterative user centered design

Expand Messages
  • Jared M. Spool
    ... In my opinion, the reason this is unclear is that it is *very* dependent on the practitioners involved. Previous experience, exposure to research data,
    Message 1 of 9 , Jan 9, 2006
    • 0 Attachment
      At 02:58 PM 1/9/2006, Desilets, Alain wrote:
      >I don't have a good sense of the relative efficiency of upfront inspection
      >(ex: Design Document Inspection) vs upfront testing (ex: Testing Paper
      >Prototype) techniques for finding usability issues. But I suspect that
      >both techniques are good, necessary and within the same range as far as
      >efficiency goes.

      In my opinion, the reason this is unclear is that it is *very* dependent on
      the practitioners involved. Previous experience, exposure to research data,
      innate skill, and a variety of other factors will affect both the results
      of an inspection and the analysis of testing.

      Something the usability/user experience community isn't addressing is how
      much this is a craft, where individual prowess trumps collective wisdom and
      process. We keep wanting to pretend what we do is an engineering
      discipline, but unfortunately, it's a craft where the skill of the
      craftsperson dictates success. This is the big elephant in the room that
      no-one is talking about.

      In research we conducted a while back, we traced where in the development
      process usability problems were introduced into the design. Our major
      finding was that they almost always were introduced at the same point:
      someone on the design team needed to make a decision for which they didn't
      have all the information necessary.

      From this finding, I took away that the more important end result from any
      type of usability activity is an informed decision.

      Design Document Inspection can work if the inspection team arrives at
      informed decisions in the process. That's going to depend on the existing
      knowledge, skills, and deductive insights of the team. Paper Prototype
      Testing arrives at informed decisions in a different manner. It's going to
      depend on the test team's skills at creating tests, recruiting users, and
      making proper inferences from the observations. Both techniques are as
      likely to steer a team wrong as they are to steer a team right.

      In other words, a poorly-executed usability activity will result in poor
      decisions. There are no guarantees.

      I think if we want to make usability into more of an engineering practice,
      we need to control some of these variables a lot better.

      Just my thoughts late this evening...

      Jared


      Jared M. Spool, Founding Principal, User Interface Engineering
      4 Lookout Lane, Unit 4d, Middleton, MA 01949
      978 777-9123 jspool@... http://www.uie.com
      Blog: http://www.uie.com/brainsparks
    • Jon Meads
      Jared Spool writes: In research we conducted a while back, we traced where in the development process usability problems were introduced into the design. Our
      Message 2 of 9 , Jan 9, 2006
      • 0 Attachment
        Jared Spool writes:
        In research we conducted a while back, we traced where in the development process usability problems were introduced into the design. Our major finding was that they almost always were introduced at the same point: someone on the design team needed to make a decision for which they didn't have all the information necessary.
        So true. The problem is that there are variables and things that we just won't know until we test. As unfortunately stated, "There are things that we don't know that we don't know." This is not justification for skipping user research (Sun Tzu had the right idea about learning the lay of the land and the nature of the enemy). It is simply recognition of an empirical aspect to UI design. While it is possible that a UI designer might create a great UI design without user research for one project, there is no guarantee that they'll be able to do so on the next project.
         
        Cheers,
        jon
         

                   Jon Meads
                   Usability Architects, Inc.
                   PO Box 3222
                   Kirkland, WA 98083-3222
         
            Voice: 425-827-9296
             Cell: 206-409-7548
              Fax: 425-827-6692
            Email: jon@...
         
                  Designing the User Experience
               http://www.usability-architects.com/


      • Larry Constantine
        Warning: Short reaction to a long post. ... User Goals are to ... this thread, and ... and my limited ... they hold up. ... might as well ... And I say amen.
        Message 3 of 9 , Jan 10, 2006
        • 0 Attachment
          Warning: Short reaction to a long post.

          Alain wrote:
          > Great insight Jeff! I never thought of it that way, but you are right,
          User Goals are to
          > the UI what Tests are to code....
          > I spent a good part of this weekend thinking about what has been said in
          this thread, and
          > trying to make sense of it based on my extensive programming experience,
          and my limited
          > experience in U*....
          > Let me post some claims which I welcome you to poke shots at to see if
          they hold up.
          > ...
          > Well, that's it. If you have made it this far in reading this post, you
          might as well
          > react to it ;-).

          And I say amen. Very thorough analysis and absolutely on the mark, Alain.

          --Larry Constantine, IDSA
        • Tobias Mayer
          ... Thanks for raising that. It inspired me to talk about that elephant. My thoughts are not usability-specific, so not repeated here. If you are interested
          Message 4 of 9 , Jan 10, 2006
          • 0 Attachment
            Jared wrote:
            > Something the usability/user experience community isn't addressing is how much this is a craft, where individual prowess trumps collective wisdom and process. We keep wanting to pretend what we do is an engineering discipline, but unfortunately, it's a craft where the skill of the craftsperson dictates success. This is the big elephant in the room that no-one is talking about.
             
            Thanks for raising that.  It inspired me to talk about that elephant.  My thoughts are not usability-specific, so not repeated here.  If you are interested see: http://agilethinking.net/artisans.html
             
            Regards,
            Tobias
            (not an engineer)


            "Jared M. Spool" <jspool@...> wrote:
            At 02:58 PM 1/9/2006, Desilets, Alain wrote:
            >I don't have a good sense of the relative efficiency of upfront inspection
            >(ex: Design Document Inspection) vs upfront testing (ex: Testing Paper
            >Prototype) techniques for finding usability issues. But I suspect that
            >both techniques are good, necessary and within the same range as far as
            >efficiency goes.

            In my opinion, the reason this is unclear is that it is *very* dependent on
            the practitioners involved. Previous experience, exposure to research data,
            innate skill, and a variety of other factors will affect both the results
            of an inspection and the analysis of testing.

            Something the usability/user experience community isn't addressing is how
            much this is a craft, where individual prowess trumps collective wisdom and
            process. We keep wanting to pretend what we do is an engineering
            discipline, but unfortunately, it's a craft where the skill of the
            craftsperson dictates success. This is the big elephant in the room that
            no-one is talking about.

            In research we conducted a while back, we traced where in the development
            process usability problems were introduced into the design. Our major
            finding was that they almost always were introduced at the same point:
            someone on the design team needed to make a decision for which they didn't
            have all the information necessary.

            From this finding, I took away that the more important end result from any
            type of usability activity is an informed decision.

            Design Document Inspection can work if the inspection team arrives at
            informed decisions in the process. That's going to depend on the existing
            knowledge, skills, and deductive insights of the team. Paper Prototype
            Testing arrives at informed decisions in a different manner. It's going to
            depend on the test team's skills at creating tests, recruiting users, and
            making proper inferences from the observations. Both techniques are as
            likely to steer a team wrong as they are to steer a team right.

            In other words, a poorly-executed usability activity will result in poor
            decisions. There are no guarantees.

            I think if we want to make usability into more of an engineering practice,
            we need to control some of these variables a lot better.

            Just my thoughts late this evening...

            Jared


            Jared M. Spool, Founding Principal, User Interface Engineering
            4 Lookout Lane, Unit 4d, Middleton, MA 01949
            978 777-9123   jspool@...  http://www.uie.com
            Blog: http://www.uie.com/brainsparks



          • Desilets, Alain
            Unfortunately, in my world, it s VERY difficult to know ahead of time what I m going to face until we re into the project awhile. And I have to quote a price
            Message 5 of 9 , Jan 19, 2006
            • 0 Attachment
              Unfortunately, in my world, it's VERY difficult to know ahead of time
              what I'm going to face until we're into the project awhile. And I have
              to quote a price for a project before I have enough data to know whether
              I can get away with little outside usability help or need a lot (or need
              a new project sponsor).

              So, I have two questions:

              1. How do you work around this problem realistically? (I'd prefer input
              from people who have tried things and been successful... I've gotten
              pretty good at assessing new clients/projects with very little
              introduction... I'm not talking about hypothetical situations, I'm
              talking real ones).

              -- Alain:
              I guess this is a problem that ALL software consultants face, not just
              agile ones right?

              I asked that question once to Joshua Kerievsky (author of "Refactoring
              to Patterns") and his answer was: "Don't do fixed cost contracts.
              Instead, negociate contracts based on access to resources and control
              over them."

              In other words, instead of selling deliverabls to the customer, you
              should sell something like: X hours of such and such team per month, and
              the privilege of steering them in such and such way.

              Makes a lot of sense to me. I know that's how a lot of professionals
              work. When you hire a lawyer for example, you agree on an hourly rate,
              not on the final product (i.e. a decision of the judge in your favour).

              The problem with this approach is of course that most clients want to
              cling to the illusion of control provided by a cost estimate. Is there
              anyone out there that actually tried this approach and was successful?

              I have only once negociated a contract like that and the client actually
              preferred this kind of arrangement to a fixed cost contract. He was a
              small startup entrepreneur with a vision for a new and highly innovative
              product. Because the product was so innovative, he had no way of knowing
              what the market was exactly, nor which parts of his vision were
              realisable or not, nor which particular variant of his vision would
              catch on. So to him, the idea of being able to pour his limited capital
              one month at a time, and pull the plug whenever he wanted was very
              attractive.

              Note that this doesn't really count as "real-life" experience because
              the contract was never finalised. The reason was not that he did not
              like the format, but rather that he was unable to raise sufficient
              venture capital to start the company (this was 2 years after the
              internet bubble burst, so VC was hard to come by at that time).
              ----
            • kauerrolemodel
              ... Actually, that s roughly the approach we use and it is the best I ve ever used for all involved. Fixed price, variable scope . That gives everyone who
              Message 6 of 9 , Jan 19, 2006
              • 0 Attachment
                --- In agile-usability@yahoogroups.com, "Desilets, Alain"
                <alain.desilets@n...> wrote:
                > In other words, instead of selling deliverabls to the customer, you
                > should sell something like: X hours of such and such team per month, and
                > the privilege of steering them in such and such way.
                > [snip]
                > The problem with this approach is of course that most clients want to
                > cling to the illusion of control provided by a cost estimate. Is there
                > anyone out there that actually tried this approach and was successful?

                Actually, that's roughly the approach we use and it is the best I've
                ever used for all involved. "Fixed price, variable scope". That
                gives everyone who wants to know the budget a number and we virtually
                always deliver on budget. That makes people happy and life easier for
                all involved.

                --- In agile-usability@yahoogroups.com, "Peter Boersma" <peter@p...>
                wrote:
                > Ken wrote:
                > > Unfortunately, in my world, it's VERY difficult to know ahead of time
                > > what I'm going to face until we're into the project awhile. And I
                have
                > > to quote a price for a project before I have enough data to know
                > > whether I can get away with little outside usability help or need
                a lot
                > > (or need a new project sponsor).
                >
                > Isn't this true for both the usability work and the software development
                > costs? Or do you need an expert at estimating the usability
                complexities of
                > the project, just like you estimate the complexity of the software
                required?

                I think the answer to that is yes.

                I just helped scope out another project today and I asked a few
                questions that suggested strongly we would need some more usability
                help. I'll be putting that in the quote. Now, we just need to land
                the contract and get someone in to help. I think the first one is
                high probability. Anyone out there care to help on the latter?

                I do have a usability expert I have used in the past (and have been
                very happy with). He won't be available much until May (and I'm
                already negotiating with him for his time then). I'm just finding
                myself in more situations that both need more help and have sponsors
                that may be willing to pay for it more than some other recent clients.

                --- In agile-usability@yahoogroups.com, "Larry Constantine"
                <lconstantine@f...> wrote:
                > A perennial problem for all of us. That is why I only work on a
                > time-and-travel basis.

                And that's the way I pay people. But I have to do it within a budget
                which, in my business, comes from negotiating a budget with my
                clients. Of course this brings me back to the original question which
                I think Peter answered as well as anyone (and should have been
                intuitively obvious when I asked it, I guess)... I just need to get
                better at estimating that part as I have gotten better at estimating
                the software part.

                By the way, by negotiating a budget with my clients and managing that
                budget myself, I have a better chance of delivering a solid product.
                When I find out "uh-oh, I should have had more usability help on this
                one", I at least have some options other than going back to the client
                with my hat in my hand begging for some more budget. The trick is
                negotiating enough buffer in the budget that gives me those options
                but doesn't inflate the project cost. That's the one I'm needing help
                on...

                By the way, for those of you who are full-time employees, you can
                sometimes spend a little more time determining the scope of the
                project, because you are getting paid as you are determining it. In
                my business, very few people want to pay me to spend time determining
                the scope. They call it the "cost of sales" (and so do I). I've
                found that I've been forced to come up with a number quickly and get a
                contract rolling so we can get the project rolling. The bad news is
                we don't have as much time to think about the cost. The good news is
                because we negotiate agile projects, we are negotiating scope all the
                time based on more feedback than you can get just talking and pushing
                paper around. (And, by the way, the negotiation I do is basically
                what your managers do at "budget time". You just don't ever see it
                from the front lines).

                So, what are the questions a usability expert would ask up front to
                estimate how much usability help a project "needs"?

                Ken
              • Desilets, Alain
                Actually, that s roughly the approach we use and it is the best I ve ever used for all involved. Fixed price, variable scope . That gives everyone who wants
                Message 7 of 9 , Jan 20, 2006
                • 0 Attachment
                  Actually, that's roughly the approach we use and it is the best I've
                  ever used for all involved. "Fixed price, variable scope". That gives
                  everyone who wants to know the budget a number and we virtually always
                  deliver on budget. That makes people happy and life easier for all
                  involved.

                  -- Alain:
                  Do you find there are problems of trust at sales time or during the
                  project? In other words, do you find your clients are afraid that you
                  will slack off (since payment is not tied to delivery of a final
                  product)?

                  The one time I negociated a fixed price, variable scope contract, I
                  found I could easily address that fear by explaining the transparency of
                  XP to the client, i.e. things like:

                  - single bull-pen area for the whole project team
                  - possibility for the client to have a representative in that bull-pen
                  area at all time (actually, this is not only possible, but highly
                  recommended)
                  - weekly or semi-weekly delivery of working prototypes

                  Do people use any other tricks besides that to address this issue of
                  trust?
                  ----
                • kauerrolemodel
                  ... I don t find that there is a lack of trust based on what you have said. I ve got three kinds of clients... a) Those I ve done work for before. For those
                  Message 8 of 9 , Jan 25, 2006
                  • 0 Attachment
                    --- In agile-usability@yahoogroups.com, "Desilets, Alain"
                    <alain.desilets@n...> wrote:
                    >
                    >> Actually, that's roughly the approach we use and it is the best I've
                    >> ever used for all involved. "Fixed price, variable scope". That gives
                    >> everyone who wants to know the budget a number and we virtually always
                    >> deliver on budget. That makes people happy and life easier for all
                    >> involved.
                    >
                    > -- Alain:
                    > Do you find there are problems of trust at sales time or during the
                    > project? In other words, do you find your clients are afraid that you
                    > will slack off (since payment is not tied to delivery of a final
                    > product)?
                    >
                    > The one time I negociated a fixed price, variable scope contract, I
                    > found I could easily address that fear by explaining the transparency of
                    > XP to the client, i.e. things like:
                    >
                    > - single bull-pen area for the whole project team
                    > - possibility for the client to have a representative in that bull-pen
                    > area at all time (actually, this is not only possible, but highly
                    > recommended)
                    > - weekly or semi-weekly delivery of working prototypes

                    I don't find that there is a lack of trust based on what you have
                    said. I've got three kinds of clients...

                    a) Those I've done work for before. For those people, it is all about
                    what they have the budget to do. They seem to believe we will deliver
                    (based on past experience) and that the process will keep us on track
                    for at least some of the reasons you mentioned.

                    b) Those I haven't done work for before and who found me because they
                    were looking for someone to outsource to who would do agile
                    development or, more specifically XP. For these people, the big issue
                    is both budget and the idea of trying to figure out what they'll have
                    at the end of the budget... "Let's see, if I can get a budget for 6
                    iterations and we get something that roughly delivers the stories
                    we've got laid out, will I get in trouble for not having more stories
                    done for that budget or will I have trouble getting the next chunk of
                    budget we'll need to get the next stories people will want."

                    c) Those I haven't done work for before and who don't know anything
                    about XP or agile. These people are all over the map. They almost
                    never expect XP or the implications of it. So, we spend a lot of time
                    trying to convince them that this is either a good or the best way of
                    approaching there project and, if they buy the general idea, trying to
                    help them figure out how to get the people they need on the business
                    side to devote the time they'll need (and of course, the budget). Of
                    course, we are also competing against whatever other alternatives they
                    think they have (which runs the gamut from big players, small local
                    players, internal players, buy vs. build, etc.). One of their
                    objections MAY be what you mentioned, but I've found that people with
                    that objection usually have a lot of other issues with respect to
                    preparedness or accepting reality that are much bigger issues and this
                    objection is usually just a symptom of a bigger underlying problem.
                    I'd rather find the bigger underlying problem earlier rather than
                    later. Sometimes we can help them work through it, other times they
                    go away. Either way, it's OK.

                    Ken
                  Your message has been successfully submitted and would be delivered to recipients shortly.