Loading ...
Sorry, an error occurred while loading the content.

4119Re: [agile-usability] Re: Online Usability Tests

Expand Messages
  • William Pietri
    Mar 25, 2008
    • 0 Attachment
      Hi, Todd. I think we agree entirely that people should do both
      quantitative and qualitative research, and both have their strengths. A
      few responses to minor points.

      Todd Zaki Warfel wrote:
      > Well, 4 isn't enough. [...]
      > 1. I would suspect that the "minor user interface issues" would have
      > been easily corrected with simply having a good informed interaction
      > designer or usability specialist assess the interface.
      > 2. Did you do a 12 user study on this interface?

      I work mainly with startups and small companies. Perhaps you can do a
      12-user study more effectively than they can, but it's well beyond
      something a lot of small shops can afford to do on a regular basis.
      Doing 4-6 people once a month is more their speed.

      It may be that their designers don't meet the level you consider good.
      However, they are generally the best the company has been able to find.
      Whatever practices I recommend have to work in that context, which
      frequently includes people wearing multiple hats.

      > I'll bet that if you did, you would have found the same issues—I've
      > done this literally hundreds of times.

      I'm sure you win that bet a lot, but in this case you would have lost.

      One substantial cause of failure was international addresses. The cost
      of a multi-continent usability study surely makes sense for some people,
      but not for the sums involved in this case. At the cost of a couple of
      days time around the office, though, it paid off nicely.

      > If you didn't, how would you know that it's beyond what you can find
      > from a 12 person study?

      Well, I said that because of a little math. Perhaps I'm doing it wrong,
      but if only 1-2% of people have some issue, the odds of finding that
      particular issue in a 12-person test don't seem particularly high. And
      if only 1 of 12 has a problem, it would be hard to say whether it's a
      pattern or a fluke. Whereas with 10,000 data points, you'll be able to
      do solid ROI calculations so that you know which fixes are worth the effort.

      > We use web metrics to help identify key abandonment areas, then
      > in-person field studies to find out the why. For example [...]

      I find it weird to keep saying this, but I really like in-person
      studies. I think they are the bees knees. Honest. I do them every chance
      I get.

      The only reason I got involved in this thread was to mention how on-line
      testing indeed can capture some things you said it couldn't, and to
      mention how some people I know are doing it. Nobody should feel obliged
      to do it that way, and they certainly shouldn't stop doing in-person

    • Show all 29 messages in this topic