Loading ...
Sorry, an error occurred while loading the content.

4131Re: [agile-usability] Re: Online Usability Tests

Expand Messages
  • William Pietri
    Mar 25, 2008
      Todd Zaki Warfel wrote:
      > Quick question for you, how did you find this w/the on-line study?
      > What did you do to measure/find something like this using an on-line
      > study? The reason I ask is that it would be nice for others to know
      > the technique so they could use it to look for this when they are
      > testing (reason I included the Craigslist item above).

      We took a running site and instrumented things so that we could see the
      raw submissions from every sign-up attempt, successful or failed. Then
      we let it run for a while and sifted through the failures looking for
      patterns. Some of the issues discovered involved the three-way
      interaction of the user, our code, and external credit-card processors,
      who have yet different ideas of what constitutes a valid address.

      That definitely doesn't catch everything, as the user has to get as far
      as clicking the submit button, which is why next time I try this I'd
      like to do a little AJAX instrumentation, uploading a record of
      keypresses, pauses, mouse movements, and the like. That still won't
      catch everything, of course. Like Col. Prescott, I like to see the
      whites of their eyes. And as you say, it can still leave the "why" a puzzle.

      > The point is that if you start to see it in a few people in a 12
      > person study, you're going to see it in hundreds or thousands with 10,000.

      I'm certainly not denying that there are a lot of great issues that will
      show up in 12-person study, and that those issues will show up in larger
      samples. I'm concerned about the opposite case. If I see it in 100 of
      10,000 (1%), then I may not see it in 1 of 12 people (8%) , and I'm even
      less likely to see it in 1 of 5 (20%).

      I suspect there's also a question of relative expertise. I think you
      mentioned you have done hundreds of these studies, and clearly you have
      spent a lot of time thinking about the usability of interfaces. The kind
      and volume of issues that you can surface in a 5-person study are
      probably much superior to what a self-taught designer can do in between
      cranking out HTML and tweaking the JavaScript. That may explain how you
      extract so much value from them.

    • Show all 29 messages in this topic