Loading ...
Sorry, an error occurred while loading the content.

4123Re: [agile-usability] Re: Online Usability Tests

Expand Messages
  • Todd Zaki Warfel
    Mar 25, 2008
    • 0 Attachment

      On Mar 25, 2008, at 4:10 AM, William Pietri wrote:
      One substantial cause of failure was international addresses. The cost of a multi-continent usability study surely makes sense for some people[...]

      This comes down to a recruiting issue. You don't need to do a multi-continent study to find this—you can use craigslist to recruit international people to solve this issue. During the study, just have them use their address from home, or the country they came from before they got here to see this. 

      Now, if this is something you don't think of to test in the study, then that's another story. 

      Quick question for you, how did you find this w/the on-line study? What did you do to measure/find something like this using an on-line study? The reason I ask is that it would be nice for others to know the technique so they could use it to look for this when they are testing (reason I included the Craigslist item above).

      Well, I said that because of a little math. Perhaps I'm doing it wrong, but if only 1-2% of people have some issue, the odds of finding that particular issue in a 12-person test don't seem particularly high. And if only 1 of 12 has a problem, it would be hard to say whether it's a pattern or a fluke. Whereas with 10,000 data points, you'll be able to do solid ROI calculations so that you know which fixes are worth the effort.

      Aw, see that's the flaw in the equation. First, what you're looking for is a pattern. Second, what you're looking for is something that might be a smaller issue by sheer number, but you know it's significant. For example, we recently had an issue I sited earlier with someone getting hung up at registration due to a space in their name. That was one person, but it's something you know is a show stopper. In short, it depends on what the issue is. Some of "knowing this" only comes with time and experience. Some of it is a no brainer. Some of these items are easier to see with 10,000 data points. Most of them are going to be something you'll see a pattern with 5-6 people and confirmed with 8-12. 

      The point is that if you start to see it in a few people in a 12 person study, you're going to see it in hundreds or thousands with 10,000. I think the issue is that people get hung up on sheer numbers instead of percentages. We rarely have issues that aren't found in 70% or more of participants in an 8-12 person study. Either way, we make sure that when we report we include:
      1. Percent of people reporting (e.g. 7/10 experienced this)
      2. Percentage of people it was an issue with (e.g. 8/10)
      3. If it's a small number, then the number of people and why we think it might require further investigation.

      For example if we did a 10 person study:
      Issue 70% of 100% reporting would be 10/10
      Issue 50% of 80% reporting 4/8

      This is particularly important to be accurate about the reporting. Simply stating 4 of our participants isn't accurate—you need to indicate that 8 came across the issue and of those 8, 4 or 50%, had trouble with it.

      I find it weird to keep saying this, but I really like in-person studies. I think they are the bees knees. Honest. I do them every chance I get.

      Obviously, I agree, but I also think that remote studies are extremely beneficial. One of the biggest benefits is that they're using their own machine and you get to view how they access things (e.g. bookmarks, pop-up blockers). Additionally, it enables you to do research w/geographically dispersed audiences. 

      For example, we did an ethnographic-based study last year for a client who had employees across the world. We did 48 interviews. We couldn't afford to fly there (time/budget), so we used remote screen sharing and phones to do the research. We had some very interesting findings and remote studies were the only way we could have done this.


      Todd Zaki Warfel
      President, Design Researcher
      Messagefirst | Designing Information. Beautifully.
      Contact Info
      Voice: (215) 825-7423
      Email: todd@...
      In theory, theory and practice are the same.
      In practice, they are not.

    • Show all 29 messages in this topic