4131Re: [agile-usability] Re: Online Usability Tests
- Mar 25, 2008Todd Zaki Warfel wrote:
>We took a running site and instrumented things so that we could see the
> Quick question for you, how did you find this w/the on-line study?
> What did you do to measure/find something like this using an on-line
> study? The reason I ask is that it would be nice for others to know
> the technique so they could use it to look for this when they are
> testing (reason I included the Craigslist item above).
raw submissions from every sign-up attempt, successful or failed. Then
we let it run for a while and sifted through the failures looking for
patterns. Some of the issues discovered involved the three-way
interaction of the user, our code, and external credit-card processors,
who have yet different ideas of what constitutes a valid address.
That definitely doesn't catch everything, as the user has to get as far
as clicking the submit button, which is why next time I try this I'd
like to do a little AJAX instrumentation, uploading a record of
keypresses, pauses, mouse movements, and the like. That still won't
catch everything, of course. Like Col. Prescott, I like to see the
whites of their eyes. And as you say, it can still leave the "why" a puzzle.
> The point is that if you start to see it in a few people in a 12I'm certainly not denying that there are a lot of great issues that will
> person study, you're going to see it in hundreds or thousands with 10,000.
show up in 12-person study, and that those issues will show up in larger
samples. I'm concerned about the opposite case. If I see it in 100 of
10,000 (1%), then I may not see it in 1 of 12 people (8%) , and I'm even
less likely to see it in 1 of 5 (20%).
I suspect there's also a question of relative expertise. I think you
mentioned you have done hundreds of these studies, and clearly you have
spent a lot of time thinking about the usability of interfaces. The kind
and volume of issues that you can surface in a 5-person study are
probably much superior to what a self-taught designer can do in between
extract so much value from them.
- << Previous post in topic Next post in topic >>