4110Re: [agile-usability] Re: Online Usability Tests
- Mar 20, 2008Todd Zaki Warfel wrote:
> Personally, if I had my choice of observing 12 participants in personI think we're talking about different kinds of quantitative methods.
> or 100 people run through an automated remote process, I'd take the 12
> in person every single time. In the 15+ years I've been doing this
> type of research, I've yet to find a pattern identified by survey and
> remote methods w/100 people that we weren't able to identify with 12
> in person.
> The main benefit we get from more quantitative methods is to satisfy
> the marketing research people who only believe in quantitative methods.
The choice I typically see isn't 12 local vs 100 remote. It's 4 versus
1,000. Or 10,000. Or 100,000. And actual users doing actual tasks versus
recruited users doing requested tasks.
The simplest version of this is just log analysis combined with some
basic instrumentation. For example, at one client I looked into failed
orders. Looking at the data, circa 10% of orders were failing at the
credit card processing stage. Some of them were legitimate failures, but
I suspected that not all of them were. So we logged every bit of
information in every order attempt.
It turned out there were a number of minor user interface issues, none
affecting more than 2% of order attempts, which is well below the power
of a 12-user study to resolve. And several related to different
international styles of entering addresses, which we couldn't have
solved with a local user study anyhow. The cost-benefit ratio was also
much better; from inception to recommendations, it was under two
person-days of work.
I'm also fond of tracking metrics. In-person user testing is good for
things you know to ask about, but you need live site metrics to catch
things that you didn't even know have changed. One client has a very
data-driven site, and manually testing all the key pages with every data
update is impossible. They track dozens of metrics, and significant
deviations in key numbers get people paged. Good metrics also lets you
catch surprises with what you thought were minor changes to the site.
And once you have some metrics, A/B testing can really pay off. Suppose
you want to know which of three new landing page versions increases
signups by 5%. You can't do that with a 12-person user test. But you can
put each version up in parallel for a week with users randomly assigned
to each, and get thousands or tens of thousands of data points.
For any of these approaches, the data can lead to additional questions.
Some of those are best answered with more data, but many can be more
effectively approached with traditional user testing.
- << Previous post in topic Next post in topic >>