4111Re: [agile-usability] Re: Online Usability Tests
- Mar 20, 2008On Mar 20, 2008, at 1:17 PM, William Pietri wrote:Well, 4 isn't enough. I wouldn't recommend any less than 5 and typically 8-12 is best, otherwise you don't have enough to start seeing significance in patterns. That's going to be one reason that 1000 is going to yield better results with web tracking—you don't have enough in your qualitative study.A couple of things here:1. I would suspect that the "minor user interface issues" would have been easily corrected with simply having a good informed interaction designer or usability specialist assess the interface.2. Did you do a 12 user study on this interface? I'll bet that if you did, you would have found the same issues—I've done this literally hundreds of times. If you didn't, how would you know that it's beyond what you can find from a 12 person study? We use web metrics to help identify key abandonment areas, then in-person field studies to find out the why. For example, we had a client who had a significant abandonment in one of their cart screens, but didn't know exactly which fields. They could have spent time coding up the fields w/JS to track every single one to figure out. Instead we did a quick study w/12 people and found out that it was a combination of two fields that was causing the problem on that screen and exactly why they were an issue. Problem fixed.Just a different approach. And yes, we used a mix of qual and quan—something we do quite often.Not sure I agree with that. Might be the way you're (or the person doing testing) is conducting the tests. Our testing format utilizes an open format with discovery method. We have some predefined tasks based on goals that we know/hope people are trying to accomplish with the site. This info comes from combined sources (e.g. metrics, sales, marketing, customer service, customer feedback). However, that's not all of it—we always include open ended discovery time to watch for things we don't expect, anticipate, our couldn't plan for—unexpected tasks. We've done this in pretty much every test in the last couple of years and every time find a number of new features, functions, and potential lines of revenue for our client.True. Our method selection is goal driven. What's your goal? That drives your method. Just to provide the counter point to that, the downside of A/B the way you're suggesting is that while it will tell you that one model increased signup by 5%, it won't tell you why. A quick 12 person study will tell you why and give you guidance on which one would probably increase sign-up. You then take that info and challenge/validate it with a quantitative study like you suggest. Or the reverse, take your A/B and do a supplemental 12 person study to find out why.Answering the why will give you far more from a design insight perspective than just seeing what happened.
- << Previous post in topic Next post in topic >>