Loading ...
Sorry, an error occurred while loading the content.

Beta Testing For Usability?

Expand Messages
  • doug.gorman@rocketmail.com
    Hello, everyone! We re about to roll out a few applications to a beta test group and I was wondering what techniques you ve all used to get useful feedback
    Message 1 of 5 , Nov 2, 2009
    • 0 Attachment
      Hello, everyone!

      We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.

      My current plan is:

      1. Interview testers before they get the beta release. The purposes of this interview are to level set about what the testers expect the application will do for them, what they are currently using to fill the gap (internet sites, etc), and what they hope the beta application does not do.

      2. Weekly polling to see how well or not the beta is living up to expectations for each tester.

      3. Weekly ethnographical studies with select testers (preferrably those who seem frustrated with the application).

      4. Bi-weekly publishing of test results with recommendations for how to proceed on issues found.

      Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?
    • Kaleb Pederson
      On Mon, Nov 2, 2009 at 12:58 PM, doug.gorman@rocketmail.com ... Nice question. ... Depending on the type of application, a true real-world indicator of the
      Message 2 of 5 , Nov 2, 2009
      • 0 Attachment
        On Mon, Nov 2, 2009 at 12:58 PM, doug.gorman@...
        <doug.gorman@...> wrote:
        > Hello, everyone!
        >
        > We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.

        Nice question.

        > Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?

        Depending on the type of application, a true real-world indicator of
        the effectiveness of various UI changes would be to A/B test them
        (http://en.wikipedia.org/wiki/A/B_testing). Assuming you're working
        with a web application, you should be able to send a certain
        percentage of users (possibly pre-screened and filtered) through the
        beta page(s) to determine the overall affect on the target metrics. If
        a single change positively effects the target metrics, like conversion
        or lead generation, then it's probably a good change and likely makes
        the application more usable.

        --
        Kaleb Pederson

        Blog - http://kalebpederson.com
        Twitter - http://twitter.com/kalebpederson
      • pmiinalainen
        Hi, I ve seen that a free form forum can also help to find solutions. For example, someone is having trouble with new feature and asks for help. That often
        Message 3 of 5 , Nov 3, 2009
        • 0 Attachment
          Hi,

          I've seen that a free form forum can also help to find solutions.

          For example, someone is having trouble with new feature and asks for help. That often leads to a discussion with multiple solutions to problem and brings several opinions and related issues to table at once.
          Other than that, your list seems to be quite comprehensive.

          Petteri

          --- In agile-usability@yahoogroups.com, "doug.gorman@..." <doug.gorman@...> wrote:
          >
          > Hello, everyone!
          >
          > We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.
          >
          > My current plan is:
          >
          > 1. Interview testers before they get the beta release. The purposes of this interview are to level set about what the testers expect the application will do for them, what they are currently using to fill the gap (internet sites, etc), and what they hope the beta application does not do.
          >
          > 2. Weekly polling to see how well or not the beta is living up to expectations for each tester.
          >
          > 3. Weekly ethnographical studies with select testers (preferrably those who seem frustrated with the application).
          >
          > 4. Bi-weekly publishing of test results with recommendations for how to proceed on issues found.
          >
          > Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?
          >
        • Lyn Bain
          Don t forget the very basic Feedback form or email address so that people can spontaneously send feedback as it occurs to them. Make sure you have a quick
          Message 4 of 5 , Nov 3, 2009
          • 0 Attachment
            Don't forget the very basic Feedback form or email address so that
            people can spontaneously send feedback as it occurs to them. Make sure
            you have a quick response to them to keep them motivated - have someone
            check for feedback frequently and respond.

            >>Have you used other approaches or techniques during betas to get
            great "real world" feedback on usability? What techniques had the
            highest yield?

            Lyn Bain
            Chili Interactive
          • doug.gorman@rocketmail.com
            Thanks, Kaleb. I ve recently found some great articles on A/B testing and it s use in product and usability development. This was an excellent one:
            Message 5 of 5 , Nov 5, 2009
            • 0 Attachment
              Thanks, Kaleb. I've recently found some great articles on A/B testing and it's use in product and usability development.

              This was an excellent one: http://blog.mixpanel.com/ab-testing-to-increase-user-engagement

              I definitely think we'll try this approach.

              --- In agile-usability@yahoogroups.com, Kaleb Pederson <kaleb.pederson@...> wrote:
              >
              > On Mon, Nov 2, 2009 at 12:58 PM, doug.gorman@...
              > <doug.gorman@...> wrote:
              > > Hello, everyone!
              > >
              > > We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.
              >
              > Nice question.
              >
              > > Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?
              >
              > Depending on the type of application, a true real-world indicator of
              > the effectiveness of various UI changes would be to A/B test them
              > (http://en.wikipedia.org/wiki/A/B_testing). Assuming you're working
              > with a web application, you should be able to send a certain
              > percentage of users (possibly pre-screened and filtered) through the
              > beta page(s) to determine the overall affect on the target metrics. If
              > a single change positively effects the target metrics, like conversion
              > or lead generation, then it's probably a good change and likely makes
              > the application more usable.
              >
              > --
              > Kaleb Pederson
              >
              > Blog - http://kalebpederson.com
              > Twitter - http://twitter.com/kalebpederson
              >
            Your message has been successfully submitted and would be delivered to recipients shortly.