Loading ...
Sorry, an error occurred while loading the content.

Re: [agile-usability] Best methods for user feedback before release

Expand Messages
  • Nick Gassman
    ... One client does amazing things with that. Last I heard, they had something like 80% of their traffic (in total about twice ba.com s traffic) on the main
    Message 1 of 16 , Jul 6, 2008
    • 0 Attachment
      On Sun, 06 Jul 2008 15:16:23 -0700, William wrote:

      >For post-release feedback, along with the other approaches you mention, you could consider pushing toward extensive metrics and a strong A/B test environment.
      One client does amazing things with that. Last I heard, they had
      something like 80% of their traffic (in total about twice ba.com's
      traffic) on the main site, and the rest in five test environments. It
      lets them do extensive and precise fine-tuning of their app.
      >
      Yes, we've done A/B testing, and it's been very useful. We're also
      looking at multivariate testing software, which will allow us to
      manipulate several variables on the same page, and statistically
      identify which ones are having an effect. As ever, it's about being
      aware of which tools are most appropriate.
      >
      >
      >From a business perspective, it seems like you're in an interesting position. Normally, a dominant company in well-developed industry
      is inclined to be pretty conservative. Why rock the boat? But given
      the turmoil in aviation, I imagine the web site is now looking more
      like an opportunity for relatively low-cost innovation and market
      differentiation.


      Hmm, I wouldn't describe us as 'dominant' (it all depends on the
      context), although we are a big player. We have a seriously large code
      base backing up our website, with a seriously large ongoing programme
      of development. The airline industry has some big challenges to face,
      and we need to find a way to reduce the cost and time to market of new
      developments. Even if the market conditions were better, you always
      need to look for better ways of doing things.

      * Nick Gassman - Usability and Standards Manager - http://ba.com *
    • Larry Constantine
      Nick, On user feedback techniques, two quick comments: one plus, one minus. Scratch focus groups. Whether in waterfall or agile, focus groups have a long
      Message 2 of 16 , Jul 7, 2008
      • 0 Attachment

        Nick,

         

        On user feedback techniques, two quick comments: one plus, one minus.

         

        Scratch focus groups. Whether in waterfall or agile, focus groups have a long record of yielding user feedback that is too often spurious and unusable.

         

        Consider collaborative usability inspections. They are fast, involve both users and the development team in a highly structured walkthrough, and yield rich results.

         

        --Larry Constantine, IDSA, ACM Fellow

        Director, Laboratory for Usage-centered Software Engineering (Lab:USE)

        Professor, Department of Mathematics & Engineering

          University of Madeira | Funchal , Portugal

          Chief Scientist, Constantine & Lockwood Ltd

      • Todd Zaki Warfel
        ... Unless you have a stellar moderator, focus groups tend to produce unreliable and near useless feedback. This is one tool I d recommend dropping from your
        Message 3 of 16 , Jul 7, 2008
        • 0 Attachment

          On Jul 5, 2008, at 5:23 PM, Nick Gassman wrote:

          Focus groups
          Get a number of users together, and talk about concepts and features. Maybe put together some pictures to show them.

          Unless you have a stellar moderator, focus groups tend to produce unreliable and near useless feedback. This is one tool I'd recommend dropping from your toolbox (unless you have a really, really, really great moderator). 

          Paper prototyping
          Run an interactive session with typical users, where one person is 'the computer' shifting bits of paper around.

          We've used this method quite a bit with Agile and in fact, I'll be teaching a workshop on Paper Prototype for Agile at the agile 2008 conference this year. Cost is low and you can get some really valuable feedback. Additionally, there are a number of techniques I teach for simulating AJAX interactions using paper. Most of the exercises in my workshop are 4-5 minutes long and end up with fully functional paper prototypes. 

          Remote testing in various flavours
          - Morae, observe and interact with individuals using a prototype
          - Put up static images somewhere public and ask for feedback
          - usertesting.com, where you put up 'stuff' and a specified demographic gives you feedback

          We've found remote testing to work well when you have a geographically dispersed group. In one case, we had a client who's employees were located in 32 countries. We used remote testing to observe their current workflows with 48 participants across several of these countries. Then used the same technique to review PDF wireframe prototypes. 

          My only caution against something like usertesting.com is that they use "professional testers," which kinda makes me wonder how "real" the feedback will be.

          Interview based 1-1 usability sessions
          Sit down with individual users and set them tasks, see what usability issues come up with your development

          We've used this method quite a bit in an Agile process. Our testing cycle is about 3-4 weeks to recruit, develop a script, prototype, test, and provide feedback. So, if done correctly, you can stay one cycle ahead of the sprint and provide feedback for the upcoming sprint. We used this method this past year for about 6 cycles for a project with a large media/telecom company and it worked quite well.

          Eye tracking
          See what people look at on a screen, and how long for

          Tells you what, but not why. The why is most important.


          Cheers!

          Todd Zaki Warfel
          President, Design Researcher
          Messagefirst | Designing Information. Beautifully.
          ----------------------------------
          Contact Info
          Voice: (215) 825-7423
          Email: todd@...
          Twitter: zakiwarfel
          ----------------------------------
          In theory, theory and practice are the same.
          In practice, they are not.

        • Larry Constantine
          Nick, On user feedback techniques, two quick comments: one plus, one minus. Scratch focus groups. Whether in waterfall or agile, focus groups have a long
          Message 4 of 16 , Jul 7, 2008
          • 0 Attachment

            Nick,

             

            On user feedback techniques, two quick comments: one plus, one minus.

             

            Scratch focus groups. Whether in waterfall or agile, focus groups have a long record of yielding user feedback that is too often spurious and unusable.

             

            Consider collaborative usability inspections. They are fast, involve both users and the development team in a highly structured walkthrough, and yield rich results.

             

            --Larry Constantine, IDSA, ACM Fellow

            Director, Laboratory for Usage-centered Software Engineering (Lab:USE)

            Professor, Department of Mathematics & Engineering

              University of Madeira | Funchal , Portugal

              Chief Scientist, Constantine & Lockwood Ltd

          • Nick Gassman
            ... Hmm. You and Larry Constantine both said this. We use focus groups quite a bit for different types of research, often concepts. It s not the right thing
            Message 5 of 16 , Jul 7, 2008
            • 0 Attachment
              On Mon, 7 Jul 2008 09:05:52 -0400, you wrote:

              >
              >On Jul 5, 2008, at 5:23 PM, Nick Gassman wrote:
              >
              >> Focus groups
              >> Get a number of users together, and talk about concepts and
              >> features. Maybe put together some pictures to show them.
              >
              >Unless you have a stellar moderator, focus groups tend to produce
              >unreliable and near useless feedback. This is one tool I'd recommend
              >dropping from your toolbox (unless you have a really, really, really
              >great moderator).
              >
              Hmm. You and Larry Constantine both said this. We use focus groups
              quite a bit for different types of research, often concepts. It's not
              the right thing for detailed usability, but to get a reaction from a
              group about the type of features they want and to stimulate some
              discussion I think it's quite useful. Are you saying that focus groups
              are no good for any type of research, or just some?

              >We've used this method quite a bit with Agile and in fact, I'll be
              >teaching a workshop on Paper Prototype for Agile at the agile 2008
              >conference this year. Cost is low and you can get some really valuable

              See you there.

              * Nick Gassman - Usability and Standards Manager - http://ba.com *
            • Todd Zaki Warfel
              ... You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you re feedback is even richer, as it has context
              Message 6 of 16 , Jul 7, 2008
              • 0 Attachment

                On Jul 7, 2008, at 3:33 PM, Nick Gassman wrote:

                but to get a reaction from a group about the type of features they want and to stimulate some discussion I think it's quite useful. Are you saying that focus groups are no good for any type of research, or just some?

                You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you're feedback is even richer, as it has context and doesn't suffer from group think. Not to mention that what people say they want and what they actually use are often quite different. You're better off doing some ethnographic based field research and watching for things people need, than asking a bunch of people "what they think they want." 

                They can be useful as a validation technique, but there are so many other methods, I don't personally put a lot of value in them.


                Cheers!

                Todd Zaki Warfel
                President, Design Researcher
                Messagefirst | Designing Information. Beautifully.
                ----------------------------------
                Contact Info
                Voice: (215) 825-7423
                Email: todd@...
                Twitter: zakiwarfel
                ----------------------------------
                In theory, theory and practice are the same.
                In practice, they are not.

              • Larry Constantine
                Nick, You said: We use focus groups quite a bit for different types of research, often concepts. It s not the right thing for detailed usability, but to get a
                Message 7 of 16 , Jul 7, 2008
                • 0 Attachment

                  Nick,

                   

                  You said:

                   

                  We use focus groups
                  quite a bit for different types of research, often concepts. It's not
                  the right thing for detailed usability, but to get a reaction from a
                  group about the type of features they want and to stimulate some
                  discussion I think it's quite useful. Are you saying that focus groups
                  are no good for any type of research, or just some?

                   

                  Years of horror stories from clients have convinced me that even for investigating concepts and what features users/customers want, focus groups are dangerously misleading. First, people do not know what they want; second, what they say is often loosely related if at all with what they want; third, what they want is only weakly correlated with what they need; and fourth, group and social effects dominate over directness in focus groups, even when led by relatively good facilitators. In the end, you really have no idea what the relationship between stated or summarized “findings” and genuine needs and interests. I have literally seen companies invest millions in product and Web development based on and validated by focus groups, only to discover that they had built the wrong system the wrong way.

                   

                  --Larry Constantine, IDSA, ACM Fellow

                    Director, Laboratory for Usage-centered Software Engineering | www.labuse.org

                    University of Madeira | Funchal , Portugal

                    Chief Scientist, Constantine & Lockwood Ltd

                • Tillier, Ivor - Oxford
                  Hi all, Its like the example I read recently on another list about the question asked to a fast-food chain s customers, Would you buy a healthier version of
                  Message 8 of 16 , Jul 8, 2008
                  • 0 Attachment

                    Hi all,

                    Its like the example I read recently on another list  about the question asked to a fast-food chain’s customers, ‘Would you buy a healthier version of our burgers?’.  In the example, many people said ‘yes’ but this didn’t result in increased sales when one was produced because although people felt they should say yes, in reality they wanted the good old high fat version.  These kind of factors can of course be compounded in group situations.

                     

                    I can see some benefit of group situations though, in that discussion might encourage users to think about, and explore, issues beyond what they thought were important (OK, a researcher can encourage this in a one–to-one, but it is only after talking to a few people will they have a wide enough range of feedback to enable them to steer the user to different areas that the user might not have considered so far - but have been raised by other users).

                     

                    Ivor

                     

                     

                    Ivor Tillier

                    Senior Web Producer

                    01865 476596 itillier@...

                     

                    Please do not print this e-mail

                     

                    From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Todd Zaki Warfel
                    Sent: 07 July 2008 20:41
                    To: agile-usability@yahoogroups.com
                    Subject: Re: [agile-usability] Best methods for user feedback before release

                     

                     

                    On Jul 7, 2008, at 3:33 PM, Nick Gassman wrote:



                    but to get a reaction from a group about the type of features they want and to stimulate some discussion I think it's quite useful. Are you saying that focus groups are no good for any type of research, or just some?

                     

                    You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you're feedback is even richer, as it has context and doesn't suffer from group think. Not to mention that what people say they want and what they actually use are often quite different. You're better off doing some ethno! graphic based field research and watching for things people need, than asking a bunch of people "what they think they want." 

                     

                    They can be useful as a validation technique, but there are so many other methods, I don't personally put a lot of value in them.

                     


                    Cheers!

                     

                    Todd Zaki Warfel

                    President, Design Researcher

                    Messagefirst | Designing Information. Beautifully.

                    ----------------------------------

                    Contact Info

                    Voice:      (215) 825-7423

                    Email:      todd@...

                    AIM:        twarf! el@...

                    Blog:        http://toddwarfel.com

                    Twitter:    zakiwarfel

                    ----------------------------------

                    In theory, theory and practice are the same.

                    In practice, they are not.

                     


                    This email (and any attachment) is confidential, may be legally privileged and is intended solely for the
                    use of the individual or entity to whom it is addressed. If you are not the intended recipient please do
                    not disclose, copy or take any action in reliance on it. If you have received this message in error please
                    tell us by reply and delete all copies on your system.
                     
                    Although this email has been scanned for viruses you should rely on your own virus check as the sender
                    accepts no liability for any damage arising out of any bug or virus infection. Please note that email
                    traffic data may be monitored and that emails may be viewed for security reasons.

                    Blackwell Publishing Limited is a private limited company registered in England with registered number 180277.

                    Registered office address: The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ.


                  • William Pietri
                    ... Although I am also very suspicious of focus groups, I have seen people get value out of something weirdly similar: parties and get-togethers. Here in the
                    Message 9 of 16 , Jul 8, 2008
                    • 0 Attachment
                      Larry Constantine wrote:

                      Years of horror stories from clients have convinced me that even for investigating concepts and what features users/customers want, focus groups are dangerously misleading. First, people do not know what they want; second, what they say is often loosely related if at all with what they want; third, what they want is only weakly correlated with what they need; and fourth, group and social effects dominate over directness in focus groups, even when led by relatively good facilitators. In the end, you really have no idea what the relationship between stated or summarized “findings” and genuine needs and interests. I have literally seen companies invest millions in product and Web development based on and validated by focus groups, only to discover that they had built the wrong system the wrong way.

                       


                      Although I am also very suspicious of focus groups, I have seen people get value out of something weirdly similar: parties and get-togethers.

                      Here in the land of startups, it's pretty common for people to have launch parties, in-office happy hours, and casual get-togethers with staff, users, and other stakeholders. There is naturally a lot of conversation around topics similar to what one might try to use a focus group to address. People seem to get a lot of value out of that conversation, walking away with ideas, background information, and a better appreciation for the lives of users.

                      Perhaps the key difference is in the informality. Nobody would think of writing up "findings" for a party; people know to treat the information gained very lightly.


                      William

                    Your message has been successfully submitted and would be delivered to recipients shortly.