Loading ...
Sorry, an error occurred while loading the content.
 

Best methods for user feedback before release

Expand Messages
  • Nick Gassman
    I asked something like this recently, but didn t get responses. Let me try it a different way. In order to get user feedback before release of a software
    Message 1 of 16 , Jul 5, 2008
      I asked something like this recently, but didn't get responses. Let me
      try it a different way.

      In order to get user feedback before release of a software
      development, there are a number of techniques that can be used. These
      include the following (the definitions are not definitive, but
      intended to stimulate debate)

      Focus groups
      Get a number of users together, and talk about concepts and features.
      Maybe put together some pictures to show them.

      Paper prototyping
      Run an interactive session with typical users, where one person is
      'the computer' shifting bits of paper around

      Remote testing in various flavours
      - Morae, observe and interact with individuals using a prototype
      - Put up static images somewhere public and ask for feedback
      - usertesting.com, where you put up 'stuff' and a specified
      demographic gives you feedback

      Interview based 1-1 usability sessions
      Sit down with individual users and set them tasks, see what usability
      issues come up with your development

      Eye tracking
      See what people look at on a screen, and how long for

      Card sorting
      Categorise and structure information

      Panels
      Recruit a panel who you talk to in a closed forum, usually online

      Users as members of the team
      Get real users to join in key discussions and meetings

      Get feedback from proxy users
      Ask the people in the next office, or your friends and relatives what
      they think


      These methods all have their place. Which ones do you use, and at what
      stage in development? Is there a conflict in scheduling testing with
      development timescales?

      Some discussions I've read appear to assume that users are easy to
      recruit to participate. Bear in mind that for us (as for for many
      others) our users are the public, not internal employees. Some of
      those public are time-poor frequent flyers, and our customers are in
      globally dispersed locations, speaking many languages.

      * Nick Gassman - Usability and Standards Manager - http://ba.com *
    • Daniel Szuc
      Hi Nick: In 2005 I ran a workshop with Whitney Quesenbery (www.wqusability.com) in Shanghai called Choosing the Right Usability Tool (the right technique for
      Message 2 of 16 , Jul 5, 2008
        Hi Nick:

        In 2005 I ran a workshop with Whitney Quesenbery (www.wqusability.com)
        in Shanghai called "Choosing the Right Usability Tool (the right
        technique for the right problem)" -- abstract below.

        Some of the variables we looked at when selecting a tool included:

        * Business landscape - usability maturity, receptiveness, relationship
        with project team
        * Risk factors - new product or concept? Who are the users? i.e. hard
        to recruit etc
        * When do you need results and who you need to communicate results to
        * Others - how many users? Stage of the project? Budget?

        This helped us provide more context when planning our approach.

        rgds,
        Dan

        --
        Daniel Szuc
        Principal Usability Consultant
        Apogee Usability Asia Ltd
        www.apogeehk.com
        Usability in Asia

        The Usability Kit - www.theusabilitykit.com

        ABSTRACT -- http://www.upachina.org/userfriendly2005/workshops_en.htm

        We have a large toolkit of usability techniques, from user interviews
        to contextual inquiry, from informal evaluation with a few users to
        large-scale formal tests. This workshop provides a framework for
        deciding what tools to use in any project. For experienced
        professionals, it will help match the business context to the best
        technique. It will also be valuable for new professionals, providing a
        good understanding of the range of options available to them in a user-
        centered approach to product design.

        Participants will be able to:
        * Identify different usability tools
        * Understand the strengths and weaknesses of each technique
        * Match business or design goals to usability tools
        * Choose the best tools for different projects

        On 6 Jul 2008, at 5:23 AM, Nick Gassman wrote:

        > I asked something like this recently, but didn't get responses. Let me
        > try it a different way.
        >
        > In order to get user feedback before release of a software
        > development, there are a number of techniques that can be used. These
        > include the following (the definitions are not definitive, but
        > intended to stimulate debate)
        >
        > Focus groups
        > Get a number of users together, and talk about concepts and features.
        > Maybe put together some pictures to show them.
        >
        > Paper prototyping
        > Run an interactive session with typical users, where one person is
        > 'the computer' shifting bits of paper around
        >
        > Remote testing in various flavours
        > - Morae, observe and interact with individuals using a prototype
        > - Put up static images somewhere public and ask for feedback
        > - usertesting.com, where you put up 'stuff' and a specified
        > demographic gives you feedback
        >
        > Interview based 1-1 usability sessions
        > Sit down with individual users and set them tasks, see what usability
        > issues come up with your development
        >
        > Eye tracking
        > See what people look at on a screen, and how long for
        >
        > Card sorting
        > Categorise and structure information
        >
        > Panels
        > Recruit a panel who you talk to in a closed forum, usually online
        >
        > Users as members of the team
        > Get real users to join in key discussions and meetings
        >
        > Get feedback from proxy users
        > Ask the people in the next office, or your friends and relatives what
        > they think
        >
        > These methods all have their place. Which ones do you use, and at what
        > stage in development? Is there a conflict in scheduling testing with
        > development timescales?
        >
        > Some discussions I've read appear to assume that users are easy to
        > recruit to participate. Bear in mind that for us (as for for many
        > others) our users are the public, not internal employees. Some of
        > those public are time-poor frequent flyers, and our customers are in
        > globally dispersed locations, speaking many languages.
        >
        > * Nick Gassman - Usability and Standards Manager - http://ba.com *
        >
        >
      • Nick Gassman
        ... Dan thanks. I m familiar with the criteria - have you applied these to Agile projects? I m hoping for some real world examples of where practitioners have
        Message 3 of 16 , Jul 6, 2008
          On Sun, 6 Jul 2008 12:43:21 +0800, Dan wrote:

          >In 2005 I ran a workshop with Whitney Quesenbery (www.wqusability.com)
          >in Shanghai called "Choosing the Right Usability Tool (the right
          >technique for the right problem)" -- abstract below.

          Dan thanks. I'm familiar with the criteria - have you applied these to
          Agile projects?

          I'm hoping for some real world examples of where practitioners have
          found that doing an Agile project means that you take a different
          approach from waterfall.

          In a waterfall development we'd normally schedule one or more
          studio-based usability sessions a couple of months away, once we'd
          developed some screens. It seems this won't work so well in an Agile
          environment, so I wonder what people do?

          To be hones, I'm slightly puzzled that in this forum there's not more
          debate about this. If we were talking about Agile from a tech
          perspective I'd understand, but how and when you get user input is
          surely a core component 'agile-usability'?

          There will be plenty more subscribers to this list than active
          participants. Some people reading this must have views from their own
          experience - I hope.

          * Nick Gassman - Usability and Standards Manager - http://ba.com *
        • Daniel Szuc
          Hi Nick: More Waterfall experience :( On the Agile projects we decided to choose 2-3 UX tools we could repeat confidently e.g interviews, usability walkthrough
          Message 4 of 16 , Jul 6, 2008
            Hi Nick:

            More Waterfall experience :(

            On the Agile projects we decided to choose 2-3 UX tools we could
            repeat confidently e.g interviews, usability walkthrough and usability
            testing. The main benefit being not to overwhelm the development team
            and be able to iterate the findings into the process more easily.

            We were also not always able to involve users directly (shock, horror)
            and relied more on Usability Walkthroughs with the team -- http://www.uxmatters.com/MT/archives/000199.php
            -- this was not always ideal as it would have been nicer to get more
            user involvement (as the business knowledge on the end users is not
            always accurate -- although they think it is ;)

            Suggest if you do look to involve users as part of rolling plan -- it
            helps to have a good contact on the business side who can source and
            rotate users in and out as you need them. Also helps get continuous
            buy in as the product iterates forward.

            rgds,
            Dan

            On 6 Jul 2008, at 6:17 PM, Nick Gassman wrote:

            > On Sun, 6 Jul 2008 12:43:21 +0800, Dan wrote:
            >
            > >In 2005 I ran a workshop with Whitney Quesenbery
            > (www.wqusability.com)
            > >in Shanghai called "Choosing the Right Usability Tool (the right
            > >technique for the right problem)" -- abstract below.
            >
            > Dan thanks. I'm familiar with the criteria - have you applied these to
            > Agile projects?
            >
            > I'm hoping for some real world examples of where practitioners have
            > found that doing an Agile project means that you take a different
            > approach from waterfall.
            >
            > In a waterfall development we'd normally schedule one or more
            > studio-based usability sessions a couple of months away, once we'd
            > developed some screens. It seems this won't work so well in an Agile
            > environment, so I wonder what people do?
            >
            > To be hones, I'm slightly puzzled that in this forum there's not more
            > debate about this. If we were talking about Agile from a tech
            > perspective I'd understand, but how and when you get user input is
            > surely a core component 'agile-usability'?
            >
            > There will be plenty more subscribers to this list than active
            > participants. Some people reading this must have views from their own
            > experience - I hope.
            >
            > * Nick Gassman - Usability and Standards Manager - http://ba.com *
            >
            >
          • William Pietri
            Hi, Nick. ... Just out of curiosity, why are you limiting it to pre-release feedback? My clients are mainly very iterative, and are of the release early,
            Message 5 of 16 , Jul 6, 2008
              Hi, Nick.

              Nick Gassman wrote:
              > In order to get user feedback before release of a software
              > development, there are a number of techniques that can be used. [...]

              Just out of curiosity, why are you limiting it to pre-release feedback?
              My clients are mainly very iterative, and are of the "release early,
              release often" mindset. Much of their user feedback comes after release,
              not before.

              > These methods all have their place. Which ones do you use, and at what
              > stage in development? Is there a conflict in scheduling testing with
              > development timescales?

              Limiting it to pre-release activity, they do pretty much all of the
              methods you mention. The one exception is that none of my clients do
              eye-tracking studies, but that probably says more about the barrier to
              entry than anything about the value of the method.

              Their choice of which to do is pretty ad hoc. It's less that development
              has stages, and more that individual units of work do, with designs
              getting more detailed as the feature becomes more likely to be in an
              iteration. They use research whenever the cost is worth it to reduce
              some risk, or when there's some important question outstanding. The type
              of research depends on the need at hand.

              To make that more efficient, some will schedule a kind of research
              regularly, and pick the purpose dynamically. E.g, they'll have a
              recruiter send them 5 users every Wednesday afternoon, but they may lock
              in the research plan Wednesday morning. Or once a month they'll take a
              key user out to lunch. Or they'll maintain an active user community, and
              turn to it whenever they want feedback.


              Hoping that helps,

              William
            • Nick Gassman
              ... That s a good question. Partly it s based on the fact that we re not currently very good at ongoing tweaking of live functionality. We are getting better
              Message 6 of 16 , Jul 6, 2008
                On Sun, 06 Jul 2008 13:15:44 -0700, William wrote:

                >
                >Just out of curiosity, why are you limiting it to pre-release feedback?
                >My clients are mainly very iterative, and are of the "release early,
                >release often" mindset. Much of their user feedback comes after release,
                >not before.
                >
                That's a good question. Partly it's based on the fact that we're not
                currently very good at ongoing tweaking of live functionality. We are
                getting better at, and we are doing a lot of metrics now that we
                weren't before, and picking up and fixing things.

                I'm also very keen that we don't use Agile as an excuse for releasing
                bad work. What I mean by that is functionality that doesn't work well
                for our customers, and therefore for the business - especially if
                talking to a few customers in advance can help to iron out some key
                issues.

                The direction I'm going at present is that it would be a rare piece of
                functionality that we'd release to our customers that hadn't had some
                form of prior usability test, although we will be leaving more things
                to the post-release measure-and-fix cycle.

                >Their choice of which to do is pretty ad hoc. It's less that development
                >has stages, and more that individual units of work do, with designs
                >getting more detailed as the feature becomes more likely to be in an
                >iteration. They use research whenever the cost is worth it to reduce
                >some risk, or when there's some important question outstanding. The type
                >of research depends on the need at hand.
                >
                Does this imply that they work out designs before they are developed
                technically in an iteration?


                * Nick Gassman - Usability and Standards Manager - http://ba.com *
              • William Pietri
                ... That sounds reasonable to me. Given the audience size and your presumed budgets, that seems quite doable. For post-release feedback, along with the other
                Message 7 of 16 , Jul 6, 2008
                  Nick Gassman wrote:
                  The direction I'm going at present is that it would be a rare piece of
                  functionality that we'd release to our customers that hadn't had some
                  form of prior usability test, although we will be leaving more things
                  to the post-release measure-and-fix cycle.
                    

                  That sounds reasonable to me. Given the audience size and your presumed budgets, that seems quite doable.

                  For post-release feedback, along with the other approaches you mention, you could consider pushing toward extensive metrics and a strong A/B test environment. One client does amazing things with that. Last I heard, they had something like 80% of their traffic (in total about twice ba.com's traffic) on the main site, and the rest in five test environments. It lets them do extensive and precise fine-tuning of their app.


                  From a business perspective, it seems like you're in an interesting position. Normally,  a dominant company in well-developed industry is inclined to be pretty conservative. Why rock the boat? But given the turmoil in aviation, I imagine the web site is now looking more like an opportunity for relatively low-cost innovation and market differentiation.

                  If that's the case, more agile, feedback-driven approaches can be very beneficial. Not only can your customers help lead you to what they want, but being able to innovate more frequently than your competitors can force them to catch up. It is indecently fun to watch competitors launch an imitation of something you did months ago, especially if you have rolled out several improvements since then.

                  Their choice of which to do is pretty ad hoc. It's less that development 
                  has stages, and more that individual units of work do, with designs 
                  getting more detailed as the feature becomes more likely to be in an 
                  iteration. They use research whenever the cost is worth it to reduce 
                  some risk, or when there's some important question outstanding. The type 
                  of research depends on the need at hand.
                  
                      
                  Does this imply that they work out designs before they are developed
                  technically in an iteration?


                  Depends on the place, really, but my one-word answer would be "no".

                  At the good ones, for any given feature, they do enough up-front design that they are reasonably sure the unit of work won't blow up in the iteration. For well-understood work, there may be no design artifacts, as people either know what it will look like or are comfortable that they can figure it out. The more novel or contentious something is, the more thinking happens in advance, just so that people know what they're talking about.

                  Estimation is an important driver of that. During a planning game, product managers and designers need to be able to explain a feature enough for developers to estimate it. Often, that's pretty straightforward. With novel features, some design decisions will get made during estimation, frequently in response to developer feedback on costs. And occasionally, estimation will raise issues that are complicated enough that a feature will get taken off the agenda for that meeting, so that more thought and/or more research can be applied.

                  Basically, any given design decision gets made at the last responsible moment. As teams get comfortable with agile approaches, they generally discover that is a lot later than they once thought. But what work gets done when varies a lot depending on team, audience, and iteration length.

                  Hoping that helps,

                  William
                • Nick Gassman
                  ... One client does amazing things with that. Last I heard, they had something like 80% of their traffic (in total about twice ba.com s traffic) on the main
                  Message 8 of 16 , Jul 6, 2008
                    On Sun, 06 Jul 2008 15:16:23 -0700, William wrote:

                    >For post-release feedback, along with the other approaches you mention, you could consider pushing toward extensive metrics and a strong A/B test environment.
                    One client does amazing things with that. Last I heard, they had
                    something like 80% of their traffic (in total about twice ba.com's
                    traffic) on the main site, and the rest in five test environments. It
                    lets them do extensive and precise fine-tuning of their app.
                    >
                    Yes, we've done A/B testing, and it's been very useful. We're also
                    looking at multivariate testing software, which will allow us to
                    manipulate several variables on the same page, and statistically
                    identify which ones are having an effect. As ever, it's about being
                    aware of which tools are most appropriate.
                    >
                    >
                    >From a business perspective, it seems like you're in an interesting position. Normally, a dominant company in well-developed industry
                    is inclined to be pretty conservative. Why rock the boat? But given
                    the turmoil in aviation, I imagine the web site is now looking more
                    like an opportunity for relatively low-cost innovation and market
                    differentiation.


                    Hmm, I wouldn't describe us as 'dominant' (it all depends on the
                    context), although we are a big player. We have a seriously large code
                    base backing up our website, with a seriously large ongoing programme
                    of development. The airline industry has some big challenges to face,
                    and we need to find a way to reduce the cost and time to market of new
                    developments. Even if the market conditions were better, you always
                    need to look for better ways of doing things.

                    * Nick Gassman - Usability and Standards Manager - http://ba.com *
                  • Larry Constantine
                    Nick, On user feedback techniques, two quick comments: one plus, one minus. Scratch focus groups. Whether in waterfall or agile, focus groups have a long
                    Message 9 of 16 , Jul 7, 2008

                      Nick,

                       

                      On user feedback techniques, two quick comments: one plus, one minus.

                       

                      Scratch focus groups. Whether in waterfall or agile, focus groups have a long record of yielding user feedback that is too often spurious and unusable.

                       

                      Consider collaborative usability inspections. They are fast, involve both users and the development team in a highly structured walkthrough, and yield rich results.

                       

                      --Larry Constantine, IDSA, ACM Fellow

                      Director, Laboratory for Usage-centered Software Engineering (Lab:USE)

                      Professor, Department of Mathematics & Engineering

                        University of Madeira | Funchal , Portugal

                        Chief Scientist, Constantine & Lockwood Ltd

                    • Todd Zaki Warfel
                      ... Unless you have a stellar moderator, focus groups tend to produce unreliable and near useless feedback. This is one tool I d recommend dropping from your
                      Message 10 of 16 , Jul 7, 2008

                        On Jul 5, 2008, at 5:23 PM, Nick Gassman wrote:

                        Focus groups
                        Get a number of users together, and talk about concepts and features. Maybe put together some pictures to show them.

                        Unless you have a stellar moderator, focus groups tend to produce unreliable and near useless feedback. This is one tool I'd recommend dropping from your toolbox (unless you have a really, really, really great moderator). 

                        Paper prototyping
                        Run an interactive session with typical users, where one person is 'the computer' shifting bits of paper around.

                        We've used this method quite a bit with Agile and in fact, I'll be teaching a workshop on Paper Prototype for Agile at the agile 2008 conference this year. Cost is low and you can get some really valuable feedback. Additionally, there are a number of techniques I teach for simulating AJAX interactions using paper. Most of the exercises in my workshop are 4-5 minutes long and end up with fully functional paper prototypes. 

                        Remote testing in various flavours
                        - Morae, observe and interact with individuals using a prototype
                        - Put up static images somewhere public and ask for feedback
                        - usertesting.com, where you put up 'stuff' and a specified demographic gives you feedback

                        We've found remote testing to work well when you have a geographically dispersed group. In one case, we had a client who's employees were located in 32 countries. We used remote testing to observe their current workflows with 48 participants across several of these countries. Then used the same technique to review PDF wireframe prototypes. 

                        My only caution against something like usertesting.com is that they use "professional testers," which kinda makes me wonder how "real" the feedback will be.

                        Interview based 1-1 usability sessions
                        Sit down with individual users and set them tasks, see what usability issues come up with your development

                        We've used this method quite a bit in an Agile process. Our testing cycle is about 3-4 weeks to recruit, develop a script, prototype, test, and provide feedback. So, if done correctly, you can stay one cycle ahead of the sprint and provide feedback for the upcoming sprint. We used this method this past year for about 6 cycles for a project with a large media/telecom company and it worked quite well.

                        Eye tracking
                        See what people look at on a screen, and how long for

                        Tells you what, but not why. The why is most important.


                        Cheers!

                        Todd Zaki Warfel
                        President, Design Researcher
                        Messagefirst | Designing Information. Beautifully.
                        ----------------------------------
                        Contact Info
                        Voice: (215) 825-7423
                        Email: todd@...
                        Twitter: zakiwarfel
                        ----------------------------------
                        In theory, theory and practice are the same.
                        In practice, they are not.

                      • Larry Constantine
                        Nick, On user feedback techniques, two quick comments: one plus, one minus. Scratch focus groups. Whether in waterfall or agile, focus groups have a long
                        Message 11 of 16 , Jul 7, 2008

                          Nick,

                           

                          On user feedback techniques, two quick comments: one plus, one minus.

                           

                          Scratch focus groups. Whether in waterfall or agile, focus groups have a long record of yielding user feedback that is too often spurious and unusable.

                           

                          Consider collaborative usability inspections. They are fast, involve both users and the development team in a highly structured walkthrough, and yield rich results.

                           

                          --Larry Constantine, IDSA, ACM Fellow

                          Director, Laboratory for Usage-centered Software Engineering (Lab:USE)

                          Professor, Department of Mathematics & Engineering

                            University of Madeira | Funchal , Portugal

                            Chief Scientist, Constantine & Lockwood Ltd

                        • Nick Gassman
                          ... Hmm. You and Larry Constantine both said this. We use focus groups quite a bit for different types of research, often concepts. It s not the right thing
                          Message 12 of 16 , Jul 7, 2008
                            On Mon, 7 Jul 2008 09:05:52 -0400, you wrote:

                            >
                            >On Jul 5, 2008, at 5:23 PM, Nick Gassman wrote:
                            >
                            >> Focus groups
                            >> Get a number of users together, and talk about concepts and
                            >> features. Maybe put together some pictures to show them.
                            >
                            >Unless you have a stellar moderator, focus groups tend to produce
                            >unreliable and near useless feedback. This is one tool I'd recommend
                            >dropping from your toolbox (unless you have a really, really, really
                            >great moderator).
                            >
                            Hmm. You and Larry Constantine both said this. We use focus groups
                            quite a bit for different types of research, often concepts. It's not
                            the right thing for detailed usability, but to get a reaction from a
                            group about the type of features they want and to stimulate some
                            discussion I think it's quite useful. Are you saying that focus groups
                            are no good for any type of research, or just some?

                            >We've used this method quite a bit with Agile and in fact, I'll be
                            >teaching a workshop on Paper Prototype for Agile at the agile 2008
                            >conference this year. Cost is low and you can get some really valuable

                            See you there.

                            * Nick Gassman - Usability and Standards Manager - http://ba.com *
                          • Todd Zaki Warfel
                            ... You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you re feedback is even richer, as it has context
                            Message 13 of 16 , Jul 7, 2008

                              On Jul 7, 2008, at 3:33 PM, Nick Gassman wrote:

                              but to get a reaction from a group about the type of features they want and to stimulate some discussion I think it's quite useful. Are you saying that focus groups are no good for any type of research, or just some?

                              You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you're feedback is even richer, as it has context and doesn't suffer from group think. Not to mention that what people say they want and what they actually use are often quite different. You're better off doing some ethnographic based field research and watching for things people need, than asking a bunch of people "what they think they want." 

                              They can be useful as a validation technique, but there are so many other methods, I don't personally put a lot of value in them.


                              Cheers!

                              Todd Zaki Warfel
                              President, Design Researcher
                              Messagefirst | Designing Information. Beautifully.
                              ----------------------------------
                              Contact Info
                              Voice: (215) 825-7423
                              Email: todd@...
                              Twitter: zakiwarfel
                              ----------------------------------
                              In theory, theory and practice are the same.
                              In practice, they are not.

                            • Larry Constantine
                              Nick, You said: We use focus groups quite a bit for different types of research, often concepts. It s not the right thing for detailed usability, but to get a
                              Message 14 of 16 , Jul 7, 2008

                                Nick,

                                 

                                You said:

                                 

                                We use focus groups
                                quite a bit for different types of research, often concepts. It's not
                                the right thing for detailed usability, but to get a reaction from a
                                group about the type of features they want and to stimulate some
                                discussion I think it's quite useful. Are you saying that focus groups
                                are no good for any type of research, or just some?

                                 

                                Years of horror stories from clients have convinced me that even for investigating concepts and what features users/customers want, focus groups are dangerously misleading. First, people do not know what they want; second, what they say is often loosely related if at all with what they want; third, what they want is only weakly correlated with what they need; and fourth, group and social effects dominate over directness in focus groups, even when led by relatively good facilitators. In the end, you really have no idea what the relationship between stated or summarized “findings” and genuine needs and interests. I have literally seen companies invest millions in product and Web development based on and validated by focus groups, only to discover that they had built the wrong system the wrong way.

                                 

                                --Larry Constantine, IDSA, ACM Fellow

                                  Director, Laboratory for Usage-centered Software Engineering | www.labuse.org

                                  University of Madeira | Funchal , Portugal

                                  Chief Scientist, Constantine & Lockwood Ltd

                              • Tillier, Ivor - Oxford
                                Hi all, Its like the example I read recently on another list about the question asked to a fast-food chain s customers, Would you buy a healthier version of
                                Message 15 of 16 , Jul 8, 2008

                                  Hi all,

                                  Its like the example I read recently on another list  about the question asked to a fast-food chain’s customers, ‘Would you buy a healthier version of our burgers?’.  In the example, many people said ‘yes’ but this didn’t result in increased sales when one was produced because although people felt they should say yes, in reality they wanted the good old high fat version.  These kind of factors can of course be compounded in group situations.

                                   

                                  I can see some benefit of group situations though, in that discussion might encourage users to think about, and explore, issues beyond what they thought were important (OK, a researcher can encourage this in a one–to-one, but it is only after talking to a few people will they have a wide enough range of feedback to enable them to steer the user to different areas that the user might not have considered so far - but have been raised by other users).

                                   

                                  Ivor

                                   

                                   

                                  Ivor Tillier

                                  Senior Web Producer

                                  01865 476596 itillier@...

                                   

                                  Please do not print this e-mail

                                   

                                  From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Todd Zaki Warfel
                                  Sent: 07 July 2008 20:41
                                  To: agile-usability@yahoogroups.com
                                  Subject: Re: [agile-usability] Best methods for user feedback before release

                                   

                                   

                                  On Jul 7, 2008, at 3:33 PM, Nick Gassman wrote:



                                  but to get a reaction from a group about the type of features they want and to stimulate some discussion I think it's quite useful. Are you saying that focus groups are no good for any type of research, or just some?

                                   

                                  You can get this same feedback on-on-one during a usability session. In fact, during a usability session, you're feedback is even richer, as it has context and doesn't suffer from group think. Not to mention that what people say they want and what they actually use are often quite different. You're better off doing some ethno! graphic based field research and watching for things people need, than asking a bunch of people "what they think they want." 

                                   

                                  They can be useful as a validation technique, but there are so many other methods, I don't personally put a lot of value in them.

                                   


                                  Cheers!

                                   

                                  Todd Zaki Warfel

                                  President, Design Researcher

                                  Messagefirst | Designing Information. Beautifully.

                                  ----------------------------------

                                  Contact Info

                                  Voice:      (215) 825-7423

                                  Email:      todd@...

                                  AIM:        twarf! el@...

                                  Blog:        http://toddwarfel.com

                                  Twitter:    zakiwarfel

                                  ----------------------------------

                                  In theory, theory and practice are the same.

                                  In practice, they are not.

                                   


                                  This email (and any attachment) is confidential, may be legally privileged and is intended solely for the
                                  use of the individual or entity to whom it is addressed. If you are not the intended recipient please do
                                  not disclose, copy or take any action in reliance on it. If you have received this message in error please
                                  tell us by reply and delete all copies on your system.
                                   
                                  Although this email has been scanned for viruses you should rely on your own virus check as the sender
                                  accepts no liability for any damage arising out of any bug or virus infection. Please note that email
                                  traffic data may be monitored and that emails may be viewed for security reasons.

                                  Blackwell Publishing Limited is a private limited company registered in England with registered number 180277.

                                  Registered office address: The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ.


                                • William Pietri
                                  ... Although I am also very suspicious of focus groups, I have seen people get value out of something weirdly similar: parties and get-togethers. Here in the
                                  Message 16 of 16 , Jul 8, 2008
                                    Larry Constantine wrote:

                                    Years of horror stories from clients have convinced me that even for investigating concepts and what features users/customers want, focus groups are dangerously misleading. First, people do not know what they want; second, what they say is often loosely related if at all with what they want; third, what they want is only weakly correlated with what they need; and fourth, group and social effects dominate over directness in focus groups, even when led by relatively good facilitators. In the end, you really have no idea what the relationship between stated or summarized “findings” and genuine needs and interests. I have literally seen companies invest millions in product and Web development based on and validated by focus groups, only to discover that they had built the wrong system the wrong way.

                                     


                                    Although I am also very suspicious of focus groups, I have seen people get value out of something weirdly similar: parties and get-togethers.

                                    Here in the land of startups, it's pretty common for people to have launch parties, in-office happy hours, and casual get-togethers with staff, users, and other stakeholders. There is naturally a lot of conversation around topics similar to what one might try to use a focus group to address. People seem to get a lot of value out of that conversation, walking away with ideas, background information, and a better appreciation for the lives of users.

                                    Perhaps the key difference is in the informality. Nobody would think of writing up "findings" for a party; people know to treat the information gained very lightly.


                                    William

                                  Your message has been successfully submitted and would be delivered to recipients shortly.