Loading ...
Sorry, an error occurred while loading the content.

Re: [agile-usability] Re: Agile User Research

Expand Messages
  • Oksana Gerasymchuk
    Hi Brett, You can accomplish all these tasks (and many more) with UserZoom ( www.userzoom.com). It allows you to create remote usability (task-based) studies,
    Message 1 of 16 , Feb 14, 2012
    • 0 Attachment
      Hi Brett, 

      You can accomplish all these tasks (and many more) with UserZoom (www.userzoom.com). 

      It allows you to create remote usability (task-based) studies, card sorts, true-intent, voice of customer, tree tests, click tests and more. 

      One of UserZoom's strengths is powerful analytics. UserZoom captures survey data, task metrics (success ratios, time on task, number of clicks, number of unique pages viewed) and participant behavior. In addition, you can employ analytics filters to analyze subsets of your data. 

      You also mentioned that you need to be able to perform these tasks quickly. With UserZoom, since everything is done remotely, you can conduct your studies in a matter of hours . You login to your UserZoom instance, create tasks, send an invitation link to your users and collect the results (in fact, the software automatically collects all the data for you).  

      Happy testing! 


      On Tue, Feb 14, 2012 at 1:50 PM, brett.christenson <bchristenson@...> wrote:
       

      I have been working in agile development environment for some years now and just moving to working with engineers has proven to be beneficial. Now I want to gather feedback beyond inside our company and before product is officially released. I have never done in depth user testing but I am looking to gather user feedback in a couple areas:
      1. Early sketch mock ups and storyboards. I think that it is important that we get user input to make sure we are starting in the right direction. My thought is that this feedback should be able to be gathered at any point in a sprint or even as way to create stories. Great use case for this would be new product or major functionality development with select beta partners or focus group.
      2. Post sprint user testing - Can users accomplish task? My concern is how to get this feedback in a timely manor to incorporate that feedback in a product release time.
      3. Identify current user needs - Where do we need to focus on the user experience? I see this more outside active sprints as a way to help generate user stories.

      I have been looking into tools like http://www.usabilla.com/ so that I can perform more structured testing on focused areas. I am also looking into using surveys to help further support ideas/concepts and direction without to much design overhead.

      Any process or methods we use need to be repeatable (can we always generate survey or user test after each sprint?)and allow designers to gather maybe some metrics that can be used to show the effect of changes on the user experience. Simple example - UI test 1 users could not complete task x. Test 2 40% users completed task x. Test 3 80% users completed task x.

      Again I know that these things are all possible but I wonder if there are any tools or techniques that other designers have used during agile development with high level of success.



      --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
      >
      > Hi Brett,
      >
      > On 9 Feb 2012, at 14:15, brett.christenson wrote:
      >
      > > I am looking for repeatable and measurable user research methods/tools that I can incorporate into agile development. Any advice would be greatly appreciated.
      >
      >
      > That's a pretty broad question :-)
      >
      > Can you talk a little bit more about your context and the kind of problems that you're looking to solve?
      >
      > Cheers,
      >
      > Adrian
      > --
      > http://quietstars.com adrianh@... twitter.com/adrianh
      > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
      >


    • gregc
      Brett, I love analytics and totally admire your desire to to create repeatable and quantifiable tests. But as I read this I think the type of testing you are
      Message 2 of 16 , Feb 15, 2012
      • 0 Attachment
        Brett,

        I love analytics and totally admire your desire to to create repeatable and quantifiable tests. But as I read this I think the type of testing you are looking to do may be enriched and more supportable with a qualitative approach that emphasizes high touch. For example, when I test early sketches, I'm looking to identify big gaps in the UI or functionality. I'll pick a small set of representative customers, I customer I know who would use this functionality, and see if they can achieve their task. I've also done these via webex and gotomeeting for distance clients. During this, I'm asking them to think outloud and we're having a conversation. When I have a polished UI mock-up, I might use a remote usability testing tool. But not at rough mocks. And I would do this work as look ahead planning to groom a story to be accepted into the sprint.

        Post Sprint - depending what you want to learn might I might do an remote usability test but also may go back to the mock-up group to see if the implementation works for them. If I want to test pure learn-ability, then the prior group is tainted. So i would find some new research subjects. And asynchronous remote usability testing works well if you're not needing to manage the relationship with client.

        For identifying current user needs, beyond what's going through support, I'd get out of the office and spend time watching users use your product, what is the trigger for using your product, what are they trying to accomplish in your product, and what is the output they create with your product, who consumes that output, and what action do they take with that output.

        Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.

        Best,

        Greg Cohen
        Author: Agile Excellence for Product Managers | 42 Rules of Product Management | Lean Product Management







        --- In agile-usability@yahoogroups.com, "brett.christenson" <bchristenson@...> wrote:
        >
        > I have been working in agile development environment for some years now and just moving to working with engineers has proven to be beneficial. Now I want to gather feedback beyond inside our company and before product is officially released. I have never done in depth user testing but I am looking to gather user feedback in a couple areas:
        > 1. Early sketch mock ups and storyboards. I think that it is important that we get user input to make sure we are starting in the right direction. My thought is that this feedback should be able to be gathered at any point in a sprint or even as way to create stories. Great use case for this would be new product or major functionality development with select beta partners or focus group.
        > 2. Post sprint user testing - Can users accomplish task? My concern is how to get this feedback in a timely manor to incorporate that feedback in a product release time.
        > 3. Identify current user needs - Where do we need to focus on the user experience? I see this more outside active sprints as a way to help generate user stories.
        >
        > I have been looking into tools like http://www.usabilla.com/ so that I can perform more structured testing on focused areas. I am also looking into using surveys to help further support ideas/concepts and direction without to much design overhead.
        >
        > Any process or methods we use need to be repeatable (can we always generate survey or user test after each sprint?)and allow designers to gather maybe some metrics that can be used to show the effect of changes on the user experience. Simple example - UI test 1 users could not complete task x. Test 2 40% users completed task x. Test 3 80% users completed task x.
        >
        > Again I know that these things are all possible but I wonder if there are any tools or techniques that other designers have used during agile development with high level of success.
        >
        > --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@> wrote:
        > >
        > > Hi Brett,
        > >
        > > On 9 Feb 2012, at 14:15, brett.christenson wrote:
        > >
        > > > I am looking for repeatable and measurable user research methods/tools that I can incorporate into agile development. Any advice would be greatly appreciated.
        > >
        > >
        > > That's a pretty broad question :-)
        > >
        > > Can you talk a little bit more about your context and the kind of problems that you're looking to solve?
        > >
        > > Cheers,
        > >
        > > Adrian
        > > --
        > > http://quietstars.com adrianh@ twitter.com/adrianh
        > > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
        > >
        >
      • Adrian Howard
        Hi folks, On 15 Feb 2012, at 17:38, gregc wrote: [snip] ... [snip] A general +1 to that. On reading Brett s description the first thing that seemed an obvious
        Message 3 of 16 , Feb 16, 2012
        • 0 Attachment
          Hi folks,

          On 15 Feb 2012, at 17:38, gregc wrote:
          [snip]
          > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
          [snip]

          A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.

          Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.

          Cheers,

          Adrian
          --
          http://quietstars.com adrianh@... twitter.com/adrianh
          t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
        • brett.christenson
          Greg - thanks for the feedback. I was getting wrapped up in the quantifiable aspect of a testing approach. Seeing some of the tools out there with test reports
          Message 4 of 16 , Feb 16, 2012
          • 0 Attachment
            Greg - thanks for the feedback. I was getting wrapped up in the quantifiable aspect of a testing approach. Seeing some of the tools out there with test reports made me think this was a great approach. I will look into your suggestions and see how we can incorporate them.

            --- In agile-usability@yahoogroups.com, "gregc" <greg@...> wrote:
            >
            > Brett,
            >
            > I love analytics and totally admire your desire to to create repeatable and quantifiable tests. But as I read this I think the type of testing you are looking to do may be enriched and more supportable with a qualitative approach that emphasizes high touch. For example, when I test early sketches, I'm looking to identify big gaps in the UI or functionality. I'll pick a small set of representative customers, I customer I know who would use this functionality, and see if they can achieve their task. I've also done these via webex and gotomeeting for distance clients. During this, I'm asking them to think outloud and we're having a conversation. When I have a polished UI mock-up, I might use a remote usability testing tool. But not at rough mocks. And I would do this work as look ahead planning to groom a story to be accepted into the sprint.
            >
            > Post Sprint - depending what you want to learn might I might do an remote usability test but also may go back to the mock-up group to see if the implementation works for them. If I want to test pure learn-ability, then the prior group is tainted. So i would find some new research subjects. And asynchronous remote usability testing works well if you're not needing to manage the relationship with client.
            >
            > For identifying current user needs, beyond what's going through support, I'd get out of the office and spend time watching users use your product, what is the trigger for using your product, what are they trying to accomplish in your product, and what is the output they create with your product, who consumes that output, and what action do they take with that output.
            >
            > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
            >
            > Best,
            >
            > Greg Cohen
            > Author: Agile Excellence for Product Managers | 42 Rules of Product Management | Lean Product Management
            >
            >
            >
            >
            >
            >
            >
            > --- In agile-usability@yahoogroups.com, "brett.christenson" <bchristenson@> wrote:
            > >
            > > I have been working in agile development environment for some years now and just moving to working with engineers has proven to be beneficial. Now I want to gather feedback beyond inside our company and before product is officially released. I have never done in depth user testing but I am looking to gather user feedback in a couple areas:
            > > 1. Early sketch mock ups and storyboards. I think that it is important that we get user input to make sure we are starting in the right direction. My thought is that this feedback should be able to be gathered at any point in a sprint or even as way to create stories. Great use case for this would be new product or major functionality development with select beta partners or focus group.
            > > 2. Post sprint user testing - Can users accomplish task? My concern is how to get this feedback in a timely manor to incorporate that feedback in a product release time.
            > > 3. Identify current user needs - Where do we need to focus on the user experience? I see this more outside active sprints as a way to help generate user stories.
            > >
            > > I have been looking into tools like http://www.usabilla.com/ so that I can perform more structured testing on focused areas. I am also looking into using surveys to help further support ideas/concepts and direction without to much design overhead.
            > >
            > > Any process or methods we use need to be repeatable (can we always generate survey or user test after each sprint?)and allow designers to gather maybe some metrics that can be used to show the effect of changes on the user experience. Simple example - UI test 1 users could not complete task x. Test 2 40% users completed task x. Test 3 80% users completed task x.
            > >
            > > Again I know that these things are all possible but I wonder if there are any tools or techniques that other designers have used during agile development with high level of success.
            > >
            > > --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@> wrote:
            > > >
            > > > Hi Brett,
            > > >
            > > > On 9 Feb 2012, at 14:15, brett.christenson wrote:
            > > >
            > > > > I am looking for repeatable and measurable user research methods/tools that I can incorporate into agile development. Any advice would be greatly appreciated.
            > > >
            > > >
            > > > That's a pretty broad question :-)
            > > >
            > > > Can you talk a little bit more about your context and the kind of problems that you're looking to solve?
            > > >
            > > > Cheers,
            > > >
            > > > Adrian
            > > > --
            > > > http://quietstars.com adrianh@ twitter.com/adrianh
            > > > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
            > > >
            > >
            >
          • brett.christenson
            So I am moving forward with User research and with everyone s advice I plan to do interactive test of users. However in an effort to structure this process I
            Message 5 of 16 , Mar 6, 2012
            • 0 Attachment
              So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
              I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

              Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

              Thanks again to everyone


              --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
              >
              > Hi folks,
              >
              > On 15 Feb 2012, at 17:38, gregc wrote:
              > [snip]
              > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
              > [snip]
              >
              > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
              >
              > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
              >
              > Cheers,
              >
              > Adrian
              > --
              > http://quietstars.com adrianh@... twitter.com/adrianh
              > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
              >
            • Dan Fitek
              There are some good resources on the Measuring the User Experience book website http://measuringux.com/ Check out Usability Test Data
              Message 6 of 16 , Mar 6, 2012
              • 0 Attachment
                There are some good resources on the Measuring the User Experience book website http://measuringux.com/

                Check out Usability Test Data Logger  and Usability Test Observation Coding Form

                -Dan

                On Tue, Mar 6, 2012 at 3:02 PM, brett.christenson <bchristenson@...> wrote:
                 

                So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
                I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

                Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

                Thanks again to everyone



                --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
                >
                > Hi folks,
                >
                > On 15 Feb 2012, at 17:38, gregc wrote:
                > [snip]
                > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
                > [snip]
                >
                > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
                >
                > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
                >
                > Cheers,
                >
                > Adrian
                > --
                > http://quietstars.com adrianh@... twitter.com/adrianh
                > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
                >


              • Jon Innes
                I second Dan s recommendation of that book. You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect
                Message 7 of 16 , Mar 6, 2012
                • 0 Attachment

                  I second Dan's recommendation of that book. 

                  You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.

                  Some other tips:
                  • Make sure the team can participate in studies as observers. This is easy to do with remote usability testing and helps them understand any issues you find.
                  • Have any observers take notes and send them to you as an email so they feel involved, but make sure any issues become part of the backlog.
                  • Hold a debrief meeting after you have observed enough users to see trends to summarize the findings.
                  • If you capture metrics, note them near the team's backlog or scrum board in a summary format. Even if its just # of users tested & # any issue. 
                  • Even when capturing recommendations on a server or wiki, it's often best to work with the PO or SM to post 1 page paper summaries of these in the team space.
                  If you are successful and start facing issues scaling this, I gave some pointers on this in a post:


                  Credit to Josh Seiden for term "test driven design" is due. While I've seen it used for QA before, it's a great way to describe best practices for Lean UX.

                  Good luck,

                  Jon


                  On Mar 6, 2012, at 7:04 PM, Dan Fitek wrote:

                   

                  There are some good resources on the Measuring the User Experience book website http://measuringux.com/

                  Check out Usability Test Data Logger  and Usability Test Observation Coding Form

                  -Dan

                  On Tue, Mar 6, 2012 at 3:02 PM, brett.christenson <bchristenson@...> wrote:
                   

                  So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
                  I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

                  Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

                  Thanks again to everyone



                  --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
                  >
                  > Hi folks,
                  >
                  > On 15 Feb 2012, at 17:38, gregc wrote:
                  > [snip]
                  > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
                  > [snip]
                  >
                  > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
                  >
                  > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
                  >
                  > Cheers,
                  >
                  > Adrian
                  > --
                  > http://quietstars.com adrianh@... twitter.com/adrianh
                  > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
                  >





                • Adrian Howard
                  Hi Brett, ... I ve written on this briefly at http://is.gd/8zsXkR - and I third Dan s recommended reading :) Cheers, Adrian -- http://quietstars.com
                  Message 8 of 16 , Mar 8, 2012
                  • 0 Attachment
                    Hi Brett,

                    On 6 Mar 2012, at 20:02, brett.christenson wrote:

                    > So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results?

                    I've written on this briefly at http://is.gd/8zsXkR - and I third Dan's recommended reading :)

                    Cheers,

                    Adrian
                    --
                    http://quietstars.com adrianh@... twitter.com/adrianh
                    t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
                  • Adrian Howard
                    On 7 Mar 2012, at 03:30, Jon Innes wrote: [snip] ... [snip] Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs
                    Message 9 of 16 , Mar 8, 2012
                    • 0 Attachment
                      On 7 Mar 2012, at 03:30, Jon Innes wrote:
                      [snip]
                      > You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.
                      [snip]

                      Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs about what's missing (it's often in the definition of "done" for the story rather than in a description of the feature).

                      Cheers,

                      Adrian
                      --
                      http://quietstars.com adrianh@... twitter.com/adrianh
                      t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
                    • Jeremy Kriegel
                      I like Adrian s approach of different color pens. I ve tended to setup an excel spreadsheet with each question/task of the test as a column and each user as a
                      Message 10 of 16 , Mar 8, 2012
                      • 0 Attachment
                        I like Adrian's approach of different color pens. I've tended to setup an excel spreadsheet with each question/task of the test as a column and each user as a row (can be reversed). That way it is easy to type structured notes and compare them across users quickly.

                        Try to use a separate note taker. It is hard enough for one person to pay attention to ask sufficiently probing questions based on what they are observing. If you also have to try and capture accurately, you will miss a lot. It is also a great opportunity to involve the PO or a developer or some other non-ux role in the test. Personally, I almost always use the PO as my note taker. They get to experience the session first hand, and changes to the backlog are a conversation between us as we both were there. It also builds support for future testing initiatives.

                        good luck,

                        -jer

                        "Be well, do good work & keep in touch."
                             - Garrison Keillor


                        On Thu, Mar 8, 2012 at 12:17 PM, Adrian Howard <adrianh@...> wrote:

                        On 7 Mar 2012, at 03:30, Jon Innes wrote:
                        [snip]
                        > You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.
                        [snip]

                        Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs about what's missing (it's often in the definition of "done" for the story rather than in a description of the feature).

                        Cheers,

                        Adrian
                        --
                        http://quietstars.com     adrianh@...     twitter.com/adrianh
                        t. +44 (0)7752 419080     skype adrianjohnhoward     del.icio.us/adrianh





                        ------------------------------------

                        Yahoo! Groups Links

                        <*> To visit your group on the web, go to:
                           http://groups.yahoo.com/group/agile-usability/

                        <*> Your email settings:
                           Individual Email | Traditional

                        <*> To change settings online go to:
                           http://groups.yahoo.com/group/agile-usability/join
                           (Yahoo! ID required)

                        <*> To change settings via email:
                           agile-usability-digest@yahoogroups.com
                           agile-usability-fullfeatured@yahoogroups.com

                        <*> To unsubscribe from this group, send an email to:
                           agile-usability-unsubscribe@yahoogroups.com

                        <*> Your use of Yahoo! Groups is subject to:
                           http://docs.yahoo.com/info/terms/



                      Your message has been successfully submitted and would be delivered to recipients shortly.