Loading ...
Sorry, an error occurred while loading the content.

Re: Agile User Research

Expand Messages
  • brett.christenson
    Greg - thanks for the feedback. I was getting wrapped up in the quantifiable aspect of a testing approach. Seeing some of the tools out there with test reports
    Message 1 of 16 , Feb 16, 2012
    • 0 Attachment
      Greg - thanks for the feedback. I was getting wrapped up in the quantifiable aspect of a testing approach. Seeing some of the tools out there with test reports made me think this was a great approach. I will look into your suggestions and see how we can incorporate them.

      --- In agile-usability@yahoogroups.com, "gregc" <greg@...> wrote:
      >
      > Brett,
      >
      > I love analytics and totally admire your desire to to create repeatable and quantifiable tests. But as I read this I think the type of testing you are looking to do may be enriched and more supportable with a qualitative approach that emphasizes high touch. For example, when I test early sketches, I'm looking to identify big gaps in the UI or functionality. I'll pick a small set of representative customers, I customer I know who would use this functionality, and see if they can achieve their task. I've also done these via webex and gotomeeting for distance clients. During this, I'm asking them to think outloud and we're having a conversation. When I have a polished UI mock-up, I might use a remote usability testing tool. But not at rough mocks. And I would do this work as look ahead planning to groom a story to be accepted into the sprint.
      >
      > Post Sprint - depending what you want to learn might I might do an remote usability test but also may go back to the mock-up group to see if the implementation works for them. If I want to test pure learn-ability, then the prior group is tainted. So i would find some new research subjects. And asynchronous remote usability testing works well if you're not needing to manage the relationship with client.
      >
      > For identifying current user needs, beyond what's going through support, I'd get out of the office and spend time watching users use your product, what is the trigger for using your product, what are they trying to accomplish in your product, and what is the output they create with your product, who consumes that output, and what action do they take with that output.
      >
      > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
      >
      > Best,
      >
      > Greg Cohen
      > Author: Agile Excellence for Product Managers | 42 Rules of Product Management | Lean Product Management
      >
      >
      >
      >
      >
      >
      >
      > --- In agile-usability@yahoogroups.com, "brett.christenson" <bchristenson@> wrote:
      > >
      > > I have been working in agile development environment for some years now and just moving to working with engineers has proven to be beneficial. Now I want to gather feedback beyond inside our company and before product is officially released. I have never done in depth user testing but I am looking to gather user feedback in a couple areas:
      > > 1. Early sketch mock ups and storyboards. I think that it is important that we get user input to make sure we are starting in the right direction. My thought is that this feedback should be able to be gathered at any point in a sprint or even as way to create stories. Great use case for this would be new product or major functionality development with select beta partners or focus group.
      > > 2. Post sprint user testing - Can users accomplish task? My concern is how to get this feedback in a timely manor to incorporate that feedback in a product release time.
      > > 3. Identify current user needs - Where do we need to focus on the user experience? I see this more outside active sprints as a way to help generate user stories.
      > >
      > > I have been looking into tools like http://www.usabilla.com/ so that I can perform more structured testing on focused areas. I am also looking into using surveys to help further support ideas/concepts and direction without to much design overhead.
      > >
      > > Any process or methods we use need to be repeatable (can we always generate survey or user test after each sprint?)and allow designers to gather maybe some metrics that can be used to show the effect of changes on the user experience. Simple example - UI test 1 users could not complete task x. Test 2 40% users completed task x. Test 3 80% users completed task x.
      > >
      > > Again I know that these things are all possible but I wonder if there are any tools or techniques that other designers have used during agile development with high level of success.
      > >
      > > --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@> wrote:
      > > >
      > > > Hi Brett,
      > > >
      > > > On 9 Feb 2012, at 14:15, brett.christenson wrote:
      > > >
      > > > > I am looking for repeatable and measurable user research methods/tools that I can incorporate into agile development. Any advice would be greatly appreciated.
      > > >
      > > >
      > > > That's a pretty broad question :-)
      > > >
      > > > Can you talk a little bit more about your context and the kind of problems that you're looking to solve?
      > > >
      > > > Cheers,
      > > >
      > > > Adrian
      > > > --
      > > > http://quietstars.com adrianh@ twitter.com/adrianh
      > > > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
      > > >
      > >
      >
    • brett.christenson
      So I am moving forward with User research and with everyone s advice I plan to do interactive test of users. However in an effort to structure this process I
      Message 2 of 16 , Mar 6, 2012
      • 0 Attachment
        So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
        I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

        Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

        Thanks again to everyone


        --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
        >
        > Hi folks,
        >
        > On 15 Feb 2012, at 17:38, gregc wrote:
        > [snip]
        > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
        > [snip]
        >
        > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
        >
        > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
        >
        > Cheers,
        >
        > Adrian
        > --
        > http://quietstars.com adrianh@... twitter.com/adrianh
        > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
        >
      • Dan Fitek
        There are some good resources on the Measuring the User Experience book website http://measuringux.com/ Check out Usability Test Data
        Message 3 of 16 , Mar 6, 2012
        • 0 Attachment
          There are some good resources on the Measuring the User Experience book website http://measuringux.com/

          Check out Usability Test Data Logger  and Usability Test Observation Coding Form

          -Dan

          On Tue, Mar 6, 2012 at 3:02 PM, brett.christenson <bchristenson@...> wrote:
           

          So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
          I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

          Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

          Thanks again to everyone



          --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
          >
          > Hi folks,
          >
          > On 15 Feb 2012, at 17:38, gregc wrote:
          > [snip]
          > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
          > [snip]
          >
          > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
          >
          > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
          >
          > Cheers,
          >
          > Adrian
          > --
          > http://quietstars.com adrianh@... twitter.com/adrianh
          > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
          >


        • Jon Innes
          I second Dan s recommendation of that book. You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect
          Message 4 of 16 , Mar 6, 2012
          • 0 Attachment

            I second Dan's recommendation of that book. 

            You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.

            Some other tips:
            • Make sure the team can participate in studies as observers. This is easy to do with remote usability testing and helps them understand any issues you find.
            • Have any observers take notes and send them to you as an email so they feel involved, but make sure any issues become part of the backlog.
            • Hold a debrief meeting after you have observed enough users to see trends to summarize the findings.
            • If you capture metrics, note them near the team's backlog or scrum board in a summary format. Even if its just # of users tested & # any issue. 
            • Even when capturing recommendations on a server or wiki, it's often best to work with the PO or SM to post 1 page paper summaries of these in the team space.
            If you are successful and start facing issues scaling this, I gave some pointers on this in a post:


            Credit to Josh Seiden for term "test driven design" is due. While I've seen it used for QA before, it's a great way to describe best practices for Lean UX.

            Good luck,

            Jon


            On Mar 6, 2012, at 7:04 PM, Dan Fitek wrote:

             

            There are some good resources on the Measuring the User Experience book website http://measuringux.com/

            Check out Usability Test Data Logger  and Usability Test Observation Coding Form

            -Dan

            On Tue, Mar 6, 2012 at 3:02 PM, brett.christenson <bchristenson@...> wrote:
             

            So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results? This will be very helpful when presenting findings to teams. Having a template for UX team members helps standardize this process and ensure the correct information is gathered and documented.
            I can also see where this would be very beneficial is when testing with same user from one sprint to another as improvements are made to the UI.

            Along the agile principal guidelines I do not want to have a 300 page document created after test just something to assist team. Google search for User Research template has not produced many results that I find very helpful.

            Thanks again to everyone



            --- In agile-usability@yahoogroups.com, Adrian Howard <adrianh@...> wrote:
            >
            > Hi folks,
            >
            > On 15 Feb 2012, at 17:38, gregc wrote:
            > [snip]
            > > Just to reiterate, I'm not knocking quantifiable research. I'm a big advocate of it. And you will want to use it. But I don't think it's effective to apply for every change in every sprint and in the usage scenarios you mentioned, I just want to make sure you're not overlooking the good old fashioned qualitative methods of going deep with a smaller sample of customers.
            > [snip]
            >
            > A general +1 to that. On reading Brett's description the first thing that seemed an obvious lack was going out and talking and doing interactive tests real users.
            >
            > Doing that frequently gives me much more bang for my buck than surveys and questionnaires. Not that they don't have value too - but setting things up to get lots regular direct user interaction is what I'd do first.
            >
            > Cheers,
            >
            > Adrian
            > --
            > http://quietstars.com adrianh@... twitter.com/adrianh
            > t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
            >





          • Adrian Howard
            Hi Brett, ... I ve written on this briefly at http://is.gd/8zsXkR - and I third Dan s recommended reading :) Cheers, Adrian -- http://quietstars.com
            Message 5 of 16 , Mar 8, 2012
            • 0 Attachment
              Hi Brett,

              On 6 Mar 2012, at 20:02, brett.christenson wrote:

              > So I am moving forward with User research and with everyone's advice I plan to do interactive test of users. However in an effort to structure this process I was wondering if anyone has a research template they use to preform tests and capture results?

              I've written on this briefly at http://is.gd/8zsXkR - and I third Dan's recommended reading :)

              Cheers,

              Adrian
              --
              http://quietstars.com adrianh@... twitter.com/adrianh
              t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
            • Adrian Howard
              On 7 Mar 2012, at 03:30, Jon Innes wrote: [snip] ... [snip] Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs
              Message 6 of 16 , Mar 8, 2012
              • 0 Attachment
                On 7 Mar 2012, at 03:30, Jon Innes wrote:
                [snip]
                > You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.
                [snip]

                Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs about what's missing (it's often in the definition of "done" for the story rather than in a description of the feature).

                Cheers,

                Adrian
                --
                http://quietstars.com adrianh@... twitter.com/adrianh
                t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
              • Jeremy Kriegel
                I like Adrian s approach of different color pens. I ve tended to setup an excel spreadsheet with each question/task of the test as a column and each user as a
                Message 7 of 16 , Mar 8, 2012
                • 0 Attachment
                  I like Adrian's approach of different color pens. I've tended to setup an excel spreadsheet with each question/task of the test as a column and each user as a row (can be reversed). That way it is easy to type structured notes and compare them across users quickly.

                  Try to use a separate note taker. It is hard enough for one person to pay attention to ask sufficiently probing questions based on what they are observing. If you also have to try and capture accurately, you will miss a lot. It is also a great opportunity to involve the PO or a developer or some other non-ux role in the test. Personally, I almost always use the PO as my note taker. They get to experience the session first hand, and changes to the backlog are a conversation between us as we both were there. It also builds support for future testing initiatives.

                  good luck,

                  -jer

                  "Be well, do good work & keep in touch."
                       - Garrison Keillor


                  On Thu, Mar 8, 2012 at 12:17 PM, Adrian Howard <adrianh@...> wrote:

                  On 7 Mar 2012, at 03:30, Jon Innes wrote:
                  [snip]
                  > You are correct to avoid lengthy reports. A much better practice for agile projects is to leverage the defect tracking system, especially if it supports screen shot uploads, which most do now. Even if it doesn't just create a link in the defect to a wiki or file server where you can annotate any screenshots that illustrate issues.
                  [snip]

                  Or even just go to the story board, pick up a card, and go have a conversation with the PO and devs about what's missing (it's often in the definition of "done" for the story rather than in a description of the feature).

                  Cheers,

                  Adrian
                  --
                  http://quietstars.com     adrianh@...     twitter.com/adrianh
                  t. +44 (0)7752 419080     skype adrianjohnhoward     del.icio.us/adrianh





                  ------------------------------------

                  Yahoo! Groups Links

                  <*> To visit your group on the web, go to:
                     http://groups.yahoo.com/group/agile-usability/

                  <*> Your email settings:
                     Individual Email | Traditional

                  <*> To change settings online go to:
                     http://groups.yahoo.com/group/agile-usability/join
                     (Yahoo! ID required)

                  <*> To change settings via email:
                     agile-usability-digest@yahoogroups.com
                     agile-usability-fullfeatured@yahoogroups.com

                  <*> To unsubscribe from this group, send an email to:
                     agile-usability-unsubscribe@yahoogroups.com

                  <*> Your use of Yahoo! Groups is subject to:
                     http://docs.yahoo.com/info/terms/



                Your message has been successfully submitted and would be delivered to recipients shortly.