Loading ...
Sorry, an error occurred while loading the content.

Re: Remote Usability Testing

Expand Messages
  • matyeo
    For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now. In part, it
    Message 1 of 14 , Jan 15, 2010
    • 0 Attachment
      For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now.

      In part, it reflects our frequent need to test across Canada, the US and internationally, at low cost. Another significant benefit is reduction in the cost of reruiting, but the benefits to the client organization are paramount - more observers from all levels of the organization, more buy-in, more drive for organizational change.

      We described these in one of our newsletter articles:

      http://www.neoinsight.com/newsletter/0706.html#RemoteUsability

      Cheers.
      Mike Atyeo
      Neo Insight

      --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
      >
      > Happy New Year!
      >
      > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
      > http://rosenfeldmedia.com/books/remote-research/
      >
      > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
      >
      > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
      >
      > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
      >
      > I would love to hear your remote-testing experiences!
      >
      > Tara Schnaible
      > Usability Analyst
      > The Nature Conservancy
      >
      > tschnaible@...
      >
      > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
      >
    • Elizabeth Bacon
      Hey folks, I also don t have any resources covering the relative efficacy of in- person and remote usability testing. My experienced designer gut says that
      Message 2 of 14 , Jan 15, 2010
      • 0 Attachment
        Hey folks,

        I also don't have any resources covering the relative efficacy of in-person and remote usability testing. My experienced designer gut says that there's no question one would learn less from remote usability testing due to the reduced opportunity to establish camaraderie & empathy and reduced ability to closely observe every human nuance & action. Nevertheless, it's an exploding practice and offers great time & cost-based efficiencies that may well offset the reduced quality/quantity of data that is obtained. 

        I had a need to do remote usability testing last year, so I did a comparative analysis of the various products on the market. I've published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en

        Enjoy! Bolt|Peters and others will be referencing this material in various forthcoming publications. Feel free to do the same, if you like, as well as letting me know if you're aware of other offerings that should go into this spreadsheet. 

        Cheers,
        Liz

        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Vice-President, IxDA / www.ixda.org
        CDO, Devise / www.devise.com
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~





      • jscottux
        There was an entire issue of UPA s User Experience magazine devoted to remote testing. You can read it here:
        Message 3 of 14 , Jan 16, 2010
        • 0 Attachment
          There was an entire issue of UPA's User Experience magazine devoted to remote testing. You can read it here:

          http://www.usabilityprofessionals.org/upa_publications/user_experience/past_issues/2008-3.html

          I also liked this article in Boxes and Arrows:

          http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it


          I'm with Mike. If we don't go to the participant, we usually have remote testing as part of the mix. Our pool of participants is too small without remote or travel or both. Our lab has morphed into a setup designed for remote testing as well as in lab participants. With that setup, I am able to gather the same rich computer data from Morae that I get with participants in the lab, which makes analysis far easier and, in our Agile world, faster.

          There are, in my opinion, a few things to watch out for:

          -- Lag. Occasionally, lag will cause a different experience for your participant if you share with them. If they share with you, the lag will cause you to observe an activity later than it actually happens. Both can be problematic for your analysis.
          -- Nonverbals. Unless your participants are able to have a webcam, you will need to gather some of your satisfaction data more explicitly. We will ask more frequently for reactions (debrief, words to describe, scaled reaction questions) to help mitigate that we can't see them saying "oh it was easy" when their face clearly shows it wasn't.
          --Observers. Certain remote setups may not allow observers. In Agile, your team needs have their skin in the game here as well; observing a few participants is a part of that. We work to make sure that our folks can see testing as it happens, despite the setup. They have to bring their observations to the findings meeting later.
          --Remote setup. Some of the tools available to us have a better user experience than others. Don't hesitate to connect yourself as a participant, and understand what your participant will see. Your testing results can be confounded by bad remote applications.

          I think we can say professionally that remote testing is firmly established and a respected method. Have at it and have fun.


          Cheers,
          Josie Scott
          TechSmith UX



          --- In agile-usability@yahoogroups.com, "matyeo" <mike@...> wrote:
          >
          >
          >
          > For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now.
          >
          > In part, it reflects our frequent need to test across Canada, the US and internationally, at low cost. Another significant benefit is reduction in the cost of reruiting, but the benefits to the client organization are paramount - more observers from all levels of the organization, more buy-in, more drive for organizational change.
          >
          > We described these in one of our newsletter articles:
          >
          > http://www.neoinsight.com/newsletter/0706.html#RemoteUsability
          >
          > Cheers.
          > Mike Atyeo
          > Neo Insight
          >
          > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
          > >
          > > Happy New Year!
          > >
          > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
          > > http://rosenfeldmedia.com/books/remote-research/
          > >
          > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
          > >
          > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
          > >
          > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
          > >
          > > I would love to hear your remote-testing experiences!
          > >
          > > Tara Schnaible
          > > Usability Analyst
          > > The Nature Conservancy
          > >
          > > tschnaible@
          > >
          > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
          > >
          >
        • Jared Spool
          Well, they aren t paying attention. I d look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I d also look to
          Message 4 of 14 , Jan 17, 2010
          • 0 Attachment
            Well, they aren't paying attention.

            I'd look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I'd also look to work that Rolf Molich is doing, particularly around CUE-8.

            Jared

            On Jan 14, 2010, at 1:09 PM, timkieschnick wrote:


            Julie Melton presented on the topic of remote usability just a couple of days ago at BayCHI , and I asked this very question.  No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog  in about two weeks (end of January).

             


            --- In agile-usability@ yahoogroups. com, "schntar" <tschnaible@. ..> wrote:
            >
            > Happy New Year!
            > 
            > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
            > http://rosenfeldmed ia.com/books/ remote-research/
            > 
            > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
            > 
            > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/ etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussi on - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
            > 
            > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
            > 
            > I would love to hear your remote-testing experiences!
            > 
            > Tara Schnaible
            > Usability Analyst
            > The Nature Conservancy
            > 
            > tschnaible@. ..
            > 
            > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
            >



          • Hernandez, Barbara
            here s a few I point folks to: An article from the ACM - http://portal.acm.org/citation.cfm?id=985921.986018 A comparison of synchronous remote and local
            Message 5 of 14 , Jan 17, 2010
            • 0 Attachment
              here's a few I point folks to:

              An article from the ACM - http://portal.acm.org/citation.cfm?id=985921.986018
              A comparison of synchronous remote and local usability studies for an expert interface
              Synchronous remote usability studies can be a convenient and cost-effective alternative to conventional local usability studies. Although they are common in the field, there has been little research comparing synchronous remote usability studies with local studies. In our comparison of remote and local studies of an expert interface, the primary differences were in the participant's and facilitator's qualitative experience. The remote and local studies agreed closely (with no significant differences) in terms of the number of usability issues found, their type, and their severity. While our comparison focuses on an expert interface and more work is needed to understand remote studies in general, our experience suggests that evaluators of expert interfaces will have comparable success identifying usability issues with either remote or local studies.

              bolt | peters also have an article - http://boltpeters.com/articles/versus.html

              We did a study for World Usability Day in 2005 - with similar results. I can send a copy along once I am back in the office in the morning.

              FWIW - we do tons of remote testing. We find that the data we get is in par with in-person testing, plus we get a peak at the users own personal environment and their computer setup. Pretty cool stuff. we have even done remote ethnography.

              Our biggest drawback is that people are more likely to bail - as in not show up for the session - with remote. There are also some challenges with technology - some tools work on the Mac some don't, some record the screen some don't, etc.

              We couldn't engage our users effectively without it.

              Regards
              Barb Hernandez
              User Experience Manager | TechSmith Corporation
              ________________________________________
              From: agile-usability@yahoogroups.com [agile-usability@yahoogroups.com] On Behalf Of Jared Spool [jspool@...]
              Sent: Sunday, January 17, 2010 7:30 PM
              To: agile-usability@yahoogroups.com
              Subject: Re: [agile-usability] Re: Remote Usability Testing

              Well, they aren't paying attention.

              I'd look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I'd also look to work that Rolf Molich is doing, particularly around CUE-8.

              Jared

              On Jan 14, 2010, at 1:09 PM, timkieschnick wrote:



              Julie Melton presented on the topic of remote usability just a c ouple of days ago at BayCHI<http://www.baychi.org/> , and I asked this very question. No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog <http://juliemelton.com/> in about two weeks (end of January).



              --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
              >
              > Happy New Year!
              >
              > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
              > http://rosenfeldmedia.com/books/remote-research/
              >
              > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
              >
              > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
              >
              > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
              >
              > I would love to hear your remote-testing experiences!
              >
              > Tara Schnaible
              > Usability Analyst
              > The Nature Conservancy
              >
              > tschnaible@...
              >
              > "If you k now a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
              >
            • Andrew Maier
              Hey Tara, I don t have much experience in the way of user testing in general, but I m also interested in the answer to this question. Before the agile
              Message 6 of 14 , Jan 18, 2010
              • 0 Attachment
                Hey Tara,

                I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.

                Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.

                However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.

                Thanks,
                Andrew Maier
                UX Booth, Editor in Chief

                --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                >
                > Happy New Year!
                >
                > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                > http://rosenfeldmedia.com/books/remote-research/
                >
                > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                >
                > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                >
                > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                >
                > I would love to hear your remote-testing experiences!
                >
                > Tara Schnaible
                > Usability Analyst
                > The Nature Conservancy
                >
                > tschnaible@...
                >
                > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                >
              • schntar
                Liz, I like your idea about breakdown of remote testing tools, I ve wondered myself about who the professional community is for Concept Feedback. I can speak
                Message 7 of 14 , Jan 19, 2010
                • 0 Attachment
                  Liz,
                  I like your idea about breakdown of remote testing tools, I've wondered myself about who the "professional community" is for Concept Feedback.
                  I can speak a bit to the camaraderie component of your testing - and say that if you have the right moderators conducting the test over the phone, it's not as much an issue. The physical body-language is lacking, but you can certainly hear a negative experience over a phone if you're paying attention.
                  A bit of background might explain this further.
                  The testing I do for my organization is largely on internal applications, with internal staff. Testers are often strangers (to the moderator) and they are never offered compensation for the hour+ it takes (we are a non-profit). If your moderator is friendly, approachable, easy-going (and focuses the call around getting an opinion in order to make the site/software better) right in the first 5 minutes of a call - you can establish camaraderie very quickly.
                  Followup emails (and especially a hand-written Thank You) also go a long way toward making people receptive to further testing down the road - or talking about their positive testing experience to other staff. This also helps our tech team in greasing the road for positive response for the changes coming. I've been surprised by the number of "referrals" I've gotten from my usability testers.
                  It can work fairly well, although I'll admit there are downfalls to this type of testing. WebEx CERTAINLY has bandwidth limitations - and that wreaks havoc on tests if you don't arrange your tests in a particular way.

                  On the other hand, it's usability at the cost of the call and WebEx.
                  Thanks for your responses, everyone - lots of great links!
                  Tara

                  Tara Schnaible
                  Usability Analyst
                  The Nature Conservancy

                  tschnaible@...
                  (312) 580-2359


                  --- In agile-usability@yahoogroups.com, Elizabeth Bacon <lists@...> wrote:
                  >
                  > Hey folks,
                  >
                  > I also don't have any resources covering the relative efficacy of in-
                  > person and remote usability testing. My experienced designer gut says
                  > that there's no question one would learn less from remote usability
                  > testing due to the reduced opportunity to establish camaraderie &
                  > empathy and reduced ability to closely observe every human nuance &
                  > action. Nevertheless, it's an exploding practice and offers great time
                  > & cost-based efficiencies that may well offset the reduced quality/
                  > quantity of data that is obtained.
                  >
                  > I had a need to do remote usability testing last year, so I did a
                  > comparative analysis of the various products on the market. I've
                  > published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en
                  >
                  > Enjoy! Bolt|Peters and others will be referencing this material in
                  > various forthcoming publications. Feel free to do the same, if you
                  > like, as well as letting me know if you're aware of other offerings
                  > that should go into this spreadsheet.
                  >
                  > Cheers,
                  > Liz
                  >
                  > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  > Vice-President, IxDA / www.ixda.org
                  > CDO, Devise / www.devise.com
                  > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  >
                • sanfret
                  Tara, I am interested in hearing more about the perceived bandwidth limitations of WebEx. We haven t had any issues with bandwidth related latency in our
                  Message 8 of 14 , Jan 19, 2010
                  • 0 Attachment
                    Tara, I am interested in hearing more about the perceived bandwidth limitations of WebEx. We haven't had any issues with bandwidth related latency in our hundreds/thousands of sessions. We don't have anyone on dial-up, but home broadband connects and office connections are issue free.

                    We are actually looking at pulling a paper together on our experiences using the WebEx suite in our research efforts. It really has been a nice tool to help the product teams get quick and immediate feedback from alpha and beta tester.

                    Regards,
                    Michael Lenz
                    Sr. Manager UX
                    Cisco


                    --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                    >
                    > Liz,
                    > I like your idea about breakdown of remote testing tools, I've wondered myself about who the "professional community" is for Concept Feedback.
                    > I can speak a bit to the camaraderie component of your testing - and say that if you have the right moderators conducting the test over the phone, it's not as much an issue. The physical body-language is lacking, but you can certainly hear a negative experience over a phone if you're paying attention.
                    > A bit of background might explain this further.
                    > The testing I do for my organization is largely on internal applications, with internal staff. Testers are often strangers (to the moderator) and they are never offered compensation for the hour+ it takes (we are a non-profit). If your moderator is friendly, approachable, easy-going (and focuses the call around getting an opinion in order to make the site/software better) right in the first 5 minutes of a call - you can establish camaraderie very quickly.
                    > Followup emails (and especially a hand-written Thank You) also go a long way toward making people receptive to further testing down the road - or talking about their positive testing experience to other staff. This also helps our tech team in greasing the road for positive response for the changes coming. I've been surprised by the number of "referrals" I've gotten from my usability testers.
                    > It can work fairly well, although I'll admit there are downfalls to this type of testing. WebEx CERTAINLY has bandwidth limitations - and that wreaks havoc on tests if you don't arrange your tests in a particular way.
                    >
                    > On the other hand, it's usability at the cost of the call and WebEx.
                    > Thanks for your responses, everyone - lots of great links!
                    > Tara
                    >
                    > Tara Schnaible
                    > Usability Analyst
                    > The Nature Conservancy
                    >
                    > tschnaible@...
                    > (312) 580-2359
                    >
                    >
                    > --- In agile-usability@yahoogroups.com, Elizabeth Bacon <lists@> wrote:
                    > >
                    > > Hey folks,
                    > >
                    > > I also don't have any resources covering the relative efficacy of in-
                    > > person and remote usability testing. My experienced designer gut says
                    > > that there's no question one would learn less from remote usability
                    > > testing due to the reduced opportunity to establish camaraderie &
                    > > empathy and reduced ability to closely observe every human nuance &
                    > > action. Nevertheless, it's an exploding practice and offers great time
                    > > & cost-based efficiencies that may well offset the reduced quality/
                    > > quantity of data that is obtained.
                    > >
                    > > I had a need to do remote usability testing last year, so I did a
                    > > comparative analysis of the various products on the market. I've
                    > > published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en
                    > >
                    > > Enjoy! Bolt|Peters and others will be referencing this material in
                    > > various forthcoming publications. Feel free to do the same, if you
                    > > like, as well as letting me know if you're aware of other offerings
                    > > that should go into this spreadsheet.
                    > >
                    > > Cheers,
                    > > Liz
                    > >
                    > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    > > Vice-President, IxDA / www.ixda.org
                    > > CDO, Devise / www.devise.com
                    > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    > >
                    >
                  • schntar
                    Andrew, You know - it s fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile
                    Message 9 of 14 , Jan 20, 2010
                    • 0 Attachment
                      Andrew,
                      You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.

                      We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                      Tara

                      --- In agile-usability@yahoogroups.com, "Andrew Maier" <andrew.maier@...> wrote:
                      >
                      > Hey Tara,
                      >
                      > I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                      >
                      > Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                      >
                      > However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                      >
                      > Thanks,
                      > Andrew Maier
                      > UX Booth, Editor in Chief
                      >
                      > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
                      > >
                      > > Happy New Year!
                      > >
                      > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                      > > http://rosenfeldmedia.com/books/remote-research/
                      > >
                      > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                      > >
                      > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                      > >
                      > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                      > >
                      > > I would love to hear your remote-testing experiences!
                      > >
                      > > Tara Schnaible
                      > > Usability Analyst
                      > > The Nature Conservancy
                      > >
                      > > tschnaible@
                      > >
                      > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                      > >
                      >
                    • William Pietri
                      If you re struggling with 4-week iterations like that, your iterations may be too long. One of the transitions people go through when adopting iterations is
                      Message 10 of 14 , Jan 20, 2010
                      • 0 Attachment
                        If you're struggling with 4-week iterations like that, your iterations
                        may be too long.

                        One of the transitions people go through when adopting iterations is
                        that they first do mini-waterfall within the iteration: The first few
                        days go to activity X, the next few to Y, etc.

                        The stage I see after that is staggered phases. The developers work in
                        iteration N. Other groups work in iterations N-1, N-2, N+1, etc.

                        I'm sure both of these work better than 6-18 month waterfall cycles, but
                        they still are stages I get my clients to move beyond.

                        William

                        On 01/20/2010 07:37 AM, schntar wrote:
                        > Andrew,
                        > You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.
                        >
                        > We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                        > Tara
                        >
                        > --- In agile-usability@yahoogroups.com, "Andrew Maier"<andrew.maier@...> wrote:
                        >
                        >> Hey Tara,
                        >>
                        >> I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                        >>
                        >> Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                        >>
                        >> However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                        >>
                        >> Thanks,
                        >> Andrew Maier
                        >> UX Booth, Editor in Chief
                        >>
                        >> --- In agile-usability@yahoogroups.com, "schntar"<tschnaible@> wrote:
                        >>
                        >>> Happy New Year!
                        >>>
                        >>> I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                        >>> http://rosenfeldmedia.com/books/remote-research/
                        >>>
                        >>> I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                        >>>
                        >>> I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                        >>>
                        >>> Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                        >>>
                        >>> I would love to hear your remote-testing experiences!
                        >>>
                        >>> Tara Schnaible
                        >>> Usability Analyst
                        >>> The Nature Conservancy
                        >>>
                        >>> tschnaible@
                        >>>
                        >>> "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                        >>>
                        >>>
                        >>
                        >
                        >
                        >
                        > ------------------------------------
                        >
                        > Yahoo! Groups Links
                        >
                        >
                        >
                        >
                      • Andrew Maier
                        Hey Tara, Conducting user testing throughout the development of an agile project is key. I would suggest a number of things that I ve tried: * After your team
                        Message 11 of 14 , Jan 27, 2010
                        • 0 Attachment
                          Hey Tara,

                          Conducting user testing throughout the development of an agile project is key. I would suggest a number of things that I've tried:

                          * After your team has storyboarded the flow of users throughout the site and created wireframes for the required interactions, re-engage the users with which you conducted your initial research. Ask them to briefly comment on the flow of a certain task. If you're using a digital wireframing tool, it never hurts to link screens together to provide a more realistic feeling.
                          * True agile development generates working code at every step of the way, so it shouldn't be too hard to run the application by those same test users on a weekly basis––at least in theory.
                          * Don't "design" the application too much up front. Considerations such as color schemes, logo placement, etc are (for the most part) ancillary to the perceived usability of your application. On the most successful projects in which I've worked, I did the front-end development in such a way that the web application provided affordances for functionality, but it wasn't necessarily beautiful until the final stages. Allowing the team to turn on a dime should the need arise.

                          Hope that helps, let me know if you think differently!
                          -Andrew


                          --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                          >
                          > Andrew,
                          > You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.
                          >
                          > We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                          > Tara
                          >
                          > --- In agile-usability@yahoogroups.com, "Andrew Maier" <andrew.maier@> wrote:
                          > >
                          > > Hey Tara,
                          > >
                          > > I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                          > >
                          > > Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                          > >
                          > > However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                          > >
                          > > Thanks,
                          > > Andrew Maier
                          > > UX Booth, Editor in Chief
                          > >
                          > > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
                          > > >
                          > > > Happy New Year!
                          > > >
                          > > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                          > > > http://rosenfeldmedia.com/books/remote-research/
                          > > >
                          > > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                          > > >
                          > > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                          > > >
                          > > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                          > > >
                          > > > I would love to hear your remote-testing experiences!
                          > > >
                          > > > Tara Schnaible
                          > > > Usability Analyst
                          > > > The Nature Conservancy
                          > > >
                          > > > tschnaible@
                          > > >
                          > > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                          > > >
                          > >
                          >
                        Your message has been successfully submitted and would be delivered to recipients shortly.