Loading ...
Sorry, an error occurred while loading the content.

Re: Remote Usability Testing

Expand Messages
  • timkieschnick
    Julie Melton presented on the topic of remote usability just a couple of days ago at BayCHI , and I asked this very question. No one
    Message 1 of 14 , Jan 14, 2010
    • 0 Attachment

      Julie Melton presented on the topic of remote usability just a couple of days ago at BayCHI , and I asked this very question.  No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog  in about two weeks (end of January).

       


      --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
      >
      > Happy New Year!
      >
      > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
      > http://rosenfeldmedia.com/books/remote-research/
      >
      > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
      >
      > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
      >
      > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
      >
      > I would love to hear your remote-testing experiences!
      >
      > Tara Schnaible
      > Usability Analyst
      > The Nature Conservancy
      >
      > tschnaible@...
      >
      > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
      >

    • chris chandler
      If you haven t already, check out: http://boltpeters.com/ They re the ones writing the forthcoming book, and there are case studies, videos and links to
      Message 2 of 14 , Jan 14, 2010
      • 0 Attachment
        If you haven't already, check out: http://boltpeters.com/

        They're the ones writing the forthcoming book, and there are case studies, videos and links to published articles on their site.

        I haven't worked directly with them, but I have seen Nate Bolt present findings and case studies at a couple of conferences (UPA and IA Summit) and think they are on to something.

        -cc



        On Thu, Jan 14, 2010 at 12:09 PM, timkieschnick <tim.kieschnick@...> wrote:
         

        Julie Melton presented on the topic of remote usability just a couple of days ago at BayCHI , and I asked this very question.  No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog  in about two weeks (end of January).

         


        --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
        >
        > Happy New Year!
        >
        > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
        > http://rosenfeldmedia.com/books/remote-research/
        >
        > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
        >
        > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
        >
        > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
        >
        > I would love to hear your remote-testing experiences!
        >
        > Tara Schnaible
        > Usability Analyst
        > The Nature Conservancy
        >
        > tschnaible@...

        >
        > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
        >

      • matyeo
        For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now. In part, it
        Message 3 of 14 , Jan 15, 2010
        • 0 Attachment
          For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now.

          In part, it reflects our frequent need to test across Canada, the US and internationally, at low cost. Another significant benefit is reduction in the cost of reruiting, but the benefits to the client organization are paramount - more observers from all levels of the organization, more buy-in, more drive for organizational change.

          We described these in one of our newsletter articles:

          http://www.neoinsight.com/newsletter/0706.html#RemoteUsability

          Cheers.
          Mike Atyeo
          Neo Insight

          --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
          >
          > Happy New Year!
          >
          > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
          > http://rosenfeldmedia.com/books/remote-research/
          >
          > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
          >
          > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
          >
          > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
          >
          > I would love to hear your remote-testing experiences!
          >
          > Tara Schnaible
          > Usability Analyst
          > The Nature Conservancy
          >
          > tschnaible@...
          >
          > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
          >
        • Elizabeth Bacon
          Hey folks, I also don t have any resources covering the relative efficacy of in- person and remote usability testing. My experienced designer gut says that
          Message 4 of 14 , Jan 15, 2010
          • 0 Attachment
            Hey folks,

            I also don't have any resources covering the relative efficacy of in-person and remote usability testing. My experienced designer gut says that there's no question one would learn less from remote usability testing due to the reduced opportunity to establish camaraderie & empathy and reduced ability to closely observe every human nuance & action. Nevertheless, it's an exploding practice and offers great time & cost-based efficiencies that may well offset the reduced quality/quantity of data that is obtained. 

            I had a need to do remote usability testing last year, so I did a comparative analysis of the various products on the market. I've published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en

            Enjoy! Bolt|Peters and others will be referencing this material in various forthcoming publications. Feel free to do the same, if you like, as well as letting me know if you're aware of other offerings that should go into this spreadsheet. 

            Cheers,
            Liz

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Vice-President, IxDA / www.ixda.org
            CDO, Devise / www.devise.com
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~





          • jscottux
            There was an entire issue of UPA s User Experience magazine devoted to remote testing. You can read it here:
            Message 5 of 14 , Jan 16, 2010
            • 0 Attachment
              There was an entire issue of UPA's User Experience magazine devoted to remote testing. You can read it here:

              http://www.usabilityprofessionals.org/upa_publications/user_experience/past_issues/2008-3.html

              I also liked this article in Boxes and Arrows:

              http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it


              I'm with Mike. If we don't go to the participant, we usually have remote testing as part of the mix. Our pool of participants is too small without remote or travel or both. Our lab has morphed into a setup designed for remote testing as well as in lab participants. With that setup, I am able to gather the same rich computer data from Morae that I get with participants in the lab, which makes analysis far easier and, in our Agile world, faster.

              There are, in my opinion, a few things to watch out for:

              -- Lag. Occasionally, lag will cause a different experience for your participant if you share with them. If they share with you, the lag will cause you to observe an activity later than it actually happens. Both can be problematic for your analysis.
              -- Nonverbals. Unless your participants are able to have a webcam, you will need to gather some of your satisfaction data more explicitly. We will ask more frequently for reactions (debrief, words to describe, scaled reaction questions) to help mitigate that we can't see them saying "oh it was easy" when their face clearly shows it wasn't.
              --Observers. Certain remote setups may not allow observers. In Agile, your team needs have their skin in the game here as well; observing a few participants is a part of that. We work to make sure that our folks can see testing as it happens, despite the setup. They have to bring their observations to the findings meeting later.
              --Remote setup. Some of the tools available to us have a better user experience than others. Don't hesitate to connect yourself as a participant, and understand what your participant will see. Your testing results can be confounded by bad remote applications.

              I think we can say professionally that remote testing is firmly established and a respected method. Have at it and have fun.


              Cheers,
              Josie Scott
              TechSmith UX



              --- In agile-usability@yahoogroups.com, "matyeo" <mike@...> wrote:
              >
              >
              >
              > For us, the advantages of remote testing far outweigh the disadvantages in most situations. We only do on-site usability testing occasionally now.
              >
              > In part, it reflects our frequent need to test across Canada, the US and internationally, at low cost. Another significant benefit is reduction in the cost of reruiting, but the benefits to the client organization are paramount - more observers from all levels of the organization, more buy-in, more drive for organizational change.
              >
              > We described these in one of our newsletter articles:
              >
              > http://www.neoinsight.com/newsletter/0706.html#RemoteUsability
              >
              > Cheers.
              > Mike Atyeo
              > Neo Insight
              >
              > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
              > >
              > > Happy New Year!
              > >
              > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
              > > http://rosenfeldmedia.com/books/remote-research/
              > >
              > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
              > >
              > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
              > >
              > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
              > >
              > > I would love to hear your remote-testing experiences!
              > >
              > > Tara Schnaible
              > > Usability Analyst
              > > The Nature Conservancy
              > >
              > > tschnaible@
              > >
              > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
              > >
              >
            • Jared Spool
              Well, they aren t paying attention. I d look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I d also look to
              Message 6 of 14 , Jan 17, 2010
              • 0 Attachment
                Well, they aren't paying attention.

                I'd look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I'd also look to work that Rolf Molich is doing, particularly around CUE-8.

                Jared

                On Jan 14, 2010, at 1:09 PM, timkieschnick wrote:


                Julie Melton presented on the topic of remote usability just a couple of days ago at BayCHI , and I asked this very question.  No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog  in about two weeks (end of January).

                 


                --- In agile-usability@ yahoogroups. com, "schntar" <tschnaible@. ..> wrote:
                >
                > Happy New Year!
                > 
                > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                > http://rosenfeldmed ia.com/books/ remote-research/
                > 
                > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                > 
                > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/ etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussi on - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                > 
                > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                > 
                > I would love to hear your remote-testing experiences!
                > 
                > Tara Schnaible
                > Usability Analyst
                > The Nature Conservancy
                > 
                > tschnaible@. ..
                > 
                > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                >



              • Hernandez, Barbara
                here s a few I point folks to: An article from the ACM - http://portal.acm.org/citation.cfm?id=985921.986018 A comparison of synchronous remote and local
                Message 7 of 14 , Jan 17, 2010
                • 0 Attachment
                  here's a few I point folks to:

                  An article from the ACM - http://portal.acm.org/citation.cfm?id=985921.986018
                  A comparison of synchronous remote and local usability studies for an expert interface
                  Synchronous remote usability studies can be a convenient and cost-effective alternative to conventional local usability studies. Although they are common in the field, there has been little research comparing synchronous remote usability studies with local studies. In our comparison of remote and local studies of an expert interface, the primary differences were in the participant's and facilitator's qualitative experience. The remote and local studies agreed closely (with no significant differences) in terms of the number of usability issues found, their type, and their severity. While our comparison focuses on an expert interface and more work is needed to understand remote studies in general, our experience suggests that evaluators of expert interfaces will have comparable success identifying usability issues with either remote or local studies.

                  bolt | peters also have an article - http://boltpeters.com/articles/versus.html

                  We did a study for World Usability Day in 2005 - with similar results. I can send a copy along once I am back in the office in the morning.

                  FWIW - we do tons of remote testing. We find that the data we get is in par with in-person testing, plus we get a peak at the users own personal environment and their computer setup. Pretty cool stuff. we have even done remote ethnography.

                  Our biggest drawback is that people are more likely to bail - as in not show up for the session - with remote. There are also some challenges with technology - some tools work on the Mac some don't, some record the screen some don't, etc.

                  We couldn't engage our users effectively without it.

                  Regards
                  Barb Hernandez
                  User Experience Manager | TechSmith Corporation
                  ________________________________________
                  From: agile-usability@yahoogroups.com [agile-usability@yahoogroups.com] On Behalf Of Jared Spool [jspool@...]
                  Sent: Sunday, January 17, 2010 7:30 PM
                  To: agile-usability@yahoogroups.com
                  Subject: Re: [agile-usability] Re: Remote Usability Testing

                  Well, they aren't paying attention.

                  I'd look for the work of Tom Tullis, who has done several comparisons of remote testing to lab testing. I'd also look to work that Rolf Molich is doing, particularly around CUE-8.

                  Jared

                  On Jan 14, 2010, at 1:09 PM, timkieschnick wrote:



                  Julie Melton presented on the topic of remote usability just a c ouple of days ago at BayCHI<http://www.baychi.org/> , and I asked this very question. No one in the room of over 100 human factors folks could point me to any published research, but Julie said she's just finished doing a comparative study of remote vs. in-person results, and she plans to publish it on her blog <http://juliemelton.com/> in about two weeks (end of January).



                  --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                  >
                  > Happy New Year!
                  >
                  > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                  > http://rosenfeldmedia.com/books/remote-research/
                  >
                  > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                  >
                  > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                  >
                  > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                  >
                  > I would love to hear your remote-testing experiences!
                  >
                  > Tara Schnaible
                  > Usability Analyst
                  > The Nature Conservancy
                  >
                  > tschnaible@...
                  >
                  > "If you k now a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                  >
                • Andrew Maier
                  Hey Tara, I don t have much experience in the way of user testing in general, but I m also interested in the answer to this question. Before the agile
                  Message 8 of 14 , Jan 18, 2010
                  • 0 Attachment
                    Hey Tara,

                    I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.

                    Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.

                    However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.

                    Thanks,
                    Andrew Maier
                    UX Booth, Editor in Chief

                    --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                    >
                    > Happy New Year!
                    >
                    > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                    > http://rosenfeldmedia.com/books/remote-research/
                    >
                    > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                    >
                    > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                    >
                    > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                    >
                    > I would love to hear your remote-testing experiences!
                    >
                    > Tara Schnaible
                    > Usability Analyst
                    > The Nature Conservancy
                    >
                    > tschnaible@...
                    >
                    > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                    >
                  • schntar
                    Liz, I like your idea about breakdown of remote testing tools, I ve wondered myself about who the professional community is for Concept Feedback. I can speak
                    Message 9 of 14 , Jan 19, 2010
                    • 0 Attachment
                      Liz,
                      I like your idea about breakdown of remote testing tools, I've wondered myself about who the "professional community" is for Concept Feedback.
                      I can speak a bit to the camaraderie component of your testing - and say that if you have the right moderators conducting the test over the phone, it's not as much an issue. The physical body-language is lacking, but you can certainly hear a negative experience over a phone if you're paying attention.
                      A bit of background might explain this further.
                      The testing I do for my organization is largely on internal applications, with internal staff. Testers are often strangers (to the moderator) and they are never offered compensation for the hour+ it takes (we are a non-profit). If your moderator is friendly, approachable, easy-going (and focuses the call around getting an opinion in order to make the site/software better) right in the first 5 minutes of a call - you can establish camaraderie very quickly.
                      Followup emails (and especially a hand-written Thank You) also go a long way toward making people receptive to further testing down the road - or talking about their positive testing experience to other staff. This also helps our tech team in greasing the road for positive response for the changes coming. I've been surprised by the number of "referrals" I've gotten from my usability testers.
                      It can work fairly well, although I'll admit there are downfalls to this type of testing. WebEx CERTAINLY has bandwidth limitations - and that wreaks havoc on tests if you don't arrange your tests in a particular way.

                      On the other hand, it's usability at the cost of the call and WebEx.
                      Thanks for your responses, everyone - lots of great links!
                      Tara

                      Tara Schnaible
                      Usability Analyst
                      The Nature Conservancy

                      tschnaible@...
                      (312) 580-2359


                      --- In agile-usability@yahoogroups.com, Elizabeth Bacon <lists@...> wrote:
                      >
                      > Hey folks,
                      >
                      > I also don't have any resources covering the relative efficacy of in-
                      > person and remote usability testing. My experienced designer gut says
                      > that there's no question one would learn less from remote usability
                      > testing due to the reduced opportunity to establish camaraderie &
                      > empathy and reduced ability to closely observe every human nuance &
                      > action. Nevertheless, it's an exploding practice and offers great time
                      > & cost-based efficiencies that may well offset the reduced quality/
                      > quantity of data that is obtained.
                      >
                      > I had a need to do remote usability testing last year, so I did a
                      > comparative analysis of the various products on the market. I've
                      > published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en
                      >
                      > Enjoy! Bolt|Peters and others will be referencing this material in
                      > various forthcoming publications. Feel free to do the same, if you
                      > like, as well as letting me know if you're aware of other offerings
                      > that should go into this spreadsheet.
                      >
                      > Cheers,
                      > Liz
                      >
                      > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      > Vice-President, IxDA / www.ixda.org
                      > CDO, Devise / www.devise.com
                      > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      >
                    • sanfret
                      Tara, I am interested in hearing more about the perceived bandwidth limitations of WebEx. We haven t had any issues with bandwidth related latency in our
                      Message 10 of 14 , Jan 19, 2010
                      • 0 Attachment
                        Tara, I am interested in hearing more about the perceived bandwidth limitations of WebEx. We haven't had any issues with bandwidth related latency in our hundreds/thousands of sessions. We don't have anyone on dial-up, but home broadband connects and office connections are issue free.

                        We are actually looking at pulling a paper together on our experiences using the WebEx suite in our research efforts. It really has been a nice tool to help the product teams get quick and immediate feedback from alpha and beta tester.

                        Regards,
                        Michael Lenz
                        Sr. Manager UX
                        Cisco


                        --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                        >
                        > Liz,
                        > I like your idea about breakdown of remote testing tools, I've wondered myself about who the "professional community" is for Concept Feedback.
                        > I can speak a bit to the camaraderie component of your testing - and say that if you have the right moderators conducting the test over the phone, it's not as much an issue. The physical body-language is lacking, but you can certainly hear a negative experience over a phone if you're paying attention.
                        > A bit of background might explain this further.
                        > The testing I do for my organization is largely on internal applications, with internal staff. Testers are often strangers (to the moderator) and they are never offered compensation for the hour+ it takes (we are a non-profit). If your moderator is friendly, approachable, easy-going (and focuses the call around getting an opinion in order to make the site/software better) right in the first 5 minutes of a call - you can establish camaraderie very quickly.
                        > Followup emails (and especially a hand-written Thank You) also go a long way toward making people receptive to further testing down the road - or talking about their positive testing experience to other staff. This also helps our tech team in greasing the road for positive response for the changes coming. I've been surprised by the number of "referrals" I've gotten from my usability testers.
                        > It can work fairly well, although I'll admit there are downfalls to this type of testing. WebEx CERTAINLY has bandwidth limitations - and that wreaks havoc on tests if you don't arrange your tests in a particular way.
                        >
                        > On the other hand, it's usability at the cost of the call and WebEx.
                        > Thanks for your responses, everyone - lots of great links!
                        > Tara
                        >
                        > Tara Schnaible
                        > Usability Analyst
                        > The Nature Conservancy
                        >
                        > tschnaible@...
                        > (312) 580-2359
                        >
                        >
                        > --- In agile-usability@yahoogroups.com, Elizabeth Bacon <lists@> wrote:
                        > >
                        > > Hey folks,
                        > >
                        > > I also don't have any resources covering the relative efficacy of in-
                        > > person and remote usability testing. My experienced designer gut says
                        > > that there's no question one would learn less from remote usability
                        > > testing due to the reduced opportunity to establish camaraderie &
                        > > empathy and reduced ability to closely observe every human nuance &
                        > > action. Nevertheless, it's an exploding practice and offers great time
                        > > & cost-based efficiencies that may well offset the reduced quality/
                        > > quantity of data that is obtained.
                        > >
                        > > I had a need to do remote usability testing last year, so I did a
                        > > comparative analysis of the various products on the market. I've
                        > > published this for all to take advantage of at: http://spreadsheets.google.com/ccc?key=0AiUefdff4crxcGMtTkFlRExtWUZ2ZUNSbEllak51RkE&hl=en
                        > >
                        > > Enjoy! Bolt|Peters and others will be referencing this material in
                        > > various forthcoming publications. Feel free to do the same, if you
                        > > like, as well as letting me know if you're aware of other offerings
                        > > that should go into this spreadsheet.
                        > >
                        > > Cheers,
                        > > Liz
                        > >
                        > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                        > > Vice-President, IxDA / www.ixda.org
                        > > CDO, Devise / www.devise.com
                        > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                        > >
                        >
                      • schntar
                        Andrew, You know - it s fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile
                        Message 11 of 14 , Jan 20, 2010
                        • 0 Attachment
                          Andrew,
                          You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.

                          We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                          Tara

                          --- In agile-usability@yahoogroups.com, "Andrew Maier" <andrew.maier@...> wrote:
                          >
                          > Hey Tara,
                          >
                          > I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                          >
                          > Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                          >
                          > However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                          >
                          > Thanks,
                          > Andrew Maier
                          > UX Booth, Editor in Chief
                          >
                          > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
                          > >
                          > > Happy New Year!
                          > >
                          > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                          > > http://rosenfeldmedia.com/books/remote-research/
                          > >
                          > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                          > >
                          > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                          > >
                          > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                          > >
                          > > I would love to hear your remote-testing experiences!
                          > >
                          > > Tara Schnaible
                          > > Usability Analyst
                          > > The Nature Conservancy
                          > >
                          > > tschnaible@
                          > >
                          > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                          > >
                          >
                        • William Pietri
                          If you re struggling with 4-week iterations like that, your iterations may be too long. One of the transitions people go through when adopting iterations is
                          Message 12 of 14 , Jan 20, 2010
                          • 0 Attachment
                            If you're struggling with 4-week iterations like that, your iterations
                            may be too long.

                            One of the transitions people go through when adopting iterations is
                            that they first do mini-waterfall within the iteration: The first few
                            days go to activity X, the next few to Y, etc.

                            The stage I see after that is staggered phases. The developers work in
                            iteration N. Other groups work in iterations N-1, N-2, N+1, etc.

                            I'm sure both of these work better than 6-18 month waterfall cycles, but
                            they still are stages I get my clients to move beyond.

                            William

                            On 01/20/2010 07:37 AM, schntar wrote:
                            > Andrew,
                            > You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.
                            >
                            > We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                            > Tara
                            >
                            > --- In agile-usability@yahoogroups.com, "Andrew Maier"<andrew.maier@...> wrote:
                            >
                            >> Hey Tara,
                            >>
                            >> I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                            >>
                            >> Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                            >>
                            >> However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                            >>
                            >> Thanks,
                            >> Andrew Maier
                            >> UX Booth, Editor in Chief
                            >>
                            >> --- In agile-usability@yahoogroups.com, "schntar"<tschnaible@> wrote:
                            >>
                            >>> Happy New Year!
                            >>>
                            >>> I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                            >>> http://rosenfeldmedia.com/books/remote-research/
                            >>>
                            >>> I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                            >>>
                            >>> I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                            >>>
                            >>> Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                            >>>
                            >>> I would love to hear your remote-testing experiences!
                            >>>
                            >>> Tara Schnaible
                            >>> Usability Analyst
                            >>> The Nature Conservancy
                            >>>
                            >>> tschnaible@
                            >>>
                            >>> "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                            >>>
                            >>>
                            >>
                            >
                            >
                            >
                            > ------------------------------------
                            >
                            > Yahoo! Groups Links
                            >
                            >
                            >
                            >
                          • Andrew Maier
                            Hey Tara, Conducting user testing throughout the development of an agile project is key. I would suggest a number of things that I ve tried: * After your team
                            Message 13 of 14 , Jan 27, 2010
                            • 0 Attachment
                              Hey Tara,

                              Conducting user testing throughout the development of an agile project is key. I would suggest a number of things that I've tried:

                              * After your team has storyboarded the flow of users throughout the site and created wireframes for the required interactions, re-engage the users with which you conducted your initial research. Ask them to briefly comment on the flow of a certain task. If you're using a digital wireframing tool, it never hurts to link screens together to provide a more realistic feeling.
                              * True agile development generates working code at every step of the way, so it shouldn't be too hard to run the application by those same test users on a weekly basis––at least in theory.
                              * Don't "design" the application too much up front. Considerations such as color schemes, logo placement, etc are (for the most part) ancillary to the perceived usability of your application. On the most successful projects in which I've worked, I did the front-end development in such a way that the web application provided affordances for functionality, but it wasn't necessarily beautiful until the final stages. Allowing the team to turn on a dime should the need arise.

                              Hope that helps, let me know if you think differently!
                              -Andrew


                              --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@...> wrote:
                              >
                              > Andrew,
                              > You know - it's fantastically interesting that you bring up Agile usability/UX testing every other iteration. When my organization migrated to Agile for projects we struggled to fit formal usability and QA into our 2-to-4 week iterations. The formal process currently indicates that we do staggered usability testing (aka, we test the previous iteration's developed code during the end/beginning of the adjacent iteration, then we have use a few days at the end of the iteration to resolve issues in developed code). It's graceless, to be honest, and difficult to bring a team back in time to the work they're no longer focused on. It's do-able, but it doesn't work well and you have to strongly consider which battles you'll fight because short of adding additional iterations to your schedule you simply don't have the time to re-develop for issues.
                              >
                              > We are currently looking a lot harder at prototype/mock-up testing in the planning phase of the iteration (aka, before it begins) as a way to try and address UI issues upfront. I've come to think that this is one of the only ways formal usability testing can really benefit an Agile project because you have more time. Another possibility I've considered is doing much more ad-hoc testing while the code is in development (however this is tricky for my group - we're all distributed and developers don't really share their code well until it's done). Have any good suggestions/modifications on these trials?
                              > Tara
                              >
                              > --- In agile-usability@yahoogroups.com, "Andrew Maier" <andrew.maier@> wrote:
                              > >
                              > > Hey Tara,
                              > >
                              > > I don't have much experience in the way of user testing in general, but I'm also interested in the answer to this question.
                              > >
                              > > Before the agile development projects I've worked on, I've conducted research to solidify what mental models users groups of our website have and where inside of those our solution will live. That's just done by asking clients about their target audience, doing competitive analysis, etc, and then conducting non-directed user interviews.
                              > >
                              > > However, during the application development process, I've looked into (but haven't tried) interviewing users on every other iteration, so as to test new features/functionality throughout the development process. I'd be interested to hear from anyone who has done bi-weekly (twice a month) remote usability tests on products being developed; how they handled that and how valuable it was.
                              > >
                              > > Thanks,
                              > > Andrew Maier
                              > > UX Booth, Editor in Chief
                              > >
                              > > --- In agile-usability@yahoogroups.com, "schntar" <tschnaible@> wrote:
                              > > >
                              > > > Happy New Year!
                              > > >
                              > > > I recently heard of a book (to be published in 2010) that deals with remote usability testing:
                              > > > http://rosenfeldmedia.com/books/remote-research/
                              > > >
                              > > > I am especially curious if anyone knows of studies that highlight the tangible differences in usability test findings between remote usability tests and lab tests (maybe something from SURL, etc?).
                              > > >
                              > > > I work in an environment where most of my testers are long-distance and 95% of my testing occurs through phone/web conf./skype/etc. We gather a lot of good data with these methods (and I have rarely seen wildly different usability results come out in-person tests vs. remote tests). However this seems to be an area of testing that doesn't have much literature/discussion - and I wonder if it's turf that's already been covered (and possibly rejected)? Maybe this is a revolution that would put too many testing labs out of business (hah)?
                              > > >
                              > > > Have others had good/bad experiences with this form of testing - and might this also be an effective way to handle the sort of lightweight testing that could be incorporated into Agile projects? I've worked with this very idea a bit (with some mixed results) but it involves adding Agile iterations on to handle post-Dev. issue fixing. (inefficient, but it works for those in projects that don't do mock-ups or mock-up testing)
                              > > >
                              > > > I would love to hear your remote-testing experiences!
                              > > >
                              > > > Tara Schnaible
                              > > > Usability Analyst
                              > > > The Nature Conservancy
                              > > >
                              > > > tschnaible@
                              > > >
                              > > > "If you know a thing only qualitatively, you know it no more than vaguely. If you know it quantitatively -- grasping some numerical measure that distinguishes it from an infinite number of other possibilities -- you are beginning to know it deeply. You comprehend some of its beauty and you gain access to its power and the understanding it provides." -- Carl Sagan
                              > > >
                              > >
                              >
                            Your message has been successfully submitted and would be delivered to recipients shortly.