Loading ...
Sorry, an error occurred while loading the content.

Re: [agile-usability] Usability Test Goals

Expand Messages
  • William Pietri
    ... I d agree. Every new version is a collection of gambles, and you get feedback to see which ones are good, which could be improved, and hints on how to
    Message 1 of 9 , Aug 30, 2005
    • 0 Attachment
      On Tue, 2005-08-30 at 19:35 +0000, bunkybrian wrote:
      > I just saw a usability test where the practitioners were Information
      > Architects. One of the comments in this test was "the goal of this
      > usability test is to validate my new design."
      >
      > From some reason, I have a problem with this statement. It seems
      > like the goal of the usability test should be to obtain customer
      > feedback about a new design.

      I'd agree.

      Every new version is a collection of gambles, and you get feedback to
      see which ones are good, which could be improved, and hints on how to
      improve. If you're only looking for validation, that misses much of the
      point.

      Do you know what motivates this attitude for these people?

      I'd also wonder how agile this project is. To me one of the most
      insidious things about the BDUF approach is that any honest person has
      to admit that a big spec involves a lot of guesses. But people who fund
      big software projects are rarely comfortable dropping hundreds of
      thousands of dollars on guesswork; they want certainty. Especially in
      competitive situations, this encourages designers to express more
      confidence than is warranted.

      I think a natural corollary of that is seeking only validation, not
      feedback. If somebody is busy insisting that the design is great, it's
      hard to honestly look for weaknesses.

      William

      --
      William Pietri <william@...>
    • Ash Donaldson
      ... Blatantly biasing the participant from the beginning. ... The goal of a usability test is to evaluate a proposed hypothesis. This is why it is recommended
      Message 2 of 9 , Aug 30, 2005
      • 0 Attachment
        On 31/08/2005, at 5:35 AM, bunkybrian wrote:
        I just saw a usability test where the practitioners were Information
        Architects.  One of the comments in this test was "the goal of this
        usability test is to validate my new design."

        Blatantly biasing the participant from the beginning.

        From some reason, I have a problem with this statement.  It seems
        like the goal of the usability test should be to obtain customer
        feedback about a new design.

        The goal of a usability test is to evaluate a proposed hypothesis.  This is why it is recommended that an objective third party, NOT the designer test the validity of any proposed designs.

        In my mind, the usability practioner is already starting out with
        the mindset that their new design is correct.  The design just
        needed to be validated by the user.

        This 'practitioner' sounds like one of the cowboys that half-read a couple of books and just started calling themselves an Information Architect (and charging accordingly).

        In addition, there were leading questions throughout the sessions
        that directed the user towards a conclusion rather than having the
        user give unsolicitated feedback.

        No. No. No!

        As you can tell, I am somewhat disappointed in this test session.  I
        wanted to get some feedback from this group as to whether the goal
        of usability testing is to validate a new design versus obtaining
        feedback from the user or something else.

        Understandably so .   I'd be advising the Information Architect's client to get their money back and make sure that they and their colleagues never use that IA or company again.  Before contracting user-centred design service like IA or usability, clients really should look for someone with some sort of qualifications (HCI, Human Factors, Social or Cog. Psych.) or at the very least, industry recommendations.


        Best regards,


        Ash Donaldson

        OZCHI 2005 Conference Chair

        chair@...


        OZCHI 2005

        Citizens Online: Considerations for today & the future

        www.ozchi.org




      • Donna Maurer
        ... Yes, that s bad. A more usual goal would be to identify whether usability issues exist that would cause people difficulty in achieving what they need to
        Message 3 of 9 , Aug 31, 2005
        • 0 Attachment
          On 30 Aug 2005 at 19:35, bunkybrian wrote:

          > Hello --
          >
          > I just saw a usability test where the practitioners were Information
          > Architects. One of the comments in this test was "the goal of this
          > usability test is to validate my new design."
          >
          > >From some reason, I have a problem with this statement. It seems
          > like the goal of the usability test should be to obtain customer
          > feedback about a new design.

          Yes, that's bad. A more usual goal would be to identify whether usability issues exist
          that would cause people difficulty in achieving what they need to do. This is less likely
          to cause bias.

          But the 'information architect' role here is irrelevant. Some IAs have good usability
          testing skills, some wouldn't know a good usability test if it jumped out and bit them.
          Usability testing skills aren't necessary for IA any more than they are for business
          analysis or graphic design. Just one more skillset that some people have and some
          don't.

          Donna
          --
          Donna Maurer
          Maadmob Interaction Design

          e: donna@...
          work: http://maadmob.com.au/
          blog: http://maadmob.net/donna/blog/
          AOL IM: maadmob
        • Ron Jeffries
          ... I m with you. While we certainly hope that some new design is a good one, it seems that a more unbiased outlook could only be helpful. Ron Jeffries
          Message 4 of 9 , Aug 31, 2005
          • 0 Attachment
            On Tuesday, August 30, 2005, at 3:35:06 PM, bunkybrian wrote:

            > As you can tell, I am somewhat disappointed in this test session. I
            > wanted to get some feedback from this group as to whether the goal
            > of usability testing is to validate a new design versus obtaining
            > feedback from the user or something else.

            I'm with you. While we certainly hope that some new design is a good
            one, it seems that a more unbiased outlook could only be helpful.

            Ron Jeffries
            www.XProgramming.com
            Here is Edward Bear, coming downstairs now, bump, bump, bump, on the back
            of his head. It is, as far as he knows, the only way of coming downstairs,
            but sometimes he feels that there really is another way, if only he could
            stop bumping for a moment and think of it. And then he feels that perhaps
            there isn't. -- A. A. Milne
          • bunkybrian
            Will -- This project is an agile one. Personally, I am of the philosophy that processes do not work. It is people that do the work (which is one of the
            Message 5 of 9 , Aug 31, 2005
            • 0 Attachment
              Will --

              This project is an agile one.

              Personally, I am of the philosophy that processes do not work. It
              is people that do the work (which is one of the principles defined
              by the Agile Manifesto).

              I have since found out that a Third Party firm is conducting the
              Usability Sessions, but this firm is primarily focused in the area
              of Market Research rather than Usability.

              Some of the leading questions might have been that the Third Party
              firm is just trying to please their customer. Of course, a lack of
              usability expertise is also a factor here.

              To me, it appears that Usability is just a checkpoint for this team,
              which is truly sad. We do have a Human Factors Center that could be
              used at my firm, which I am the Usability Lead.

              Politics is preventing this team from using us.

              Anyway, thanks for the feedback. I appreciate everyone's comments.

              Bye,
              Brian

              --- In agile-usability@yahoogroups.com, William Pietri
              <william@s...> wrote:
              > On Tue, 2005-08-30 at 19:35 +0000, bunkybrian wrote:
              > > I just saw a usability test where the practitioners were
              Information
              > > Architects. One of the comments in this test was "the goal of
              this
              > > usability test is to validate my new design."
              > >
              > > From some reason, I have a problem with this statement. It
              seems
              > > like the goal of the usability test should be to obtain customer
              > > feedback about a new design.
              >
              > I'd agree.
              >
              > Every new version is a collection of gambles, and you get feedback
              to
              > see which ones are good, which could be improved, and hints on how
              to
              > improve. If you're only looking for validation, that misses much
              of the
              > point.
              >
              > Do you know what motivates this attitude for these people?
              >
              > I'd also wonder how agile this project is. To me one of the most
              > insidious things about the BDUF approach is that any honest person
              has
              > to admit that a big spec involves a lot of guesses. But people who
              fund
              > big software projects are rarely comfortable dropping hundreds of
              > thousands of dollars on guesswork; they want certainty. Especially
              in
              > competitive situations, this encourages designers to express more
              > confidence than is warranted.
              >
              > I think a natural corollary of that is seeking only validation, not
              > feedback. If somebody is busy insisting that the design is great,
              it's
              > hard to honestly look for weaknesses.
              >
              > William
              >
              > --
              > William Pietri <william@s...>
            • Jon Meads
              Brian, What a lot of people don t realize is that good usability testing requires (1) Designing the test to answer specific questions about the usability of
              Message 6 of 9 , Aug 31, 2005
              • 0 Attachment
                Brian,

                What a lot of people don't realize is that good usability testing requires
                (1) Designing the test to answer specific questions about the usability of
                the product and (2) A good facilitator who understands how to conduct the
                test without biasing the test participant.

                I've seen an awful lot of "usability tests" done by untrained people that
                are truly awful - they tend to range from demos ("Here's what you can do.
                What do you think of that?") to accolade seeking affirmations such as the
                one you mention ("Do you like my baby? Ain't she beautiful?").

                Good usability testing, like good system testing, requires someone who
                understands what they are doing and how to do it right.

                Cheers,
                jon

                -----Original Message-----
                From: agile-usability@yahoogroups.com
                [mailto:agile-usability@yahoogroups.com] On Behalf Of bunkybrian
                Sent: Tuesday, August 30, 2005 12:35 PM
                To: agile-usability@yahoogroups.com
                Subject: [agile-usability] Usability Test Goals

                Hello --

                I just saw a usability test where the practitioners were Information
                Architects. One of the comments in this test was "the goal of this
                usability test is to validate my new design."

                >From some reason, I have a problem with this statement. It seems
                like the goal of the usability test should be to obtain customer feedback
                about a new design.

                In my mind, the usability practioner is already starting out with the
                mindset that their new design is correct. The design just needed to be
                validated by the user.

                In addition, there were leading questions throughout the sessions that
                directed the user towards a conclusion rather than having the user give
                unsolicitated feedback.

                As you can tell, I am somewhat disappointed in this test session. I wanted
                to get some feedback from this group as to whether the goal of usability
                testing is to validate a new design versus obtaining feedback from the user
                or something else.

                Thanks,
                Brian Sullivan







                ------------------------ Yahoo! Groups Sponsor --------------------~--> Get
                Bzzzy! (real tools to help you find a job). Welcome to the Sweet Life.
                http://us.click.yahoo.com/A77XvD/vlQLAA/TtwFAA/dpFolB/TM
                --------------------------------------------------------------------~->


                Yahoo! Groups Links
              • William Pietri
                ... Indeed! I agree completely. I m sad to hear that they ve gone to the trouble of following a process without embracing the spirit behind it. That s a good
                Message 7 of 9 , Aug 31, 2005
                • 0 Attachment
                  On Wed, 2005-08-31 at 14:20 +0000, bunkybrian wrote:
                  > Personally, I am of the philosophy that processes do not work.[...]
                  > To me, it appears that Usability is just a checkpoint for this team,
                  > which is truly sad.

                  Indeed! I agree completely. I'm sad to hear that they've gone to the
                  trouble of following a process without embracing the spirit behind it.
                  That's a good way to do a lot of work for relatively little benefit.

                  Hopefully, though, the practice of frequent releases will help. If they
                  keep hearing from their users that the thing is a pain, maybe they'll do
                  their usability testing more frequently and more sincerely.

                  William

                  --
                  William Pietri <william@...>
                Your message has been successfully submitted and would be delivered to recipients shortly.