Loading ...
Sorry, an error occurred while loading the content.
 

RE: {Possible Spam?} [agile-usability] Re: just one bug's enough to make a program useless

Expand Messages
  • Desilets, Alain
    In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin). Often,
    Message 1 of 23 , Jan 3, 2006
      In shops that use manual testing, whether scripted or exploratory,
      the testers are the people in the shop who use the product most (by a
      huge margin). Often, they're repeating the same workflow over and
      over, with small variations. Inefficiencies in the workflow really
      start to drive you crazy when you hit them again and again and again
      - whether you're a tester or user. Opportunities for improvement leap
      out at you.

      -- Alain:
      I agree with this. I write a lot of unit tests, but I do a lot of manual
      testing on top of it. And through that, I get a lot of excellent
      insights into where the usability hotspots are, and not only for expert
      users. Like you said, when you use the software over and over again,
      even the lightest pain starts jumping at you.

      Of course, I don't rely just on that for assessing usability of the
      system. I also pay a lot of attention to customer feedback. Everytime
      someone asks for help is treated as an occasion to get insights into
      usability problems.
      ----
    • Desilets, Alain
      Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the test I was getting at. You hit on it here. There s a
      Message 2 of 23 , Jan 3, 2006
        Hope I didn't come across as slamming testers. ;-) Actually, it's
        not the tester, but the nature of the test I was getting at. You hit
        on it here. There's a difference between testing that the feature
        performas as specified and actually using the product as intended and
        making an often subjective judgement about how well it works. That's
        the kind of testing that could be done in house by testers.

        I have observed a pattern where testers become acutely aware of
        inefficiencies in the workflow for the test cases they execute often
        without regard for how realistically the test case reflects an actual
        users' goals or usage.

        -- Alain:
        I for one, never use manual to repeatively test the same scenario.
        That's something better reserved for machines (i.e. done through
        automated unit testing).

        When I talked about manual testing (in my response to Bryan's posting),
        I was talking about exploratory testing, which, as I practice it
        involves two things:

        - trying to come up with creative ways to destroy the system
        - using the system to carry out some realistic task (i.e. a task that a
        a real user might do)

        The second kind of testing does allow me to spot a lot of usability
        problems at the design level, but also bugs that stand in the way of
        usability.
        ----
      • Phlip
        ... Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink
        Message 3 of 23 , Jan 3, 2006
          Desilets, Alain wrote:

          > Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
          > work on my part. And that includes testing, writing unit tests, and
          > integrating into the main version. I could do it in my sleep!

          Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid thing.

          That's actually useful in some situations, because you can do "blink
          comparator" from one keystroke.

          Never mind.

          --
          Emily Litella
        • Brian Marick
          ... Agreed. Going beyond what you say: a tester without understanding of the domain will not only make bad usability suggestions, she will also report too many
          Message 4 of 23 , Jan 3, 2006
            On Jan 3, 2006, at 2:18 PM, Jeff Patton wrote:
            >
            > I have observed a pattern where testers become acutely aware of
            > inefficiencies in the workflow for the test cases they execute often
            > without regard for how realistically the test case reflects an actual
            > users' goals or usage.
            >
            > For example I used to do a lot of work in a brick and mortar retail
            > environment. Testers often asked for features to create items to
            > sell with less mandatory attributes, or navigate directly from item
            > creation to transaction entry, or to easily remove items. While all
            > these seemed logical, in large retail organizations those creating
            > items weren't the ones entering transactions. And removing items
            > with any transactional history had larger legal implications. The
            > feature suggestions would have indeed improved workflow for the
            > tester executing test cases, but not necessarily the user doing work.
            >
            > Now I know enlightened testers such as Brian and others wouldn't fall
            > into this trap. But to help the less enlightened I've always found
            > it important to have a strong understanding of the applications users
            > and their workflow. A good user model and task model serve that
            > purpose. Some concisely written user scenarios help with that also.
            > It's hard to remember sometimes that we're not the user and the
            > models help remind us.

            Agreed. Going beyond what you say: a tester without understanding of
            the domain will not only make bad usability suggestions, she will
            also report too many bugs that don't matter to real users and too few
            bugs that do.

            As Michael Bolton and a host of others would instantly remind me, the
            good tester is also operating under the assumption that the good user
            model and good task model are wrong in some important ways, and that
            part of her job is probably to find out how. (Similarly, her own
            understanding is doubtless wrong, too. It's the Turtles of
            Uncertainty all the way down.)


            P.S. Making changes to improve the usability for testers does have
            business value, unless their time costs nothing. I have some faith,
            but no evidence, that it would also improve the code, much like
            catering to JUnit or Fit does.

            -----
            Brian Marick, independent consultant
            Mostly on agile methods with a testing slant
            www.exampler.com, www.testing.com/cgi-bin/blog
            Book in progress: www.exampler.com/book
          • Desilets, Alain
            Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink comparator
            Message 5 of 23 , Jan 3, 2006
              Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid
              thing.

              That's actually useful in some situations, because you can do "blink
              comparator" from one keystroke.

              Never mind.

              -- Alain:
              Interesting point. Do you think this is something that you would be
              likely to find out through upfront design and paper prototyping?

              My guess is that this is exactly the kind of thing that does not become
              apparent until (even to the end user) until you look at real users
              laying their hands on the real thing and trying to carry out a real
              task.
              ----
            • Jeff Patton
              ... In the situation I was thinking about, making changes to the delivered product to better support testers would have broken business rules in the
              Message 6 of 23 , Jan 3, 2006
                --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                wrote:
                > P.S. Making changes to improve the usability for testers does have
                > business value, unless their time costs nothing. I have some faith,
                > but no evidence, that it would also improve the code, much like
                > catering to JUnit or Fit does.

                In the situation I was thinking about, making changes to the delivered
                product to better support testers would have broken business rules in
                the application. Basically it would have made it easier/faster for the
                tester to set up and tear down test data by forgoing some of the rules
                around creating and deleting - and also installing quick navigation to
                jump from creation directly to transaction entry, again, inapropriate
                for our app, but workflow the testers did often.

                Alain is right that lots of these tedious bits of testing should be
                automated.

                You're right that spending a little money to help testers complete
                their work faster - or not go crazy doing it - is a good idea. We just
                need to know why we're doing it. If it was a feature strictly for
                testing, I'd want some way to hide or disable it from a production
                release.

                But, your point is well taken. Also Michael's point you referred to as
                well.

                thanks,

                -Jeff
              • Larry Constantine
                ... Many bugs (both code bugs and usability defects) can be avoided by effective design practices and found prior to having running software through
                Message 7 of 23 , Jan 4, 2006
                  Alain wrote:

                  > By their nature, bugs cannot be caught by design. They can only be
                  > caught after the buggy implementation has been implemented. This is one
                  > of the great advantage of early delivery and TDD. They allow you to get
                  > actual running software in the hands of users early, and catch bugs
                  > early on in the process.

                  Many bugs (both code bugs and usability defects) can be avoided by effective
                  design practices and found prior to having running software through
                  inspections and walkthroughs. Multiple research studies and extensive
                  practice have shown inspections to be more cost effective than is testing
                  for finding and eliminating bugs (both kinds).

                  --Larry Constantine, IDSA [mailto:lconstantine@...]
                    Chief Scientist | Constantine & Lockwood, Ltd.

                  > -----Original Message-----
                  > From: agile-usability@yahoogroups.com
                  [mailto:agile-usability@yahoogroups.com] On Behalf
                  > Of Desilets, Alain
                  > Sent: Tuesday, 03 January 2006 11:40 AM
                  > To: agile-usability@yahoogroups.com
                  > Subject: RE: [agile-usability] just one bug's enough to make a program
                  useless
                  >
                  > How can we usabilitists catch these blind spots in design, before they
                  > lose us customers?
                  >
                  > -- Alain:
                  > Sorry for late response... Catching up on pre-Xmas break email.
                  >
                  > ----
                  >
                  >
                  >
                  > Yahoo! Groups Links
                  >
                  >
                  >
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.