Loading ...
Sorry, an error occurred while loading the content.

bug or missing feature?, was: just one bug's enough to make a program useless

Expand Messages
  • Jeff Patton
    ... designed, ... want to ... a ... again ... leap ... Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the
    Message 1 of 23 , Jan 3, 2006
    View Source
    • 0 Attachment
      --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
      wrote:
      > On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:
      >
      > > Others already gave answers to the "how do you catch these blind
      > > spots?" question. Test. But test doesn't refer to anything
      > > automated,
      > > or something a tester does to make sure a feature works as
      designed,
      > > but rather observe users actually trying to do the stuff they
      want to
      > > do with the application.
      >
      > In shops that use manual testing, whether scripted or exploratory,
      > the testers are the people in the shop who use the product most (by
      a
      > huge margin). Often, they're repeating the same workflow over and
      > over, with small variations. Inefficiencies in the workflow really
      > start to drive you crazy when you hit them again and again and
      again
      > - whether you're a tester or user. Opportunities for improvement
      leap
      > out at you.

      Hope I didn't come across as slamming testers. ;-) Actually, it's
      not the tester, but the nature of the test I was getting at. You hit
      on it here. There's a difference between testing that the feature
      performas as specified and actually using the product as intended and
      making an often subjective judgement about how well it works. That's
      the kind of testing that could be done in house by testers.

      I have observed a pattern where testers become acutely aware of
      inefficiencies in the workflow for the test cases they execute often
      without regard for how realistically the test case reflects an actual
      users' goals or usage.

      For example I used to do a lot of work in a brick and mortar retail
      environment. Testers often asked for features to create items to
      sell with less mandatory attributes, or navigate directly from item
      creation to transaction entry, or to easily remove items. While all
      these seemed logical, in large retail organizations those creating
      items weren't the ones entering transactions. And removing items
      with any transactional history had larger legal implications. The
      feature suggestions would have indeed improved workflow for the
      tester executing test cases, but not necessarily the user doing work.

      Now I know enlightened testers such as Brian and others wouldn't fall
      into this trap. But to help the less enlightened I've always found
      it important to have a strong understanding of the applications users
      and their workflow. A good user model and task model serve that
      purpose. Some concisely written user scenarios help with that also.
      It's hard to remember sometimes that we're not the user and the
      models help remind us.

      > I've never seen a case where tester comments on the design were
      > intentionally valued. Sometimes testers don't bother. Sometimes
      they
      > put the bugs into the bug tracking system, usually at the "feature
      > request" (aka "lowest") level.
      >
      > When I've seen tester observations actually become valued, it's
      been
      > because the tester felt comfortable going directly to the
      designer.
      > (In my one experience with that, my relationship to the designer
      felt
      > like the relationships I see between testers and customers/product
      > owners/goal donors.)

      I think it's important to suggest features in some context - by that
      I mean some model understanding of the user that allows us to
      determine if the feature is indeed good. A feature suggestion on its
      own doesn't give us any context to judge how good it is. Designers
      or product owners generally carry a model of the users and usage in
      their head if not explicitly on paper. They're better able to judge
      if the feature is or isn't valuable.

      > Good experience design may have a better way to see what testers
      see,
      > but perhaps testers fit into the Big Picture of Agile UX.

      Yes - I'd agree - but not all testers. This "testing for optimizing
      user experience" isn't something I've observed all testers agree
      with, even in an agile context. That's the kind of testing I'd like
      to see done. Specifically who does it and how is the bit to be
      resolved in the organization doing it.

      It's interesting also that this isn't the type of test that can be
      automated.
    • Desilets, Alain
      -- Alain: You forgot two important and very likely interventions in the conversation below. I insert them below. ... Customer: I undid twice and could not step
      Message 2 of 23 , Jan 3, 2006
      View Source
      • 0 Attachment
        -- Alain:
        You forgot two important and very likely interventions in the
        conversation below. I insert them below.
        ---

        Customer: I undid twice and could not step back through edits.

        Team: That's not a bug it's a feature. (Try the History panel.)

        Customer: Okay, the bug is Undo isn't wired to History.

        Boss: There's a work-around. Ship it.

        -- Alain:
        At this point, the developper would probably say:

        Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
        work on my part. And that includes testing, writing unit tests, and
        integrating into the main version. I could do it in my sleep!

        Boss: OK then, make it so.
        -----
      • Desilets, Alain
        In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin). Often,
        Message 3 of 23 , Jan 3, 2006
        View Source
        • 0 Attachment
          In shops that use manual testing, whether scripted or exploratory,
          the testers are the people in the shop who use the product most (by a
          huge margin). Often, they're repeating the same workflow over and
          over, with small variations. Inefficiencies in the workflow really
          start to drive you crazy when you hit them again and again and again
          - whether you're a tester or user. Opportunities for improvement leap
          out at you.

          -- Alain:
          I agree with this. I write a lot of unit tests, but I do a lot of manual
          testing on top of it. And through that, I get a lot of excellent
          insights into where the usability hotspots are, and not only for expert
          users. Like you said, when you use the software over and over again,
          even the lightest pain starts jumping at you.

          Of course, I don't rely just on that for assessing usability of the
          system. I also pay a lot of attention to customer feedback. Everytime
          someone asks for help is treated as an occasion to get insights into
          usability problems.
          ----
        • Desilets, Alain
          Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the test I was getting at. You hit on it here. There s a
          Message 4 of 23 , Jan 3, 2006
          View Source
          • 0 Attachment
            Hope I didn't come across as slamming testers. ;-) Actually, it's
            not the tester, but the nature of the test I was getting at. You hit
            on it here. There's a difference between testing that the feature
            performas as specified and actually using the product as intended and
            making an often subjective judgement about how well it works. That's
            the kind of testing that could be done in house by testers.

            I have observed a pattern where testers become acutely aware of
            inefficiencies in the workflow for the test cases they execute often
            without regard for how realistically the test case reflects an actual
            users' goals or usage.

            -- Alain:
            I for one, never use manual to repeatively test the same scenario.
            That's something better reserved for machines (i.e. done through
            automated unit testing).

            When I talked about manual testing (in my response to Bryan's posting),
            I was talking about exploratory testing, which, as I practice it
            involves two things:

            - trying to come up with creative ways to destroy the system
            - using the system to carry out some realistic task (i.e. a task that a
            a real user might do)

            The second kind of testing does allow me to spot a lot of usability
            problems at the design level, but also bugs that stand in the way of
            usability.
            ----
          • Phlip
            ... Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink
            Message 5 of 23 , Jan 3, 2006
            View Source
            • 0 Attachment
              Desilets, Alain wrote:

              > Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
              > work on my part. And that includes testing, writing unit tests, and
              > integrating into the main version. I could do it in my sleep!

              Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid thing.

              That's actually useful in some situations, because you can do "blink
              comparator" from one keystroke.

              Never mind.

              --
              Emily Litella
            • Brian Marick
              ... Agreed. Going beyond what you say: a tester without understanding of the domain will not only make bad usability suggestions, she will also report too many
              Message 6 of 23 , Jan 3, 2006
              View Source
              • 0 Attachment
                On Jan 3, 2006, at 2:18 PM, Jeff Patton wrote:
                >
                > I have observed a pattern where testers become acutely aware of
                > inefficiencies in the workflow for the test cases they execute often
                > without regard for how realistically the test case reflects an actual
                > users' goals or usage.
                >
                > For example I used to do a lot of work in a brick and mortar retail
                > environment. Testers often asked for features to create items to
                > sell with less mandatory attributes, or navigate directly from item
                > creation to transaction entry, or to easily remove items. While all
                > these seemed logical, in large retail organizations those creating
                > items weren't the ones entering transactions. And removing items
                > with any transactional history had larger legal implications. The
                > feature suggestions would have indeed improved workflow for the
                > tester executing test cases, but not necessarily the user doing work.
                >
                > Now I know enlightened testers such as Brian and others wouldn't fall
                > into this trap. But to help the less enlightened I've always found
                > it important to have a strong understanding of the applications users
                > and their workflow. A good user model and task model serve that
                > purpose. Some concisely written user scenarios help with that also.
                > It's hard to remember sometimes that we're not the user and the
                > models help remind us.

                Agreed. Going beyond what you say: a tester without understanding of
                the domain will not only make bad usability suggestions, she will
                also report too many bugs that don't matter to real users and too few
                bugs that do.

                As Michael Bolton and a host of others would instantly remind me, the
                good tester is also operating under the assumption that the good user
                model and good task model are wrong in some important ways, and that
                part of her job is probably to find out how. (Similarly, her own
                understanding is doubtless wrong, too. It's the Turtles of
                Uncertainty all the way down.)


                P.S. Making changes to improve the usability for testers does have
                business value, unless their time costs nothing. I have some faith,
                but no evidence, that it would also improve the code, much like
                catering to JUnit or Fit does.

                -----
                Brian Marick, independent consultant
                Mostly on agile methods with a testing slant
                www.exampler.com, www.testing.com/cgi-bin/blog
                Book in progress: www.exampler.com/book
              • Desilets, Alain
                Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink comparator
                Message 7 of 23 , Jan 3, 2006
                View Source
                • 0 Attachment
                  Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid
                  thing.

                  That's actually useful in some situations, because you can do "blink
                  comparator" from one keystroke.

                  Never mind.

                  -- Alain:
                  Interesting point. Do you think this is something that you would be
                  likely to find out through upfront design and paper prototyping?

                  My guess is that this is exactly the kind of thing that does not become
                  apparent until (even to the end user) until you look at real users
                  laying their hands on the real thing and trying to carry out a real
                  task.
                  ----
                • Jeff Patton
                  ... In the situation I was thinking about, making changes to the delivered product to better support testers would have broken business rules in the
                  Message 8 of 23 , Jan 3, 2006
                  View Source
                  • 0 Attachment
                    --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                    wrote:
                    > P.S. Making changes to improve the usability for testers does have
                    > business value, unless their time costs nothing. I have some faith,
                    > but no evidence, that it would also improve the code, much like
                    > catering to JUnit or Fit does.

                    In the situation I was thinking about, making changes to the delivered
                    product to better support testers would have broken business rules in
                    the application. Basically it would have made it easier/faster for the
                    tester to set up and tear down test data by forgoing some of the rules
                    around creating and deleting - and also installing quick navigation to
                    jump from creation directly to transaction entry, again, inapropriate
                    for our app, but workflow the testers did often.

                    Alain is right that lots of these tedious bits of testing should be
                    automated.

                    You're right that spending a little money to help testers complete
                    their work faster - or not go crazy doing it - is a good idea. We just
                    need to know why we're doing it. If it was a feature strictly for
                    testing, I'd want some way to hide or disable it from a production
                    release.

                    But, your point is well taken. Also Michael's point you referred to as
                    well.

                    thanks,

                    -Jeff
                  • Larry Constantine
                    ... Many bugs (both code bugs and usability defects) can be avoided by effective design practices and found prior to having running software through
                    Message 9 of 23 , Jan 4, 2006
                    View Source
                    • 0 Attachment
                      Alain wrote:

                      > By their nature, bugs cannot be caught by design. They can only be
                      > caught after the buggy implementation has been implemented. This is one
                      > of the great advantage of early delivery and TDD. They allow you to get
                      > actual running software in the hands of users early, and catch bugs
                      > early on in the process.

                      Many bugs (both code bugs and usability defects) can be avoided by effective
                      design practices and found prior to having running software through
                      inspections and walkthroughs. Multiple research studies and extensive
                      practice have shown inspections to be more cost effective than is testing
                      for finding and eliminating bugs (both kinds).

                      --Larry Constantine, IDSA [mailto:lconstantine@...]
                        Chief Scientist | Constantine & Lockwood, Ltd.

                      > -----Original Message-----
                      > From: agile-usability@yahoogroups.com
                      [mailto:agile-usability@yahoogroups.com] On Behalf
                      > Of Desilets, Alain
                      > Sent: Tuesday, 03 January 2006 11:40 AM
                      > To: agile-usability@yahoogroups.com
                      > Subject: RE: [agile-usability] just one bug's enough to make a program
                      useless
                      >
                      > How can we usabilitists catch these blind spots in design, before they
                      > lose us customers?
                      >
                      > -- Alain:
                      > Sorry for late response... Catching up on pre-Xmas break email.
                      >
                      > ----
                      >
                      >
                      >
                      > Yahoo! Groups Links
                      >
                      >
                      >
                      >
                    Your message has been successfully submitted and would be delivered to recipients shortly.