Loading ...
Sorry, an error occurred while loading the content.

Re: [agile-usability] Re: just one bug's enough to make a program useless

Expand Messages
  • Phlip
    ... Alain implied early feedback from customers does. Customer: I undid twice and could not step back through edits. Team: That s not a bug it s a feature.
    Message 1 of 23 , Jan 3, 2006
    • 0 Attachment
      Jeff Patton wrote:

      > Looking up to Phlip's original post, I was seeing the bug reported as
      > not supporting multiple levels of undo. Then we found there was a
      > history panel - an alternatively named feature that accomplished
      > similar results to undo. It appears all these features performed as
      > the publisher expected - so none of these were bugs. Practices like
      > TDD don't catch things like this.

      Alain implied early feedback from customers does.

      Customer: I undid twice and could not step back through edits.

      Team: That's not a bug it's a feature. (Try the History panel.)

      Customer: Okay, the bug is Undo isn't wired to History.

      Boss: There's a work-around. Ship it.

      End user: I undid twice and could not step back through edits.

      And so on; the feedback cycle engages!

      > * a process or methodology that's concerned with the usability of it's
      > product should include some form of usability testing - that is
      > observation of target users using the product to accomplish real work -
      > not bug detection.

      Naivete is priceless.

      I suspect that the entire GUI industry is one big blind spot, because
      nobody made the leap to X. I don't know what it is either, because we
      are all familiar with windows and mice.

      --
      Phlip
      http://www.greencheese.org/ZeekLand <-- NOT a blog!!
    • Brian Marick
      ... In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin).
      Message 2 of 23 , Jan 3, 2006
      • 0 Attachment
        On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:

        > Others already gave answers to the "how do you catch these blind
        > spots?" question. Test. But test doesn't refer to anything
        > automated,
        > or something a tester does to make sure a feature works as designed,
        > but rather observe users actually trying to do the stuff they want to
        > do with the application.

        In shops that use manual testing, whether scripted or exploratory,
        the testers are the people in the shop who use the product most (by a
        huge margin). Often, they're repeating the same workflow over and
        over, with small variations. Inefficiencies in the workflow really
        start to drive you crazy when you hit them again and again and again
        - whether you're a tester or user. Opportunities for improvement leap
        out at you.

        Testers also tend to become expert users, who have a different usage
        profile than novices. It seems to me that most of the people used in
        usability testing are novices (to the new features, at least) and do
        not have chances to become experts. But I've rarely worked on a
        product with the formal role of experience designer.

        I've never seen a case where tester comments on the design were
        intentionally valued. Sometimes testers don't bother. Sometimes they
        put the bugs into the bug tracking system, usually at the "feature
        request" (aka "lowest") level.

        When I've seen tester observations actually become valued, it's been
        because the tester felt comfortable going directly to the designer.
        (In my one experience with that, my relationship to the designer felt
        like the relationships I see between testers and customers/product
        owners/goal donors.)

        Good experience design may have a better way to see what testers see,
        but perhaps testers fit into the Big Picture of Agile UX.

        -----
        Brian Marick, independent consultant
        Mostly on agile methods with a testing slant
        www.exampler.com, www.testing.com/cgi-bin/blog
        Book in progress: www.exampler.com/book
      • Jeff Patton
        ... designed, ... want to ... a ... again ... leap ... Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the
        Message 3 of 23 , Jan 3, 2006
        • 0 Attachment
          --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
          wrote:
          > On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:
          >
          > > Others already gave answers to the "how do you catch these blind
          > > spots?" question. Test. But test doesn't refer to anything
          > > automated,
          > > or something a tester does to make sure a feature works as
          designed,
          > > but rather observe users actually trying to do the stuff they
          want to
          > > do with the application.
          >
          > In shops that use manual testing, whether scripted or exploratory,
          > the testers are the people in the shop who use the product most (by
          a
          > huge margin). Often, they're repeating the same workflow over and
          > over, with small variations. Inefficiencies in the workflow really
          > start to drive you crazy when you hit them again and again and
          again
          > - whether you're a tester or user. Opportunities for improvement
          leap
          > out at you.

          Hope I didn't come across as slamming testers. ;-) Actually, it's
          not the tester, but the nature of the test I was getting at. You hit
          on it here. There's a difference between testing that the feature
          performas as specified and actually using the product as intended and
          making an often subjective judgement about how well it works. That's
          the kind of testing that could be done in house by testers.

          I have observed a pattern where testers become acutely aware of
          inefficiencies in the workflow for the test cases they execute often
          without regard for how realistically the test case reflects an actual
          users' goals or usage.

          For example I used to do a lot of work in a brick and mortar retail
          environment. Testers often asked for features to create items to
          sell with less mandatory attributes, or navigate directly from item
          creation to transaction entry, or to easily remove items. While all
          these seemed logical, in large retail organizations those creating
          items weren't the ones entering transactions. And removing items
          with any transactional history had larger legal implications. The
          feature suggestions would have indeed improved workflow for the
          tester executing test cases, but not necessarily the user doing work.

          Now I know enlightened testers such as Brian and others wouldn't fall
          into this trap. But to help the less enlightened I've always found
          it important to have a strong understanding of the applications users
          and their workflow. A good user model and task model serve that
          purpose. Some concisely written user scenarios help with that also.
          It's hard to remember sometimes that we're not the user and the
          models help remind us.

          > I've never seen a case where tester comments on the design were
          > intentionally valued. Sometimes testers don't bother. Sometimes
          they
          > put the bugs into the bug tracking system, usually at the "feature
          > request" (aka "lowest") level.
          >
          > When I've seen tester observations actually become valued, it's
          been
          > because the tester felt comfortable going directly to the
          designer.
          > (In my one experience with that, my relationship to the designer
          felt
          > like the relationships I see between testers and customers/product
          > owners/goal donors.)

          I think it's important to suggest features in some context - by that
          I mean some model understanding of the user that allows us to
          determine if the feature is indeed good. A feature suggestion on its
          own doesn't give us any context to judge how good it is. Designers
          or product owners generally carry a model of the users and usage in
          their head if not explicitly on paper. They're better able to judge
          if the feature is or isn't valuable.

          > Good experience design may have a better way to see what testers
          see,
          > but perhaps testers fit into the Big Picture of Agile UX.

          Yes - I'd agree - but not all testers. This "testing for optimizing
          user experience" isn't something I've observed all testers agree
          with, even in an agile context. That's the kind of testing I'd like
          to see done. Specifically who does it and how is the bit to be
          resolved in the organization doing it.

          It's interesting also that this isn't the type of test that can be
          automated.
        • Desilets, Alain
          ... one ... Looking up to Phlip s original post, I was seeing the bug reported as not supporting multiple levels of undo. Then we found there was a history
          Message 4 of 23 , Jan 3, 2006
          • 0 Attachment
            > How can we usabilitists catch these blind spots in design, before they

            > lose us customers?
            >
            > -- Alain:
            > Sorry for late response... Catching up on pre-Xmas break email.
            >
            > By their nature, bugs cannot be caught by design. They can only be
            > caught after the buggy implementation has been implemented. This is
            one
            > of the great advantage of early delivery and TDD....
            > ----

            Looking up to Phlip's original post, I was seeing the bug reported as
            not supporting multiple levels of undo. Then we found there was a
            history panel - an alternatively named feature that accomplished
            similar results to undo. It appears all these features performed as
            the publisher expected - so none of these were bugs. Practices like
            TDD don't catch things like this.

            -- Alain:
            Given the title of Phlip's posting, I was writing with the assumption
            that the reported problem was a bug (i.e. developpers did implement a
            multi-level undo feature but it somehow did not work on Phlip's machine)
            as opposed to a design problem (i.e. feature deemed not necessary when
            it was in fact essential, or feature exists, but is hidden).

            So I stand by what I said. If the problem is a bug, no amount of design
            related work will find it.

            In this particular case, it turns out what Phlip thought was a bug was
            in fact a problem with bad design... The feature existed but it was
            hidden too well. But I have seen cases where a bug makes an otherwise
            well designed software completely unusable.

            The point is this. If you want your software to be usable, you have to
            pay attention to the design, but also make sure that the implementation
            works correctly. And the only way to identify cases where it does not is
            to test the actual implementation continuously through TDD, and also,
            put it in the hands of real test users as quickly as possible.
            ----
          • Desilets, Alain
            -- Alain: You forgot two important and very likely interventions in the conversation below. I insert them below. ... Customer: I undid twice and could not step
            Message 5 of 23 , Jan 3, 2006
            • 0 Attachment
              -- Alain:
              You forgot two important and very likely interventions in the
              conversation below. I insert them below.
              ---

              Customer: I undid twice and could not step back through edits.

              Team: That's not a bug it's a feature. (Try the History panel.)

              Customer: Okay, the bug is Undo isn't wired to History.

              Boss: There's a work-around. Ship it.

              -- Alain:
              At this point, the developper would probably say:

              Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
              work on my part. And that includes testing, writing unit tests, and
              integrating into the main version. I could do it in my sleep!

              Boss: OK then, make it so.
              -----
            • Desilets, Alain
              In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin). Often,
              Message 6 of 23 , Jan 3, 2006
              • 0 Attachment
                In shops that use manual testing, whether scripted or exploratory,
                the testers are the people in the shop who use the product most (by a
                huge margin). Often, they're repeating the same workflow over and
                over, with small variations. Inefficiencies in the workflow really
                start to drive you crazy when you hit them again and again and again
                - whether you're a tester or user. Opportunities for improvement leap
                out at you.

                -- Alain:
                I agree with this. I write a lot of unit tests, but I do a lot of manual
                testing on top of it. And through that, I get a lot of excellent
                insights into where the usability hotspots are, and not only for expert
                users. Like you said, when you use the software over and over again,
                even the lightest pain starts jumping at you.

                Of course, I don't rely just on that for assessing usability of the
                system. I also pay a lot of attention to customer feedback. Everytime
                someone asks for help is treated as an occasion to get insights into
                usability problems.
                ----
              • Desilets, Alain
                Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the test I was getting at. You hit on it here. There s a
                Message 7 of 23 , Jan 3, 2006
                • 0 Attachment
                  Hope I didn't come across as slamming testers. ;-) Actually, it's
                  not the tester, but the nature of the test I was getting at. You hit
                  on it here. There's a difference between testing that the feature
                  performas as specified and actually using the product as intended and
                  making an often subjective judgement about how well it works. That's
                  the kind of testing that could be done in house by testers.

                  I have observed a pattern where testers become acutely aware of
                  inefficiencies in the workflow for the test cases they execute often
                  without regard for how realistically the test case reflects an actual
                  users' goals or usage.

                  -- Alain:
                  I for one, never use manual to repeatively test the same scenario.
                  That's something better reserved for machines (i.e. done through
                  automated unit testing).

                  When I talked about manual testing (in my response to Bryan's posting),
                  I was talking about exploratory testing, which, as I practice it
                  involves two things:

                  - trying to come up with creative ways to destroy the system
                  - using the system to carry out some realistic task (i.e. a task that a
                  a real user might do)

                  The second kind of testing does allow me to spot a lot of usability
                  problems at the design level, but also bugs that stand in the way of
                  usability.
                  ----
                • Phlip
                  ... Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink
                  Message 8 of 23 , Jan 3, 2006
                  • 0 Attachment
                    Desilets, Alain wrote:

                    > Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
                    > work on my part. And that includes testing, writing unit tests, and
                    > integrating into the main version. I could do it in my sleep!

                    Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid thing.

                    That's actually useful in some situations, because you can do "blink
                    comparator" from one keystroke.

                    Never mind.

                    --
                    Emily Litella
                  • Brian Marick
                    ... Agreed. Going beyond what you say: a tester without understanding of the domain will not only make bad usability suggestions, she will also report too many
                    Message 9 of 23 , Jan 3, 2006
                    • 0 Attachment
                      On Jan 3, 2006, at 2:18 PM, Jeff Patton wrote:
                      >
                      > I have observed a pattern where testers become acutely aware of
                      > inefficiencies in the workflow for the test cases they execute often
                      > without regard for how realistically the test case reflects an actual
                      > users' goals or usage.
                      >
                      > For example I used to do a lot of work in a brick and mortar retail
                      > environment. Testers often asked for features to create items to
                      > sell with less mandatory attributes, or navigate directly from item
                      > creation to transaction entry, or to easily remove items. While all
                      > these seemed logical, in large retail organizations those creating
                      > items weren't the ones entering transactions. And removing items
                      > with any transactional history had larger legal implications. The
                      > feature suggestions would have indeed improved workflow for the
                      > tester executing test cases, but not necessarily the user doing work.
                      >
                      > Now I know enlightened testers such as Brian and others wouldn't fall
                      > into this trap. But to help the less enlightened I've always found
                      > it important to have a strong understanding of the applications users
                      > and their workflow. A good user model and task model serve that
                      > purpose. Some concisely written user scenarios help with that also.
                      > It's hard to remember sometimes that we're not the user and the
                      > models help remind us.

                      Agreed. Going beyond what you say: a tester without understanding of
                      the domain will not only make bad usability suggestions, she will
                      also report too many bugs that don't matter to real users and too few
                      bugs that do.

                      As Michael Bolton and a host of others would instantly remind me, the
                      good tester is also operating under the assumption that the good user
                      model and good task model are wrong in some important ways, and that
                      part of her job is probably to find out how. (Similarly, her own
                      understanding is doubtless wrong, too. It's the Turtles of
                      Uncertainty all the way down.)


                      P.S. Making changes to improve the usability for testers does have
                      business value, unless their time costs nothing. I have some faith,
                      but no evidence, that it would also improve the code, much like
                      catering to JUnit or Fit does.

                      -----
                      Brian Marick, independent consultant
                      Mostly on agile methods with a testing slant
                      www.exampler.com, www.testing.com/cgi-bin/blog
                      Book in progress: www.exampler.com/book
                    • Desilets, Alain
                      Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink comparator
                      Message 10 of 23 , Jan 3, 2006
                      • 0 Attachment
                        Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid
                        thing.

                        That's actually useful in some situations, because you can do "blink
                        comparator" from one keystroke.

                        Never mind.

                        -- Alain:
                        Interesting point. Do you think this is something that you would be
                        likely to find out through upfront design and paper prototyping?

                        My guess is that this is exactly the kind of thing that does not become
                        apparent until (even to the end user) until you look at real users
                        laying their hands on the real thing and trying to carry out a real
                        task.
                        ----
                      • Jeff Patton
                        ... In the situation I was thinking about, making changes to the delivered product to better support testers would have broken business rules in the
                        Message 11 of 23 , Jan 3, 2006
                        • 0 Attachment
                          --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                          wrote:
                          > P.S. Making changes to improve the usability for testers does have
                          > business value, unless their time costs nothing. I have some faith,
                          > but no evidence, that it would also improve the code, much like
                          > catering to JUnit or Fit does.

                          In the situation I was thinking about, making changes to the delivered
                          product to better support testers would have broken business rules in
                          the application. Basically it would have made it easier/faster for the
                          tester to set up and tear down test data by forgoing some of the rules
                          around creating and deleting - and also installing quick navigation to
                          jump from creation directly to transaction entry, again, inapropriate
                          for our app, but workflow the testers did often.

                          Alain is right that lots of these tedious bits of testing should be
                          automated.

                          You're right that spending a little money to help testers complete
                          their work faster - or not go crazy doing it - is a good idea. We just
                          need to know why we're doing it. If it was a feature strictly for
                          testing, I'd want some way to hide or disable it from a production
                          release.

                          But, your point is well taken. Also Michael's point you referred to as
                          well.

                          thanks,

                          -Jeff
                        • Larry Constantine
                          ... Many bugs (both code bugs and usability defects) can be avoided by effective design practices and found prior to having running software through
                          Message 12 of 23 , Jan 4, 2006
                          • 0 Attachment
                            Alain wrote:

                            > By their nature, bugs cannot be caught by design. They can only be
                            > caught after the buggy implementation has been implemented. This is one
                            > of the great advantage of early delivery and TDD. They allow you to get
                            > actual running software in the hands of users early, and catch bugs
                            > early on in the process.

                            Many bugs (both code bugs and usability defects) can be avoided by effective
                            design practices and found prior to having running software through
                            inspections and walkthroughs. Multiple research studies and extensive
                            practice have shown inspections to be more cost effective than is testing
                            for finding and eliminating bugs (both kinds).

                            --Larry Constantine, IDSA [mailto:lconstantine@...]
                              Chief Scientist | Constantine & Lockwood, Ltd.

                            > -----Original Message-----
                            > From: agile-usability@yahoogroups.com
                            [mailto:agile-usability@yahoogroups.com] On Behalf
                            > Of Desilets, Alain
                            > Sent: Tuesday, 03 January 2006 11:40 AM
                            > To: agile-usability@yahoogroups.com
                            > Subject: RE: [agile-usability] just one bug's enough to make a program
                            useless
                            >
                            > How can we usabilitists catch these blind spots in design, before they
                            > lose us customers?
                            >
                            > -- Alain:
                            > Sorry for late response... Catching up on pre-Xmas break email.
                            >
                            > ----
                            >
                            >
                            >
                            > Yahoo! Groups Links
                            >
                            >
                            >
                            >
                          Your message has been successfully submitted and would be delivered to recipients shortly.