Loading ...
Sorry, an error occurred while loading the content.

RE: [agile-usability] just one bug's enough to make a program useless

Expand Messages
  • Desilets, Alain
    How can we usabilitists catch these blind spots in design, before they lose us customers? -- Alain: Sorry for late response... Catching up on pre-Xmas break
    Message 1 of 23 , Jan 3, 2006
    • 0 Attachment
      How can we usabilitists catch these blind spots in design, before they
      lose us customers?

      -- Alain:
      Sorry for late response... Catching up on pre-Xmas break email.

      By their nature, bugs cannot be caught by design. They can only be
      caught after the buggy implementation has been implemented. This is one
      of the great advantage of early delivery and TDD. They allow you to get
      actual running software in the hands of users early, and catch bugs
      early on in the process.
      ----
    • Jeff Patton
      ... one ... Looking up to Phlip s original post, I was seeing the bug reported as not supporting multiple levels of undo. Then we found there was a history
      Message 2 of 23 , Jan 3, 2006
      • 0 Attachment
        --- In agile-usability@yahoogroups.com, "Desilets, Alain"
        <alain.desilets@n...> wrote:
        >
        > How can we usabilitists catch these blind spots in design, before they
        > lose us customers?
        >
        > -- Alain:
        > Sorry for late response... Catching up on pre-Xmas break email.
        >
        > By their nature, bugs cannot be caught by design. They can only be
        > caught after the buggy implementation has been implemented. This is
        one
        > of the great advantage of early delivery and TDD....
        > ----

        Looking up to Phlip's original post, I was seeing the bug reported as
        not supporting multiple levels of undo. Then we found there was a
        history panel - an alternatively named feature that accomplished
        similar results to undo. It appears all these features performed as
        the publisher expected - so none of these were bugs. Practices like
        TDD don't catch things like this.

        There's a couple things going on here that caught my eye/brain:

        I like that Phlip referred to this as a blind spot in _design_. One of
        my pet peeves is the use of the word design to refer almost strictly to
        the technical junk under the UI. In this case the word design refers
        to the decision to include or exclude a feature. This is the sort
        of /design/ work that's aften called "requirements" work in many
        shops.

        Others already gave answers to the "how do you catch these blind
        spots?" question. Test. But test doesn't refer to anything automated,
        or something a tester does to make sure a feature works as designed,
        but rather observe users actually trying to do the stuff they want to
        do with the application.

        So, recapping, the two things that popped for me:
        * design is something that occurrs before code is written as part of a
        process to choose features. [That's no news for the UCD community,
        possible big news to requirements engineers.]
        * a process or methodology that's concerned with the usability of it's
        product should include some form of usability testing - that is
        observation of target users using the product to accomplish real work -
        not bug detection.

        So, if there's an agile approach that's usability friendly, it's gunna
        have these sorts of concepts stitched in.

        Thanks,

        -Jeff
      • Phlip
        ... Alain implied early feedback from customers does. Customer: I undid twice and could not step back through edits. Team: That s not a bug it s a feature.
        Message 3 of 23 , Jan 3, 2006
        • 0 Attachment
          Jeff Patton wrote:

          > Looking up to Phlip's original post, I was seeing the bug reported as
          > not supporting multiple levels of undo. Then we found there was a
          > history panel - an alternatively named feature that accomplished
          > similar results to undo. It appears all these features performed as
          > the publisher expected - so none of these were bugs. Practices like
          > TDD don't catch things like this.

          Alain implied early feedback from customers does.

          Customer: I undid twice and could not step back through edits.

          Team: That's not a bug it's a feature. (Try the History panel.)

          Customer: Okay, the bug is Undo isn't wired to History.

          Boss: There's a work-around. Ship it.

          End user: I undid twice and could not step back through edits.

          And so on; the feedback cycle engages!

          > * a process or methodology that's concerned with the usability of it's
          > product should include some form of usability testing - that is
          > observation of target users using the product to accomplish real work -
          > not bug detection.

          Naivete is priceless.

          I suspect that the entire GUI industry is one big blind spot, because
          nobody made the leap to X. I don't know what it is either, because we
          are all familiar with windows and mice.

          --
          Phlip
          http://www.greencheese.org/ZeekLand <-- NOT a blog!!
        • Brian Marick
          ... In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin).
          Message 4 of 23 , Jan 3, 2006
          • 0 Attachment
            On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:

            > Others already gave answers to the "how do you catch these blind
            > spots?" question. Test. But test doesn't refer to anything
            > automated,
            > or something a tester does to make sure a feature works as designed,
            > but rather observe users actually trying to do the stuff they want to
            > do with the application.

            In shops that use manual testing, whether scripted or exploratory,
            the testers are the people in the shop who use the product most (by a
            huge margin). Often, they're repeating the same workflow over and
            over, with small variations. Inefficiencies in the workflow really
            start to drive you crazy when you hit them again and again and again
            - whether you're a tester or user. Opportunities for improvement leap
            out at you.

            Testers also tend to become expert users, who have a different usage
            profile than novices. It seems to me that most of the people used in
            usability testing are novices (to the new features, at least) and do
            not have chances to become experts. But I've rarely worked on a
            product with the formal role of experience designer.

            I've never seen a case where tester comments on the design were
            intentionally valued. Sometimes testers don't bother. Sometimes they
            put the bugs into the bug tracking system, usually at the "feature
            request" (aka "lowest") level.

            When I've seen tester observations actually become valued, it's been
            because the tester felt comfortable going directly to the designer.
            (In my one experience with that, my relationship to the designer felt
            like the relationships I see between testers and customers/product
            owners/goal donors.)

            Good experience design may have a better way to see what testers see,
            but perhaps testers fit into the Big Picture of Agile UX.

            -----
            Brian Marick, independent consultant
            Mostly on agile methods with a testing slant
            www.exampler.com, www.testing.com/cgi-bin/blog
            Book in progress: www.exampler.com/book
          • Jeff Patton
            ... designed, ... want to ... a ... again ... leap ... Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the
            Message 5 of 23 , Jan 3, 2006
            • 0 Attachment
              --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
              wrote:
              > On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:
              >
              > > Others already gave answers to the "how do you catch these blind
              > > spots?" question. Test. But test doesn't refer to anything
              > > automated,
              > > or something a tester does to make sure a feature works as
              designed,
              > > but rather observe users actually trying to do the stuff they
              want to
              > > do with the application.
              >
              > In shops that use manual testing, whether scripted or exploratory,
              > the testers are the people in the shop who use the product most (by
              a
              > huge margin). Often, they're repeating the same workflow over and
              > over, with small variations. Inefficiencies in the workflow really
              > start to drive you crazy when you hit them again and again and
              again
              > - whether you're a tester or user. Opportunities for improvement
              leap
              > out at you.

              Hope I didn't come across as slamming testers. ;-) Actually, it's
              not the tester, but the nature of the test I was getting at. You hit
              on it here. There's a difference between testing that the feature
              performas as specified and actually using the product as intended and
              making an often subjective judgement about how well it works. That's
              the kind of testing that could be done in house by testers.

              I have observed a pattern where testers become acutely aware of
              inefficiencies in the workflow for the test cases they execute often
              without regard for how realistically the test case reflects an actual
              users' goals or usage.

              For example I used to do a lot of work in a brick and mortar retail
              environment. Testers often asked for features to create items to
              sell with less mandatory attributes, or navigate directly from item
              creation to transaction entry, or to easily remove items. While all
              these seemed logical, in large retail organizations those creating
              items weren't the ones entering transactions. And removing items
              with any transactional history had larger legal implications. The
              feature suggestions would have indeed improved workflow for the
              tester executing test cases, but not necessarily the user doing work.

              Now I know enlightened testers such as Brian and others wouldn't fall
              into this trap. But to help the less enlightened I've always found
              it important to have a strong understanding of the applications users
              and their workflow. A good user model and task model serve that
              purpose. Some concisely written user scenarios help with that also.
              It's hard to remember sometimes that we're not the user and the
              models help remind us.

              > I've never seen a case where tester comments on the design were
              > intentionally valued. Sometimes testers don't bother. Sometimes
              they
              > put the bugs into the bug tracking system, usually at the "feature
              > request" (aka "lowest") level.
              >
              > When I've seen tester observations actually become valued, it's
              been
              > because the tester felt comfortable going directly to the
              designer.
              > (In my one experience with that, my relationship to the designer
              felt
              > like the relationships I see between testers and customers/product
              > owners/goal donors.)

              I think it's important to suggest features in some context - by that
              I mean some model understanding of the user that allows us to
              determine if the feature is indeed good. A feature suggestion on its
              own doesn't give us any context to judge how good it is. Designers
              or product owners generally carry a model of the users and usage in
              their head if not explicitly on paper. They're better able to judge
              if the feature is or isn't valuable.

              > Good experience design may have a better way to see what testers
              see,
              > but perhaps testers fit into the Big Picture of Agile UX.

              Yes - I'd agree - but not all testers. This "testing for optimizing
              user experience" isn't something I've observed all testers agree
              with, even in an agile context. That's the kind of testing I'd like
              to see done. Specifically who does it and how is the bit to be
              resolved in the organization doing it.

              It's interesting also that this isn't the type of test that can be
              automated.
            • Desilets, Alain
              ... one ... Looking up to Phlip s original post, I was seeing the bug reported as not supporting multiple levels of undo. Then we found there was a history
              Message 6 of 23 , Jan 3, 2006
              • 0 Attachment
                > How can we usabilitists catch these blind spots in design, before they

                > lose us customers?
                >
                > -- Alain:
                > Sorry for late response... Catching up on pre-Xmas break email.
                >
                > By their nature, bugs cannot be caught by design. They can only be
                > caught after the buggy implementation has been implemented. This is
                one
                > of the great advantage of early delivery and TDD....
                > ----

                Looking up to Phlip's original post, I was seeing the bug reported as
                not supporting multiple levels of undo. Then we found there was a
                history panel - an alternatively named feature that accomplished
                similar results to undo. It appears all these features performed as
                the publisher expected - so none of these were bugs. Practices like
                TDD don't catch things like this.

                -- Alain:
                Given the title of Phlip's posting, I was writing with the assumption
                that the reported problem was a bug (i.e. developpers did implement a
                multi-level undo feature but it somehow did not work on Phlip's machine)
                as opposed to a design problem (i.e. feature deemed not necessary when
                it was in fact essential, or feature exists, but is hidden).

                So I stand by what I said. If the problem is a bug, no amount of design
                related work will find it.

                In this particular case, it turns out what Phlip thought was a bug was
                in fact a problem with bad design... The feature existed but it was
                hidden too well. But I have seen cases where a bug makes an otherwise
                well designed software completely unusable.

                The point is this. If you want your software to be usable, you have to
                pay attention to the design, but also make sure that the implementation
                works correctly. And the only way to identify cases where it does not is
                to test the actual implementation continuously through TDD, and also,
                put it in the hands of real test users as quickly as possible.
                ----
              • Desilets, Alain
                -- Alain: You forgot two important and very likely interventions in the conversation below. I insert them below. ... Customer: I undid twice and could not step
                Message 7 of 23 , Jan 3, 2006
                • 0 Attachment
                  -- Alain:
                  You forgot two important and very likely interventions in the
                  conversation below. I insert them below.
                  ---

                  Customer: I undid twice and could not step back through edits.

                  Team: That's not a bug it's a feature. (Try the History panel.)

                  Customer: Okay, the bug is Undo isn't wired to History.

                  Boss: There's a work-around. Ship it.

                  -- Alain:
                  At this point, the developper would probably say:

                  Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
                  work on my part. And that includes testing, writing unit tests, and
                  integrating into the main version. I could do it in my sleep!

                  Boss: OK then, make it so.
                  -----
                • Desilets, Alain
                  In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin). Often,
                  Message 8 of 23 , Jan 3, 2006
                  • 0 Attachment
                    In shops that use manual testing, whether scripted or exploratory,
                    the testers are the people in the shop who use the product most (by a
                    huge margin). Often, they're repeating the same workflow over and
                    over, with small variations. Inefficiencies in the workflow really
                    start to drive you crazy when you hit them again and again and again
                    - whether you're a tester or user. Opportunities for improvement leap
                    out at you.

                    -- Alain:
                    I agree with this. I write a lot of unit tests, but I do a lot of manual
                    testing on top of it. And through that, I get a lot of excellent
                    insights into where the usability hotspots are, and not only for expert
                    users. Like you said, when you use the software over and over again,
                    even the lightest pain starts jumping at you.

                    Of course, I don't rely just on that for assessing usability of the
                    system. I also pay a lot of attention to customer feedback. Everytime
                    someone asks for help is treated as an occasion to get insights into
                    usability problems.
                    ----
                  • Desilets, Alain
                    Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the test I was getting at. You hit on it here. There s a
                    Message 9 of 23 , Jan 3, 2006
                    • 0 Attachment
                      Hope I didn't come across as slamming testers. ;-) Actually, it's
                      not the tester, but the nature of the test I was getting at. You hit
                      on it here. There's a difference between testing that the feature
                      performas as specified and actually using the product as intended and
                      making an often subjective judgement about how well it works. That's
                      the kind of testing that could be done in house by testers.

                      I have observed a pattern where testers become acutely aware of
                      inefficiencies in the workflow for the test cases they execute often
                      without regard for how realistically the test case reflects an actual
                      users' goals or usage.

                      -- Alain:
                      I for one, never use manual to repeatively test the same scenario.
                      That's something better reserved for machines (i.e. done through
                      automated unit testing).

                      When I talked about manual testing (in my response to Bryan's posting),
                      I was talking about exploratory testing, which, as I practice it
                      involves two things:

                      - trying to come up with creative ways to destroy the system
                      - using the system to carry out some realistic task (i.e. a task that a
                      a real user might do)

                      The second kind of testing does allow me to spot a lot of usability
                      problems at the design level, but also bugs that stand in the way of
                      usability.
                      ----
                    • Phlip
                      ... Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink
                      Message 10 of 23 , Jan 3, 2006
                      • 0 Attachment
                        Desilets, Alain wrote:

                        > Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
                        > work on my part. And that includes testing, writing unit tests, and
                        > integrating into the main version. I could do it in my sleep!

                        Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid thing.

                        That's actually useful in some situations, because you can do "blink
                        comparator" from one keystroke.

                        Never mind.

                        --
                        Emily Litella
                      • Brian Marick
                        ... Agreed. Going beyond what you say: a tester without understanding of the domain will not only make bad usability suggestions, she will also report too many
                        Message 11 of 23 , Jan 3, 2006
                        • 0 Attachment
                          On Jan 3, 2006, at 2:18 PM, Jeff Patton wrote:
                          >
                          > I have observed a pattern where testers become acutely aware of
                          > inefficiencies in the workflow for the test cases they execute often
                          > without regard for how realistically the test case reflects an actual
                          > users' goals or usage.
                          >
                          > For example I used to do a lot of work in a brick and mortar retail
                          > environment. Testers often asked for features to create items to
                          > sell with less mandatory attributes, or navigate directly from item
                          > creation to transaction entry, or to easily remove items. While all
                          > these seemed logical, in large retail organizations those creating
                          > items weren't the ones entering transactions. And removing items
                          > with any transactional history had larger legal implications. The
                          > feature suggestions would have indeed improved workflow for the
                          > tester executing test cases, but not necessarily the user doing work.
                          >
                          > Now I know enlightened testers such as Brian and others wouldn't fall
                          > into this trap. But to help the less enlightened I've always found
                          > it important to have a strong understanding of the applications users
                          > and their workflow. A good user model and task model serve that
                          > purpose. Some concisely written user scenarios help with that also.
                          > It's hard to remember sometimes that we're not the user and the
                          > models help remind us.

                          Agreed. Going beyond what you say: a tester without understanding of
                          the domain will not only make bad usability suggestions, she will
                          also report too many bugs that don't matter to real users and too few
                          bugs that do.

                          As Michael Bolton and a host of others would instantly remind me, the
                          good tester is also operating under the assumption that the good user
                          model and good task model are wrong in some important ways, and that
                          part of her job is probably to find out how. (Similarly, her own
                          understanding is doubtless wrong, too. It's the Turtles of
                          Uncertainty all the way down.)


                          P.S. Making changes to improve the usability for testers does have
                          business value, unless their time costs nothing. I have some faith,
                          but no evidence, that it would also improve the code, much like
                          catering to JUnit or Fit does.

                          -----
                          Brian Marick, independent consultant
                          Mostly on agile methods with a testing slant
                          www.exampler.com, www.testing.com/cgi-bin/blog
                          Book in progress: www.exampler.com/book
                        • Desilets, Alain
                          Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink comparator
                          Message 12 of 23 , Jan 3, 2006
                          • 0 Attachment
                            Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid
                            thing.

                            That's actually useful in some situations, because you can do "blink
                            comparator" from one keystroke.

                            Never mind.

                            -- Alain:
                            Interesting point. Do you think this is something that you would be
                            likely to find out through upfront design and paper prototyping?

                            My guess is that this is exactly the kind of thing that does not become
                            apparent until (even to the end user) until you look at real users
                            laying their hands on the real thing and trying to carry out a real
                            task.
                            ----
                          • Jeff Patton
                            ... In the situation I was thinking about, making changes to the delivered product to better support testers would have broken business rules in the
                            Message 13 of 23 , Jan 3, 2006
                            • 0 Attachment
                              --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                              wrote:
                              > P.S. Making changes to improve the usability for testers does have
                              > business value, unless their time costs nothing. I have some faith,
                              > but no evidence, that it would also improve the code, much like
                              > catering to JUnit or Fit does.

                              In the situation I was thinking about, making changes to the delivered
                              product to better support testers would have broken business rules in
                              the application. Basically it would have made it easier/faster for the
                              tester to set up and tear down test data by forgoing some of the rules
                              around creating and deleting - and also installing quick navigation to
                              jump from creation directly to transaction entry, again, inapropriate
                              for our app, but workflow the testers did often.

                              Alain is right that lots of these tedious bits of testing should be
                              automated.

                              You're right that spending a little money to help testers complete
                              their work faster - or not go crazy doing it - is a good idea. We just
                              need to know why we're doing it. If it was a feature strictly for
                              testing, I'd want some way to hide or disable it from a production
                              release.

                              But, your point is well taken. Also Michael's point you referred to as
                              well.

                              thanks,

                              -Jeff
                            • Larry Constantine
                              ... Many bugs (both code bugs and usability defects) can be avoided by effective design practices and found prior to having running software through
                              Message 14 of 23 , Jan 4, 2006
                              • 0 Attachment
                                Alain wrote:

                                > By their nature, bugs cannot be caught by design. They can only be
                                > caught after the buggy implementation has been implemented. This is one
                                > of the great advantage of early delivery and TDD. They allow you to get
                                > actual running software in the hands of users early, and catch bugs
                                > early on in the process.

                                Many bugs (both code bugs and usability defects) can be avoided by effective
                                design practices and found prior to having running software through
                                inspections and walkthroughs. Multiple research studies and extensive
                                practice have shown inspections to be more cost effective than is testing
                                for finding and eliminating bugs (both kinds).

                                --Larry Constantine, IDSA [mailto:lconstantine@...]
                                  Chief Scientist | Constantine & Lockwood, Ltd.

                                > -----Original Message-----
                                > From: agile-usability@yahoogroups.com
                                [mailto:agile-usability@yahoogroups.com] On Behalf
                                > Of Desilets, Alain
                                > Sent: Tuesday, 03 January 2006 11:40 AM
                                > To: agile-usability@yahoogroups.com
                                > Subject: RE: [agile-usability] just one bug's enough to make a program
                                useless
                                >
                                > How can we usabilitists catch these blind spots in design, before they
                                > lose us customers?
                                >
                                > -- Alain:
                                > Sorry for late response... Catching up on pre-Xmas break email.
                                >
                                > ----
                                >
                                >
                                >
                                > Yahoo! Groups Links
                                >
                                >
                                >
                                >
                              Your message has been successfully submitted and would be delivered to recipients shortly.