Loading ...
Sorry, an error occurred while loading the content.

Re: [agile-usability] just one bug's enough to make a program useless

Expand Messages
  • Ron Jeffries
    ... Wow. Color makes a BIG difference. Love it! Really draws me into the story. Very interesting to see and feel the change. Note to usability self: color
    Message 1 of 23 , Dec 29, 2005
    • 0 Attachment
      On Thursday, December 29, 2005, at 2:36:03 AM, Phlip wrote:

      > http://www.greencheese.org/ZeekLand

      > Rat snuck into the most expensive restaurant possible, and ordered
      > like 26 dishes. Their arrival requires full color, so you can tell the
      > difference between the giant popovers and the kudzu falafel.

      Wow. Color makes a BIG difference. Love it! Really draws me into the
      story. Very interesting to see and feel the change.

      Note to usability self: color matters.

      > Naturally, knowing the Undo feature was there helped me not use it.

      That's cool -- and interesting. I wonder why ...

      > And for some strange reason, GIMP and PhotoShop have almost the same
      > feature set!

      Surprise indeed ...

      Ron Jeffries
      www.XProgramming.com
      Speculation or experimentation - which is more likely to give the correct answer?
    • Desilets, Alain
      How can we usabilitists catch these blind spots in design, before they lose us customers? -- Alain: Sorry for late response... Catching up on pre-Xmas break
      Message 2 of 23 , Jan 3, 2006
      • 0 Attachment
        How can we usabilitists catch these blind spots in design, before they
        lose us customers?

        -- Alain:
        Sorry for late response... Catching up on pre-Xmas break email.

        By their nature, bugs cannot be caught by design. They can only be
        caught after the buggy implementation has been implemented. This is one
        of the great advantage of early delivery and TDD. They allow you to get
        actual running software in the hands of users early, and catch bugs
        early on in the process.
        ----
      • Jeff Patton
        ... one ... Looking up to Phlip s original post, I was seeing the bug reported as not supporting multiple levels of undo. Then we found there was a history
        Message 3 of 23 , Jan 3, 2006
        • 0 Attachment
          --- In agile-usability@yahoogroups.com, "Desilets, Alain"
          <alain.desilets@n...> wrote:
          >
          > How can we usabilitists catch these blind spots in design, before they
          > lose us customers?
          >
          > -- Alain:
          > Sorry for late response... Catching up on pre-Xmas break email.
          >
          > By their nature, bugs cannot be caught by design. They can only be
          > caught after the buggy implementation has been implemented. This is
          one
          > of the great advantage of early delivery and TDD....
          > ----

          Looking up to Phlip's original post, I was seeing the bug reported as
          not supporting multiple levels of undo. Then we found there was a
          history panel - an alternatively named feature that accomplished
          similar results to undo. It appears all these features performed as
          the publisher expected - so none of these were bugs. Practices like
          TDD don't catch things like this.

          There's a couple things going on here that caught my eye/brain:

          I like that Phlip referred to this as a blind spot in _design_. One of
          my pet peeves is the use of the word design to refer almost strictly to
          the technical junk under the UI. In this case the word design refers
          to the decision to include or exclude a feature. This is the sort
          of /design/ work that's aften called "requirements" work in many
          shops.

          Others already gave answers to the "how do you catch these blind
          spots?" question. Test. But test doesn't refer to anything automated,
          or something a tester does to make sure a feature works as designed,
          but rather observe users actually trying to do the stuff they want to
          do with the application.

          So, recapping, the two things that popped for me:
          * design is something that occurrs before code is written as part of a
          process to choose features. [That's no news for the UCD community,
          possible big news to requirements engineers.]
          * a process or methodology that's concerned with the usability of it's
          product should include some form of usability testing - that is
          observation of target users using the product to accomplish real work -
          not bug detection.

          So, if there's an agile approach that's usability friendly, it's gunna
          have these sorts of concepts stitched in.

          Thanks,

          -Jeff
        • Phlip
          ... Alain implied early feedback from customers does. Customer: I undid twice and could not step back through edits. Team: That s not a bug it s a feature.
          Message 4 of 23 , Jan 3, 2006
          • 0 Attachment
            Jeff Patton wrote:

            > Looking up to Phlip's original post, I was seeing the bug reported as
            > not supporting multiple levels of undo. Then we found there was a
            > history panel - an alternatively named feature that accomplished
            > similar results to undo. It appears all these features performed as
            > the publisher expected - so none of these were bugs. Practices like
            > TDD don't catch things like this.

            Alain implied early feedback from customers does.

            Customer: I undid twice and could not step back through edits.

            Team: That's not a bug it's a feature. (Try the History panel.)

            Customer: Okay, the bug is Undo isn't wired to History.

            Boss: There's a work-around. Ship it.

            End user: I undid twice and could not step back through edits.

            And so on; the feedback cycle engages!

            > * a process or methodology that's concerned with the usability of it's
            > product should include some form of usability testing - that is
            > observation of target users using the product to accomplish real work -
            > not bug detection.

            Naivete is priceless.

            I suspect that the entire GUI industry is one big blind spot, because
            nobody made the leap to X. I don't know what it is either, because we
            are all familiar with windows and mice.

            --
            Phlip
            http://www.greencheese.org/ZeekLand <-- NOT a blog!!
          • Brian Marick
            ... In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin).
            Message 5 of 23 , Jan 3, 2006
            • 0 Attachment
              On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:

              > Others already gave answers to the "how do you catch these blind
              > spots?" question. Test. But test doesn't refer to anything
              > automated,
              > or something a tester does to make sure a feature works as designed,
              > but rather observe users actually trying to do the stuff they want to
              > do with the application.

              In shops that use manual testing, whether scripted or exploratory,
              the testers are the people in the shop who use the product most (by a
              huge margin). Often, they're repeating the same workflow over and
              over, with small variations. Inefficiencies in the workflow really
              start to drive you crazy when you hit them again and again and again
              - whether you're a tester or user. Opportunities for improvement leap
              out at you.

              Testers also tend to become expert users, who have a different usage
              profile than novices. It seems to me that most of the people used in
              usability testing are novices (to the new features, at least) and do
              not have chances to become experts. But I've rarely worked on a
              product with the formal role of experience designer.

              I've never seen a case where tester comments on the design were
              intentionally valued. Sometimes testers don't bother. Sometimes they
              put the bugs into the bug tracking system, usually at the "feature
              request" (aka "lowest") level.

              When I've seen tester observations actually become valued, it's been
              because the tester felt comfortable going directly to the designer.
              (In my one experience with that, my relationship to the designer felt
              like the relationships I see between testers and customers/product
              owners/goal donors.)

              Good experience design may have a better way to see what testers see,
              but perhaps testers fit into the Big Picture of Agile UX.

              -----
              Brian Marick, independent consultant
              Mostly on agile methods with a testing slant
              www.exampler.com, www.testing.com/cgi-bin/blog
              Book in progress: www.exampler.com/book
            • Jeff Patton
              ... designed, ... want to ... a ... again ... leap ... Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the
              Message 6 of 23 , Jan 3, 2006
              • 0 Attachment
                --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                wrote:
                > On Jan 3, 2006, at 11:32 AM, Jeff Patton wrote:
                >
                > > Others already gave answers to the "how do you catch these blind
                > > spots?" question. Test. But test doesn't refer to anything
                > > automated,
                > > or something a tester does to make sure a feature works as
                designed,
                > > but rather observe users actually trying to do the stuff they
                want to
                > > do with the application.
                >
                > In shops that use manual testing, whether scripted or exploratory,
                > the testers are the people in the shop who use the product most (by
                a
                > huge margin). Often, they're repeating the same workflow over and
                > over, with small variations. Inefficiencies in the workflow really
                > start to drive you crazy when you hit them again and again and
                again
                > - whether you're a tester or user. Opportunities for improvement
                leap
                > out at you.

                Hope I didn't come across as slamming testers. ;-) Actually, it's
                not the tester, but the nature of the test I was getting at. You hit
                on it here. There's a difference between testing that the feature
                performas as specified and actually using the product as intended and
                making an often subjective judgement about how well it works. That's
                the kind of testing that could be done in house by testers.

                I have observed a pattern where testers become acutely aware of
                inefficiencies in the workflow for the test cases they execute often
                without regard for how realistically the test case reflects an actual
                users' goals or usage.

                For example I used to do a lot of work in a brick and mortar retail
                environment. Testers often asked for features to create items to
                sell with less mandatory attributes, or navigate directly from item
                creation to transaction entry, or to easily remove items. While all
                these seemed logical, in large retail organizations those creating
                items weren't the ones entering transactions. And removing items
                with any transactional history had larger legal implications. The
                feature suggestions would have indeed improved workflow for the
                tester executing test cases, but not necessarily the user doing work.

                Now I know enlightened testers such as Brian and others wouldn't fall
                into this trap. But to help the less enlightened I've always found
                it important to have a strong understanding of the applications users
                and their workflow. A good user model and task model serve that
                purpose. Some concisely written user scenarios help with that also.
                It's hard to remember sometimes that we're not the user and the
                models help remind us.

                > I've never seen a case where tester comments on the design were
                > intentionally valued. Sometimes testers don't bother. Sometimes
                they
                > put the bugs into the bug tracking system, usually at the "feature
                > request" (aka "lowest") level.
                >
                > When I've seen tester observations actually become valued, it's
                been
                > because the tester felt comfortable going directly to the
                designer.
                > (In my one experience with that, my relationship to the designer
                felt
                > like the relationships I see between testers and customers/product
                > owners/goal donors.)

                I think it's important to suggest features in some context - by that
                I mean some model understanding of the user that allows us to
                determine if the feature is indeed good. A feature suggestion on its
                own doesn't give us any context to judge how good it is. Designers
                or product owners generally carry a model of the users and usage in
                their head if not explicitly on paper. They're better able to judge
                if the feature is or isn't valuable.

                > Good experience design may have a better way to see what testers
                see,
                > but perhaps testers fit into the Big Picture of Agile UX.

                Yes - I'd agree - but not all testers. This "testing for optimizing
                user experience" isn't something I've observed all testers agree
                with, even in an agile context. That's the kind of testing I'd like
                to see done. Specifically who does it and how is the bit to be
                resolved in the organization doing it.

                It's interesting also that this isn't the type of test that can be
                automated.
              • Desilets, Alain
                ... one ... Looking up to Phlip s original post, I was seeing the bug reported as not supporting multiple levels of undo. Then we found there was a history
                Message 7 of 23 , Jan 3, 2006
                • 0 Attachment
                  > How can we usabilitists catch these blind spots in design, before they

                  > lose us customers?
                  >
                  > -- Alain:
                  > Sorry for late response... Catching up on pre-Xmas break email.
                  >
                  > By their nature, bugs cannot be caught by design. They can only be
                  > caught after the buggy implementation has been implemented. This is
                  one
                  > of the great advantage of early delivery and TDD....
                  > ----

                  Looking up to Phlip's original post, I was seeing the bug reported as
                  not supporting multiple levels of undo. Then we found there was a
                  history panel - an alternatively named feature that accomplished
                  similar results to undo. It appears all these features performed as
                  the publisher expected - so none of these were bugs. Practices like
                  TDD don't catch things like this.

                  -- Alain:
                  Given the title of Phlip's posting, I was writing with the assumption
                  that the reported problem was a bug (i.e. developpers did implement a
                  multi-level undo feature but it somehow did not work on Phlip's machine)
                  as opposed to a design problem (i.e. feature deemed not necessary when
                  it was in fact essential, or feature exists, but is hidden).

                  So I stand by what I said. If the problem is a bug, no amount of design
                  related work will find it.

                  In this particular case, it turns out what Phlip thought was a bug was
                  in fact a problem with bad design... The feature existed but it was
                  hidden too well. But I have seen cases where a bug makes an otherwise
                  well designed software completely unusable.

                  The point is this. If you want your software to be usable, you have to
                  pay attention to the design, but also make sure that the implementation
                  works correctly. And the only way to identify cases where it does not is
                  to test the actual implementation continuously through TDD, and also,
                  put it in the hands of real test users as quickly as possible.
                  ----
                • Desilets, Alain
                  -- Alain: You forgot two important and very likely interventions in the conversation below. I insert them below. ... Customer: I undid twice and could not step
                  Message 8 of 23 , Jan 3, 2006
                  • 0 Attachment
                    -- Alain:
                    You forgot two important and very likely interventions in the
                    conversation below. I insert them below.
                    ---

                    Customer: I undid twice and could not step back through edits.

                    Team: That's not a bug it's a feature. (Try the History panel.)

                    Customer: Okay, the bug is Undo isn't wired to History.

                    Boss: There's a work-around. Ship it.

                    -- Alain:
                    At this point, the developper would probably say:

                    Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
                    work on my part. And that includes testing, writing unit tests, and
                    integrating into the main version. I could do it in my sleep!

                    Boss: OK then, make it so.
                    -----
                  • Desilets, Alain
                    In shops that use manual testing, whether scripted or exploratory, the testers are the people in the shop who use the product most (by a huge margin). Often,
                    Message 9 of 23 , Jan 3, 2006
                    • 0 Attachment
                      In shops that use manual testing, whether scripted or exploratory,
                      the testers are the people in the shop who use the product most (by a
                      huge margin). Often, they're repeating the same workflow over and
                      over, with small variations. Inefficiencies in the workflow really
                      start to drive you crazy when you hit them again and again and again
                      - whether you're a tester or user. Opportunities for improvement leap
                      out at you.

                      -- Alain:
                      I agree with this. I write a lot of unit tests, but I do a lot of manual
                      testing on top of it. And through that, I get a lot of excellent
                      insights into where the usability hotspots are, and not only for expert
                      users. Like you said, when you use the software over and over again,
                      even the lightest pain starts jumping at you.

                      Of course, I don't rely just on that for assessing usability of the
                      system. I also pay a lot of attention to customer feedback. Everytime
                      someone asks for help is treated as an occasion to get insights into
                      usability problems.
                      ----
                    • Desilets, Alain
                      Hope I didn t come across as slamming testers. ;-) Actually, it s not the tester, but the nature of the test I was getting at. You hit on it here. There s a
                      Message 10 of 23 , Jan 3, 2006
                      • 0 Attachment
                        Hope I didn't come across as slamming testers. ;-) Actually, it's
                        not the tester, but the nature of the test I was getting at. You hit
                        on it here. There's a difference between testing that the feature
                        performas as specified and actually using the product as intended and
                        making an often subjective judgement about how well it works. That's
                        the kind of testing that could be done in house by testers.

                        I have observed a pattern where testers become acutely aware of
                        inefficiencies in the workflow for the test cases they execute often
                        without regard for how realistically the test case reflects an actual
                        users' goals or usage.

                        -- Alain:
                        I for one, never use manual to repeatively test the same scenario.
                        That's something better reserved for machines (i.e. done through
                        automated unit testing).

                        When I talked about manual testing (in my response to Bryan's posting),
                        I was talking about exploratory testing, which, as I practice it
                        involves two things:

                        - trying to come up with creative ways to destroy the system
                        - using the system to carry out some realistic task (i.e. a task that a
                        a real user might do)

                        The second kind of testing does allow me to spot a lot of usability
                        problems at the design level, but also bugs that stand in the way of
                        usability.
                        ----
                      • Phlip
                        ... Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink
                        Message 11 of 23 , Jan 3, 2006
                        • 0 Attachment
                          Desilets, Alain wrote:

                          > Developper: Wait a minute... Wiring Ctrl-Z to History is just 1 hour of
                          > work on my part. And that includes testing, writing unit tests, and
                          > integrating into the main version. I could do it in my sleep!

                          Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid thing.

                          That's actually useful in some situations, because you can do "blink
                          comparator" from one keystroke.

                          Never mind.

                          --
                          Emily Litella
                        • Brian Marick
                          ... Agreed. Going beyond what you say: a tester without understanding of the domain will not only make bad usability suggestions, she will also report too many
                          Message 12 of 23 , Jan 3, 2006
                          • 0 Attachment
                            On Jan 3, 2006, at 2:18 PM, Jeff Patton wrote:
                            >
                            > I have observed a pattern where testers become acutely aware of
                            > inefficiencies in the workflow for the test cases they execute often
                            > without regard for how realistically the test case reflects an actual
                            > users' goals or usage.
                            >
                            > For example I used to do a lot of work in a brick and mortar retail
                            > environment. Testers often asked for features to create items to
                            > sell with less mandatory attributes, or navigate directly from item
                            > creation to transaction entry, or to easily remove items. While all
                            > these seemed logical, in large retail organizations those creating
                            > items weren't the ones entering transactions. And removing items
                            > with any transactional history had larger legal implications. The
                            > feature suggestions would have indeed improved workflow for the
                            > tester executing test cases, but not necessarily the user doing work.
                            >
                            > Now I know enlightened testers such as Brian and others wouldn't fall
                            > into this trap. But to help the less enlightened I've always found
                            > it important to have a strong understanding of the applications users
                            > and their workflow. A good user model and task model serve that
                            > purpose. Some concisely written user scenarios help with that also.
                            > It's hard to remember sometimes that we're not the user and the
                            > models help remind us.

                            Agreed. Going beyond what you say: a tester without understanding of
                            the domain will not only make bad usability suggestions, she will
                            also report too many bugs that don't matter to real users and too few
                            bugs that do.

                            As Michael Bolton and a host of others would instantly remind me, the
                            good tester is also operating under the assumption that the good user
                            model and good task model are wrong in some important ways, and that
                            part of her job is probably to find out how. (Similarly, her own
                            understanding is doubtless wrong, too. It's the Turtles of
                            Uncertainty all the way down.)


                            P.S. Making changes to improve the usability for testers does have
                            business value, unless their time costs nothing. I have some faith,
                            but no evidence, that it would also improve the code, much like
                            catering to JUnit or Fit does.

                            -----
                            Brian Marick, independent consultant
                            Mostly on agile methods with a testing slant
                            www.exampler.com, www.testing.com/cgi-bin/blog
                            Book in progress: www.exampler.com/book
                          • Desilets, Alain
                            Maybe PhotoShop defended the feature where Undo-Undo redoes the undid thing. That s actually useful in some situations, because you can do blink comparator
                            Message 13 of 23 , Jan 3, 2006
                            • 0 Attachment
                              Maybe PhotoShop defended the "feature" where Undo-Undo redoes the undid
                              thing.

                              That's actually useful in some situations, because you can do "blink
                              comparator" from one keystroke.

                              Never mind.

                              -- Alain:
                              Interesting point. Do you think this is something that you would be
                              likely to find out through upfront design and paper prototyping?

                              My guess is that this is exactly the kind of thing that does not become
                              apparent until (even to the end user) until you look at real users
                              laying their hands on the real thing and trying to carry out a real
                              task.
                              ----
                            • Jeff Patton
                              ... In the situation I was thinking about, making changes to the delivered product to better support testers would have broken business rules in the
                              Message 14 of 23 , Jan 3, 2006
                              • 0 Attachment
                                --- In agile-usability@yahoogroups.com, Brian Marick <marick@t...>
                                wrote:
                                > P.S. Making changes to improve the usability for testers does have
                                > business value, unless their time costs nothing. I have some faith,
                                > but no evidence, that it would also improve the code, much like
                                > catering to JUnit or Fit does.

                                In the situation I was thinking about, making changes to the delivered
                                product to better support testers would have broken business rules in
                                the application. Basically it would have made it easier/faster for the
                                tester to set up and tear down test data by forgoing some of the rules
                                around creating and deleting - and also installing quick navigation to
                                jump from creation directly to transaction entry, again, inapropriate
                                for our app, but workflow the testers did often.

                                Alain is right that lots of these tedious bits of testing should be
                                automated.

                                You're right that spending a little money to help testers complete
                                their work faster - or not go crazy doing it - is a good idea. We just
                                need to know why we're doing it. If it was a feature strictly for
                                testing, I'd want some way to hide or disable it from a production
                                release.

                                But, your point is well taken. Also Michael's point you referred to as
                                well.

                                thanks,

                                -Jeff
                              • Larry Constantine
                                ... Many bugs (both code bugs and usability defects) can be avoided by effective design practices and found prior to having running software through
                                Message 15 of 23 , Jan 4, 2006
                                • 0 Attachment
                                  Alain wrote:

                                  > By their nature, bugs cannot be caught by design. They can only be
                                  > caught after the buggy implementation has been implemented. This is one
                                  > of the great advantage of early delivery and TDD. They allow you to get
                                  > actual running software in the hands of users early, and catch bugs
                                  > early on in the process.

                                  Many bugs (both code bugs and usability defects) can be avoided by effective
                                  design practices and found prior to having running software through
                                  inspections and walkthroughs. Multiple research studies and extensive
                                  practice have shown inspections to be more cost effective than is testing
                                  for finding and eliminating bugs (both kinds).

                                  --Larry Constantine, IDSA [mailto:lconstantine@...]
                                    Chief Scientist | Constantine & Lockwood, Ltd.

                                  > -----Original Message-----
                                  > From: agile-usability@yahoogroups.com
                                  [mailto:agile-usability@yahoogroups.com] On Behalf
                                  > Of Desilets, Alain
                                  > Sent: Tuesday, 03 January 2006 11:40 AM
                                  > To: agile-usability@yahoogroups.com
                                  > Subject: RE: [agile-usability] just one bug's enough to make a program
                                  useless
                                  >
                                  > How can we usabilitists catch these blind spots in design, before they
                                  > lose us customers?
                                  >
                                  > -- Alain:
                                  > Sorry for late response... Catching up on pre-Xmas break email.
                                  >
                                  > ----
                                  >
                                  >
                                  >
                                  > Yahoo! Groups Links
                                  >
                                  >
                                  >
                                  >
                                Your message has been successfully submitted and would be delivered to recipients shortly.