1173Re: users don't self-report reliably
- May 5, 2005It just dawned on me why I wouldn't have noticed or cared about the
first example you gave, Hugh --- I show up to write use cases, and we
need to write a use case for any system function whether it is used
hourly or once a blue moon. This is a crucial difference
in "requirements gathering" (what exactly should the system do)
and "envisioning" and "usability" (how and how much does it improve
the life of the user/organization).
In envisioning and usability, frequency and the related ROI matter
hugely. In writing a functional spec, we just need to know that it
gets done ever. Thus, if users tell me this use case is the 60% item,
I'll make a note of that for other people on the project to worry
about, and collect the use case. If they tell me this use case is
just to handle the odd-ball situation, I'll make a note of that for
other people on the project to worry about, and collect the use
case. My goal is to capture /all/ and not miss one. Your goal is to
identify the ROI factor.
The second case below is different. That falls into
my "capture /all/" bucket, and so I am all ears listening and
questioning, looking for all the infrequent, oddball, not-officially-
supported pathways. That's where I get a clam-level tipoff... and at
that point I head straight for the surface and above, asking them
what is on their minds, why they would be doing that --- I don't try
to deduce what is on their minds from the clam-level data - I simply
ask them. And my experience here (without having any other reference
point) is that they DO report reliably in this situation. ...
Consider, if you will the second case you report on: "we saw people
doing something called "roll transfers"-tossing a stack of wafers
from one carrier to another without using the machine that was
especially designed for the purpose-in violation of procedure. We
might, with good facilitation have discovered that people break the
rules this way. We would not, I think, have understood why people
risk their jobs (or at least their performance appraisals) to do this
kind of transfer when they don't have to. "
So I'll offer that the reverse is true of what you just wrote: that
using facilitatin it is hard to DETECT that situation - role playing
and observation are helpful here - but having detected it, we can
UNDERSTAND the 'why' using facilitation.
Here's my matching example... we're talking about giving medications
in a hospital using a new barcode reader gadget. We're showing some
draft use cases to a team that had done a beta test with some hand-
held / bar code reader device for a month. We're talking about
scanning the meds at the bed side, and one of the nurses says, "Do we
have to be at the bedside? I used to carry a bar code of patient in
my pocket and scan everything in the prep room..." We all got stuck
at that moment, because it was clear that she was circumventing
something fundamental in the way that system had been set up (to be
done only at the patient bedside). After some hemming and hawing in
the room, we asked her, "Why would you be doing that? If we're going
to design a new system, it should support what you're trying to do,
so what were you trying to do?" And she said, "It's to save face in
front of the patient - I don't want to get to the bed and find I have
the wrong dose or something." After which, the other nurses agreed to
What was serendipidous was her mentioning it. What wasn't
serendipidous was her being able to say why she was doing it. I don't
think we needed weeks of observation and note taking to deduce this;
it was enough just to ask her.
We have since written a use case called "Optionally pre-check the
meds", and are finding it popular among the nursing staff in the
various hospitals once they read and are reminded what it is for
(that there is no actual med administration going on, it is only
for "peace of mind"). And it has triggered in their minds various
situations that matter to us about where they might be standing with
this that or the other medication needing this that or the other bit
of procedure, so that we can write the requirements to support ALL
the places and behaviors they will need (however rare), while
prohibiting the specific ones the law and the hospital prohibits.
Reminder: we were lucky in this, because we got to talk to a team
that had used a scanner for a month, so we had their memories to work
with. I don't think we would have discovered this 'optionally
recheck' use case without those memories. The people who originally
designed that scanner system didn't have that luxury.
Thus I motivate the need for clam-level details, and that these are
hard to come by in facilitated sessions (ergo, the facilitator has to
deliberately work out ways to get clam-level stuff to pop out).
However, I argue that facilitation works fine for discovering the
Why, and that self-reporting is quite reliable here.
--- In email@example.com, "Hugh Beyer" <beyer@i...>
> On practically the first project we worked on as a business, theuser
> population consisted of system managers, who uniformly reportedthat 60-70%
> of their work was focused on diagnosis and troubleshooting offailing
> systems. CI field interviews revealed that maybe 10-20% of theirwork was
> diagnosis and troubleshooting. Most of their time was spent onroutine
> maintenance tasks. We've found this is typical-users can't tell youhow they
> allot their time really-they put too much emphasis on what isthe "real"
> job, the interesting, valuable, knowledge work, and overlook thetime spent
> on "unimportant" parts of the job.how
> On another project, we were studying chip fabrication. We were told
> rigid all their requirements were for handling the wafers and howcarefully
> they were followed. But when we observed, we saw people doingsomething
> called "roll transfers"-tossing a stack of wafers from one carrierto
> another without using the machine that was especially designed forthe
> purpose-in violation of procedure. We might, with good facilitationhave
> discovered that people break the rules this way. We would not, Ithink, have
> understood why people risk their jobs (or at least their performance
> appraisals) to do this kind of transfer when they don't have to.
- << Previous post in topic Next post in topic >>