Loading ...
Sorry, an error occurred while loading the content.

Field work vs Design

Expand Messages
  • Desilets, Alain
    It seems to me that much of what is discussed on this list falls more in the category of Design than Field work. I think Design is a good thing, as long as you
    Message 1 of 8 , Jul 11, 2006
    • 0 Attachment
      It seems to me that much of what is discussed on this list falls more in the category of Design than Field work.

      I think Design is a good thing, as long as you don't overdo it and don't' try to decide everything upfront. However, I am starting to think that some amount of Field Work may help tremendously too.

      I have recently started doing some field work, where I observe translators doing their work, in order to come up with tools that can help them in their work. I have found that to be invaluable in terms of helping me with Design.

      So far, every hour that I spend watching a translator at work has spawned at least one idea for a tool that could help them. Those are ideas that I would not have come up with if I hadn't observed the translator. Also, they are ideas the translators themselves would not have come up with, either because they don't know what the technology can and can't do, or because they never actually watched themselve work and thought about the technological implications of what they do. But when I describe the idea to translators and describe how it would help in specific scenarios I have witnessed, they usually go: "Wow! Never thought of that but yes, that would be really great!".

      The only problem with field work, is that it seems pretty slow. Or at least, it's pretty slow the way I do it. It takes me about 3 days to observe a subject and analyse the data. Mind you, I do it very systematically, using a Grounded Theory kind of approach because ultimately, I need to be able to publish my findings in scientific venues. I bet I could do it in one day or so. But still, there is the elapsed time issue because I need to schedule observation sessions with subjects and that takes time.

      So my question is this. Do people know how to do this kind of field work quickly? I know that Karen Holtzblatt has come up with a book called Rapid CD, but I haven't read it yet.


      Alain Désilets, MASc
      Agent de recherches/Research Officer
      Institut de technologie de l'information du CNRC /
      NRC Institute for Information Technology

      alain.desilets@...
      Tél/Tel (613) 990-2813
      Facsimile/télécopieur: (613) 952-7151

      Conseil national de recherches Canada, M50, 1200 chemin Montréal,
      Ottawa (Ontario) K1A 0R6
      National Research Council Canada, M50, 1200 Montreal Rd., Ottawa, ON
      K1A 0R6

      Gouvernement du Canada | Government of Canada
    • Desiree Sy
      ... [...] ... Yes, this type of work can be done more quickly. The approach that our team takes is a blended just-in-time set of incremental formative
      Message 2 of 8 , Jul 11, 2006
      • 0 Attachment
        > The only problem with field work, is that it seems pretty
        > slow. Or at least, it's pretty slow the way I do it. It takes
        > me about 3 days to observe a subject and analyse the data.
        [...]
        > So my question is this. Do people know how to do this kind of
        > field work quickly? I know that Karen Holtzblatt has come up
        > with a book called Rapid CD, but I haven't read it yet.

        Yes, this type of work can be done more quickly.

        The approach that our team takes is a blended just-in-time set
        of incremental formative usability investigations.

        (To give some context for these examples, we design complex 2D
        and 3D graphics software. It's shrinkwrapped.)

        So, for example, if we are at a customer site, and the current
        development cycle (that's what we call "sprints") is, say,
        Cycle 5, we might combine the following activities:

        - Usability test early design prototypes for Cycle 6+ (we then
        hand off validated design prototypes to developers in time
        for Cycle 6)
        - Usability test (with customer data) the cut-of-the-day,
        representing work up to Cycle 4, and some parts of Cycle 5
        work done by developers. (Incidentally, if you had asked me
        years ago if I'd ever bring the *cut-of-the-day* to a client
        site for testing, I'd have asked you what you were smoking...
        I love Agile)
        - Field work, in a focused area of interest determined by the
        highest priority design card or feature cards, for Cycle 8+.

        This is described in more detail in Lynn Miller's paper from
        Agile 2005: http://abstractics.com/reference/LynnMillersAliasReport.pdf

        Reporting is done the day after the visit. Some discussion of
        how reporting occurs is described in a presentation I gave at
        UPA 2005 called "Strategy & tactics for Agile design: a design
        case study." The high level version is that we report at scrum,
        and we have an area dedicated to the User Experience team where
        we use cards to track both usability/design issues and also
        items of interest from field work.

        In addition to this, we try and bring the field to us so to
        speak, using some techniques when actual users are coming
        in-house to us for usability testing.

        For example, as part of our screening criteria for usability
        testing, we ask potential testers to submit samples of their
        work using the intended area of design (or closest corresponding
        area, in the case of new functionality). We walk through these
        samples in a phone interview (where we also explain how to get
        here, and some logistics, etc.) We ask them to bring in the
        data file on a USB key.

        We then might use this file at the beginning of the test
        session to walk through some high-level workflows, and then
        drill down and demonstrate specific interface interactions in
        substantially more detail. Very often, in instances where we
        know that there are a number of different workflows that are
        possible test scenarios, this allows us (because a user has
        demonstrated that they use a particular workflow) to select
        a better or more representative usability scenario in the
        subsequent test.

        Thus, are combining a form of contextual investigation into
        a usability test session.

        This is described in more detail in a presentation I gave at
        UPA 2006, "Formative usability investigations for open-ended
        tasks" although I don't really go into the Agile-specific
        pieces in much detail.

        It is definitely not only possible to conduct field work in
        an Agile context, but critical to a context-rich design to
        do so.

        -Desirée
        _____________________________________________________
        Desirée Sy Sr. Interaction Designer
        work: (416) 874-8296 User Experience Group
        fax: (416) 369-6150 Autodesk, Inc (formerly Alias)
      • Desilets, Alain
        Thx Désirée, this is very helpful. I ll definitely read the papers you mention. I m trying to get a better feel for how upfront the fieldwork is done, and
        Message 3 of 8 , Jul 12, 2006
        • 0 Attachment
          Message

           

          Thx Désirée, this is very helpful. I'll definitely read the papers you mention.

          I'm trying to get a better feel for how upfront the fieldwork is done, and how much effort is put into it vs development.

          If I understand you correctly, you are saying that the field work is about 3 cycles ahead of development, right? How long are your cycles? How many cycles would you have in a typical project?

          Also, it seems like you guys are doing a LOT of U* work (kudo to you guys and also to the Alias managers who support it). Can you give us an idea of how what is the ratio between development work vs U*? Also, within the U* effort, how much of that is spent on doing field-work, versus design work, versus testing running software?

          How many U* people typically work on

           Alain

          .

        • Desiree Sy
          ... Our Agile teams have the concept of a Cycle Zero where some preliminary work is done to align the entire team on the overall direction of a project. This
          Message 4 of 8 , Jul 12, 2006
          • 0 Attachment
            > I'm trying to get a better feel for how upfront the
            > fieldwork is done, and how much effort is put into it vs
            > development.

            Our Agile teams have the concept of a "Cycle Zero" where some
            preliminary work is done to align the entire team on the
            overall direction of a project. This is generally not as
            "heavy" as a similar stage in a non-Agile project might be.
            There may be brief field work associated with this, but in
            most cases, since we work on software which versions, we will
            have *already* completed the field work, so the necessary
            data to feed into this stage will have been done.

            We also use product validation research to verify whether or
            not to go ahead with a product concept prior to beginning a
            project. Investigations in the field are a critical component
            of this.

            > If I understand you correctly, you are saying that the
            > field work is about 3 cycles ahead of development, right?

            I was approximating for the sake of clarity. The degree of
            design granularity is going to determine how close or how far
            in advance the field work occurs. It could be that there's a
            quick issue for a design one cycle ahead that you can add to
            a field visit for investigation. In any given project, there
            are usually one or two very large designs (per UX person) that
            require a more significant investigation that will span many
            cycles of work. The key is to break these large designs into
            incremental mini-prototypes -- and, correspondingly, to break
            the field investigation components into incremental mini-bites.

            > How
            > long are your cycles? How many cycles would you have in a
            > typical project?

            What's a typical project? :) Different teams have slightly
            different takes. The cycles on the main projects I've worked
            on were 2 weeks (in the initial architectural stages of the
            first 2 months, perhaps a little longer). I know that other
            Agile projects in the company have had longer cycles of 3-4
            weeks. Projects tend to be about a year in length, plus or
            minus a bit.

            But, of course, the real answer is the same one you will always
            receive from a designer, which is: It depends.

            > Also, it seems like you guys are doing a LOT of U* work
            >(kudo to you guys and also to the Alias managers who support
            > it). Can you give us an idea of how what is the ratio between
            > development work vs U*? Also, within the U* effort, how much
            > of that is spent on doing field-work, versus design work, versus
            > testing running software?

            The way that the UX team works here is unique, as far as I've
            been able to determine from conversations with other
            practitioners at conferences like UPA and DUX.

            First, all interaction designers do all the core activities of
            design: we do user ethnography (which is more broad than "field
            work", as it includes other data elicitation methods), we design
            prototypes, and we verify those prototypes with usability testing.
            So we don't have a usability group that is separate from a design
            group. To answer your last question about the split between these
            activities... sorry, but it depends. Obviously, different designs
            have different requirements.

            The overall answer is that our goal is to spend the minimal
            required effort on each stage... but no *less* than that!

            The other big difference is that our team works from the very
            beginning of a project to the point it's released, and UX team
            members are assigned in pairs to a given project. The corollary
            to this is that, obviously, there are a lot of projects that
            have no UX input at all. The decision about whether or not we
            will work on a project is a strategic one; we tend to work on
            projects with what I'll call a higher design risk or weighting.
            (For example, a project with an innovative or unusual type of
            UI interaction, or an application where the user experience is
            the key competitive differentiator.)

            One more thing I should clear up because I might have misled
            unintentionally, is that when we are testing implemented code,
            we are *usability testing* it -- we are not doing QA testing!
            We have great QA guys who are QA testing. We can, however,
            often help by bringing from the field to our QA staff better
            and more representative test cases, and in some lovely cases,
            even end-user data for test cases. Since QA teams attend our
            scrums, they'll get the field work data with the developers,
            the day after the visit.

            -Desirée
          • Desilets, Alain
            Another tought on this. As much as I feel the value of field work, I am aware that it can only focus on one particular part of the problem at a time. You gain
            Message 5 of 8 , Jul 13, 2006
            • 0 Attachment
              Message
              Another tought on this.
               
              As much as I feel the value of field work, I am aware that it can only focus on one particular part of the problem at a time. You gain a lot of depth in understanding a particular user, or a particular department that will be using the system, but you don't have much breadth.
               
              I wonder if there is a way to get both the breadth and depth by running a collaborative focus group with a bunch of people who are in the field all the time (users, customer support, marketing, etc...) together and involve them in a collaborative fact finding exercise.
               
              I am quite aware of the dangers of focus groups. Often people will talk about stuff that they think they need as opposed to stuff they actually need. But I wonder if you couldn't avoid this kind of problem by forcing people to ground what they say in actual experience and to somehow quantify how significant or important the statement is.
               
              For example, for a focus group on intranet software, you might ask people to tell you about the problems they have with intranet software, and insist that they phrase it in precise terms like:
               
              - "Several times a month, I ask my sysadmin to setup a new private space for a group of us on the intranet. It always takes days before it's up and running. He tells me it takes him about 2 hours to set one up, and he's really busy."
               
              - "Most people in my entourage find that it takes too long to post stuff on the intranet (eventhough it only takes about 2 minutes to do). So they don't contribute to it much."
               
              You could then treat these statements as though they were notes taken from an actual field study, and do qualitative analysis on them.
               
              Have people had experience with this kind of approach? What are the pros and cons?
               
              Alain
               
               
            • Jon Meads
              Alain, The way you broaden the scope is to visit with more users. You want to see how they do their work and it s best to get a broad spectrum that covers
              Message 6 of 8 , Jul 13, 2006
              • 0 Attachment
                Alain,
                 
                The way you broaden the scope is to visit with more users. You want to see how they do their work and it's best to get a broad spectrum that covers different environments and user roles. If there are budget or time problems, then try to cut down on the amount of time you spend with each user and only stick with a few into any depth. I find that you can usually get plenty of information by spending just an hour or two with each user. Like anything else, you need to plan out the user study according to budget and time constraints but it's a good bet that a week of good user study could eliminate several weeks of refactoring time. It all goes back to risk and how comfortable you are with your understanding and knowledge of the various users, their characteristics, their usage roles, and their environments of use.
                 
                Also, there's no law against going back out to visit users again if you come up with questions that come up that you don't have answers for from the initial studies. The main thing is to sufficiently understand the users and their needs so that you can create a product that will work for them.
                 
                But I would not use focus groups to get usage information. Focus groups are good for understanding value. But if you want to understand how people do their work, you really need to go and observe them doing their work.
                 
                Cheers,
                jon


                From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Desilets, Alain
                Sent: Thursday, July 13, 2006 6:45 AM
                To: agile-usability@yahoogroups.com
                Subject: [agile-usability] Field work vs Focus Groups

                Another tought on this.
                 
                As much as I feel the value of field work, I am aware that it can only focus on one particular part of the problem at a time. You gain a lot of depth in understanding a particular user, or a particular department that will be using the system, but you don't have much breadth.
                 
                I wonder if there is a way to get both the breadth and depth by running a collaborative focus group with a bunch of people who are in the field all the time (users, customer support, marketing, etc...) together and involve them in a collaborative fact finding exercise.
                 
                I am quite aware of the dangers of focus groups. Often people will talk about stuff that they think they need as opposed to stuff they actually need. But I wonder if you couldn't avoid this kind of problem by forcing people to ground what they say in actual experience and to somehow quantify how significant or important the statement is.
                 
                For example, for a focus group on intranet software, you might ask people to tell you about the problems they have with intranet software, and insist that they phrase it in precise terms like:
                 
                - "Several times a month, I ask my sysadmin to setup a new private space for a group of us on the intranet. It always takes days before it's up and running. He tells me it takes him about 2 hours to set one up, and he's really busy."
                 
                - "Most people in my entourage find that it takes too long to post stuff on the intranet (eventhough it only takes about 2 minutes to do). So they don't contribute to it much."
                 
                You could then treat these statements as though they were notes taken from an actual field study, and do qualitative analysis on them.
                 
                Have people had experience with this kind of approach? What are the pros and cons?
                 
                Alain
                 
                 

              • Desilets, Alain
                But I would not use focus groups to get usage information. Focus groups are good for understanding value. But if you want to understand how people do their
                Message 7 of 8 , Jul 13, 2006
                • 0 Attachment
                  Message
                  But I would not use focus groups to get usage information. Focus groups are good for understanding value. But if you want to understand how people do their work, you really need to go and observe them doing their work.
                   
                  Cheers,
                  jon
                   
                  -- Alain:
                  That's interesting. Can you elaborate on that? I guess "value" is sort of getting at the kind of broader thing that might be missed by just looking at a small number of people are doing.
                  ----
                   


                   
                  .

                • Jon Meads
                  Bottomline, value means how they prioritize features (which may or may not affect usability) and how much they would pay. Focus groups can also be very
                  Message 8 of 8 , Jul 13, 2006
                  • 0 Attachment
                    Bottomline, "value" means how they prioritize features (which may or may not affect usability) and how much they would pay.  Focus groups can also be very misleading unless facilitated by an expert who understands how to run them and what is possible to get from them - which is about 20% of the people who are in the business of conducting focus groups.
                     
                    Cheers,
                    jon


                    From: agile-usability@yahoogroups.com [mailto:agile-usability@yahoogroups.com] On Behalf Of Desilets, Alain
                    Sent: Thursday, July 13, 2006 10:04 AM
                    To: agile-usability@yahoogroups.com
                    Subject: RE: [agile-usability] Field work vs Focus Groups

                    But I would not use focus groups to get usage information. Focus groups are good for understanding value. But if you want to understand how people do their work, you really need to go and observe them doing their work.
                     
                    Cheers,
                    jon
                     
                    -- Alain:
                    That's interesting. Can you elaborate on that? I guess "value" is sort of getting at the kind of broader thing that might be missed by just looking at a small number of people are doing.
                    ----
                     


                     
                    .

                  Your message has been successfully submitted and would be delivered to recipients shortly.