Loading ...
Sorry, an error occurred while loading the content.

from qualitative user testing to quantitative user research

Expand Messages
  • Stefan Wobben [Concept7]
    On a regular basis (weekly) our company performs qualitative user testing with eyetracking. I want to gather data on a aggregate level. A study typically
    Message 1 of 6 , Sep 30, 2008
    • 0 Attachment
      from qualitative user testing to quantitative user research

      On a regular basis  (weekly) our company performs qualitative user testing with eyetracking.

      I want to gather data on a aggregate level.

      A study typically consists of one webiste, 3 to 5 tasks (from orientation tot specific search) with 6 users. The usergroup usually is divers, which means there are differences between gender, age, internet experience and level of education.

      During a study we measure fixations, facial expression, verbal comments, screencast and an observer makes notes (coding scheme is positive, remarkable, problem).

      I wonder if there is a way that i can gather data on a meta-level so in the end we are able to perform statitic analyses. The problem however is we there are different websites, different tasks and different participants.

      On the other hand people have several things in common and websites have several features in common.

      If i want to make a statement 'user understand that blue underlined text is a hyperlink' i can do this with different websites, different tasks and different studies because click on the hyperlink is what they all share.

      Bryan and Jeffrey Eisenberg often speak of 4-buying modalities. This is a user segmentation on basis of MBTI personality traits. I would like to gather data if the 4 types search and read different information types and have other differences in their surfing behavior.

      Another thing i would like to examine in more detail is if there are differences in surfing behavior and gender type.

      What should i change on our current method of testing to be able to gather data to be able to perform statistical analyses and get more insight.

      If spoken to two researcher who can't seem to help me. There are too many differences for a succesfull experiment.

      I can't believe there isn't any other possibility. Does anybody has ideas? and/or interested in performing such type of studies?

      Kind regards

      Stefan Wobben
      stefan@...

    • Andy Edmonds
      Stefan: You d probably find this topic more productive on a different list, but there are some clear precedents for what you seek. For example, User Interface
      Message 2 of 6 , Oct 1, 2008
      • 0 Attachment
        Stefan: You'd probably find this topic more productive on a different
        list, but there are some clear precedents for what you seek.

        For example, User Interface Engineering (UIE), has done some nice high
        profile work across studies to show that longer links tend to perform
        better. IIRC, they correlated task success with link length across a
        bunch of studies. Read more at
        http://www.uie.com/reports/scent_of_information/

        In general, you're looking at "between subjects design" and a very
        large error component. You'll need a lot of data.

        The UIE example shows how you can apply a method of reducing each
        study to a datapoint and then analyzing across studies as an experiment.

        hth,
        Andy Edmonds
        http://surfmind.com

        --- In agile-usability@yahoogroups.com, "Stefan Wobben [Concept7]"
        <stefan@...> wrote:
        ...
        > I wonder if there is a way that i can gather data on a meta-level so
        in the end we are able to perform statitic analyses. The problem
        however is we there are different websites, different tasks and
        different participants.
        ...
        > I can't believe there isn't any other possibility. Does anybody has
        ideas? and/or interested in performing such type of studies?
      • James Page
        ... overview of which method to use. Designed for medical researches but most of methods apply to usability. Also look at the website
        Message 3 of 6 , Oct 1, 2008
        • 0 Attachment

          Get hold of a copy of "Statistics in Medicine". It gives you a good overview of which method to use. Designed for medical researches but most of methods apply to usability. 
           
          Also look at the website http://www.measuringusability.com/ for a good overview of statistics and usability. 

          I would agree that you need lots of data. 

          James


           
          On Wed, Oct 1, 2008 at 11:41 AM, Andy Edmonds <andyed@...> wrote:

          Stefan: You'd probably find this topic more productive on a different
          list, but there are some clear precedents for what you seek.

          For example, User Interface Engineering (UIE), has done some nice high
          profile work across studies to show that longer links tend to perform
          better. IIRC, they correlated task success with link length across a
          bunch of studies. Read more at
          http://www.uie.com/reports/scent_of_information/

          In general, you're looking at "between subjects design" and a very
          large error component. You'll need a lot of data.

          The UIE example shows how you can apply a method of reducing each
          study to a datapoint and then analyzing across studies as an experiment.

          hth,
          Andy Edmonds
          http://surfmind.com

          --- In agile-usability@yahoogroups.com, "Stefan Wobben [Concept7]"
          <stefan@...> wrote:
          ...


          > I wonder if there is a way that i can gather data on a meta-level so
          in the end we are able to perform statitic analyses. The problem
          however is we there are different websites, different tasks and
          different participants.
          ...

          > I can't believe there isn't any other possibility. Does anybody has
          ideas? and/or interested in performing such type of studies?


        • Stefan
          Thank you Andy. Thank you James. I ve ordered scent of information and statistics in medicine so i can some research on the topic. As the remark you need lots
          Message 4 of 6 , Oct 1, 2008
          • 0 Attachment
            Thank you Andy.
            Thank you James.

            I've ordered scent of information and statistics in medicine so i can some research on the
            topic.

            As the remark 'you need lots of data' well thats what we have :)

            I'll keep you updated about my progress.
          • Jared Spool
            ... I hope you won t be disappointed with the Scent of Information report. It doesn t really go into our methodology. That s not why we created it. What you re
            Message 5 of 6 , Oct 1, 2008
            • 0 Attachment
              On Oct 1, 2008, at 4:09 PM, Stefan wrote:

              > I've ordered scent of information and statistics in medicine so i
              > can some research on the
              > topic.

              I hope you won't be disappointed with the Scent of Information report.
              It doesn't really go into our methodology. That's not why we created it.

              What you're trying to do is *really* hard and will take huge amounts
              of preparation and some massively impressive statistics horsepower. I
              know because we've tried multiple times and only managed to pull it
              off some of the time.

              There are scaling and ground issues. Ground issues are about
              connecting the independent data collection points to a common element,
              so you can say that these measures are talking about the same thing.
              Scaling issues are about ensuring that Good/Bad in one scale mean the
              same as Good/Bad in another scale.

              This is very heavy duty science you're trying to do. I'm betting, if
              you want to do it well, it will triple-to-quintuple your research costs.

              I'm betting that's why you've not found a lot of resources on it. You
              need a deep pocket to make it really work.

              Jared

              Jared M. Spool
              User Interface Engineering
              510 Turnpike St., Suite 102, North Andover, MA 01845
              e: jspool@... p: +1 978 327 5561
              http://uie.com Blog: http://uie.com/brainsparks
            • Stefan
              Jared, Have you read the usability problem taxonomy ? of Keenan, Hartson, Kafura and Schulman? What do you think of using this classification scheme for
              Message 6 of 6 , Oct 12, 2008
              • 0 Attachment
                Jared,

                Have you read 'the usability problem taxonomy'? of Keenan, Hartson, Kafura and
                Schulman? What do you think of using this classification scheme for between-subjects
                studies?

                Stefan

                --- In agile-usability@yahoogroups.com, Jared Spool <jspool@...> wrote:
                >
                >
                > On Oct 1, 2008, at 4:09 PM, Stefan wrote:
                >
                > > I've ordered scent of information and statistics in medicine so i
                > > can some research on the
                > > topic.
                >
                > I hope you won't be disappointed with the Scent of Information report.
                > It doesn't really go into our methodology. That's not why we created it.
                >
                > What you're trying to do is *really* hard and will take huge amounts
                > of preparation and some massively impressive statistics horsepower. I
                > know because we've tried multiple times and only managed to pull it
                > off some of the time.
                >
                > There are scaling and ground issues. Ground issues are about
                > connecting the independent data collection points to a common element,
                > so you can say that these measures are talking about the same thing.
                > Scaling issues are about ensuring that Good/Bad in one scale mean the
                > same as Good/Bad in another scale.
                >
                > This is very heavy duty science you're trying to do. I'm betting, if
                > you want to do it well, it will triple-to-quintuple your research costs.
                >
                > I'm betting that's why you've not found a lot of resources on it. You
                > need a deep pocket to make it really work.
                >
                > Jared
                >
                > Jared M. Spool
                > User Interface Engineering
                > 510 Turnpike St., Suite 102, North Andover, MA 01845
                > e: jspool@... p: +1 978 327 5561
                > http://uie.com Blog: http://uie.com/brainsparks
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.