Loading ...
Sorry, an error occurred while loading the content.
 

Re: [webanalytics] Re: Multivariate testing: must a control group size match test size?

Expand Messages
  • Dylan Lewis
    Hi, For those of us that are wondering about this in A/B tests, there is a nice short paragraph about assigning equal users to treatments. It is located in
    Message 1 of 10 , Oct 15, 2008
      Hi,

      For those of us that are wondering about this in A/B tests, there is a nice
      short paragraph about assigning equal users to treatments.

      It is located in section 6.2.4, of the paper located here:
      http://exp-platform.com/hippo_long.aspx

      This paper is a great read!

      Thanks,

      Dylan
      (Moderator, Analyst@Intuit)





      On Wed, Oct 15, 2008 at 8:47 AM, JT Buser <jbuser@...> wrote:

      > Dave,
      >
      > Agree about bringing in someone to design, analyze and interperet the
      > test. I always say, about 95% (I am a stats guy, I can make up
      > numbers) of the work in doing an analysis lies within setting it up
      > correctly (identifying factors, controlling for extraneous factors,
      > etc.) so that you are getting the answer you want. Always comes down
      > to Should I vs. Can I....Yes, I CAN always do an analysis. SHOULD I,
      > is a whole different ball game. In order to do so, one must
      > understand exactly what models are available, but letting the question
      > dictate the model, not the other way around.
      >
      > JT
      >
      >
      > --- In webanalytics@yahoogroups.com <webanalytics%40yahoogroups.com>,
      > "\"Wandering\" Dave Rhee"
      > <wdaveonline@...> wrote:
      > >
      > > Hey, JT -- awesome demo videos for your A/B calculators! I
      > especially
      > > like the power analysis setting in the advanced calculator, very
      > nice,
      > > as is the backwards-calculation of the actual confidence interval.
      > > (Ophir -- I see your calculator also has this feature -- nice to
      > have
      > > it in an easy-to-bookmark web page!)
      > >
      > > JT -- good points about using the vendor's MVT tools -- one would
      > hope
      > > that they're already using an appropriate analysis for web data
      > (e.g.,
      > > one-tailed, appropriate distribution curve, etc.), but it's always
      > > nice to have an independent pair of eyes validate the tool, if you
      > > have access to such a resource. I think for something like MVT,
      > it's
      > > well worth hiring an outside consultant to help design, analyze, and
      > > interpret the first few tests, just to help you step past some of
      > the
      > > most obvious mistakes. I think beyond that, it's relatively simple
      > to
      > > do "another test like the last one" on one's own.
      > >
      > > WDave
      > >
      >
      >
      >


      [Non-text portions of this message have been removed]
    • Gerry
      Not a statitician either, but there is more to this than has been stated - basically to be confident in the output you need to ensure that the numbers are
      Message 2 of 10 , Oct 16, 2008
        Not a statitician either, but there is more to this than has been
        stated - basically to be confident in the output you need to ensure
        that the numbers are down to the difference.

        Groups do NOT need to be the same size but you must get a reasonable
        conversion in both for it to be significantly significant - i.e. if
        your conversion is 2% and you test only 100 people the % will be
        heavily influenced by one or two sales.

        It is quite common for the test group to be sizably smaller - i.e. 10
        or 20% of the population, this is assuming that this size group will
        return large enough numbers and the conversion will be significant
        enough to be a true representation, if the hard conversion isn't
        giving you enough data, also try a softer conversion such as "add to
        basket" or getting to a funnel point higher than the actual purchase,
        if they go into the same funnel and the test has no relation to the
        final steps in the funnel you might be able to test this point of
        conversion and have far greater confidence in the outcome.

        Split 50/50 are in some way ideal, but if early results indicate an
        improvement in funnel B I would be tempted to go 80% funnel B and 20%
        A if that makes sense.

        G



        --- In webanalytics@yahoogroups.com, "Michael D. Greenberg"
        <mdgberg99@...> wrote:
        >
        > There are two factors that impact an appropriate control group
        size: The size of your test group and your estimated lift in
        response in the test group vs. control.
        >
        > The larger your test group, the smaller a control group can be in
        relation to the test group (and vice versa).
        > The smaller the difference in expected response between the test
        and control groups, the larger a sample must be.
        >
        > So if you expect very small differences in response (say 1.1% in
        the test group vs. a typical 1.0% in the control group), you need a
        pretty large control group to detect a significant difference in
        behavior.
        >
        > While it gets much more complicated in multivariate, the basic
        relationship above will still hold true.
        >
        > --- On Wed, 10/15/08, fredeilam <prusak@...> wrote:
        >
        > > From: fredeilam <prusak@...>
        > > Subject: [webanalytics] Re: Multivariate testing: must a control
        group size match test size?
        > > To: webanalytics@yahoogroups.com
        > > Date: Wednesday, October 15, 2008, 6:50 AM
        > > There is a difference between could, should and must :)
        > >
        > > I'm not a statistics expert, but intuitively, the
        > > control and test
        > > groups *should* be the same size.
        > >
        > > So I ran some numbers to test my theory.
        > >
        > > A while back I created an online conversion rate confidence
        > >
        > > calculator:
        > >
        > > http://www.prusak.com/tools/conversion-confidence-calculator/
        > >
        > > I think the numbers speak for themselves:
        > >
        > > Lets say group B is currently showing a 15% improvement
        > > over group A.
        > >
        > > Option 1:
        > > Group A 3000 visits. 300 conversions
        > > Group B 3000 visits. 345 conversions
        > > Confidence Level: 93% <- highest
        > >
        > >
        > > Option 2:
        > > Group A 2000 visits. 200 conversions
        > > Group B 4000 visits. 460 conversions
        > > Confidence Level: 92% <- slightly lower
        > >
        > >
        > > Option 3:
        > > Group A 2000 visits. 200 conversions
        > > Group B 3000 visits. 345 conversions
        > > Confidence Level: 90% <- lowest
        > >
        > > - Ophir
        > >
        > >
        > > --- In webanalytics@yahoogroups.com, "Kevin
        > > Higgins"
        > > <kevinjhiggins@> wrote:
        > > >
        > > > Hi everyone,
        > > >
        > > > I'm new to MTV and wonder if you could help with a
        > > question. Must
        > > the size
        > > > of a control group always match testing groups? For
        > > example, if each
        > > of the
        > > > testing groups are 5,000 should the control be of
        > > relative same size
        > > (5,000
        > > > as well). Could there be a lesser number for the
        > > control which
        > > would still
        > > > lend statistical relevance?
        > > >
        > > > Kevin
        > > >
        > > >
        > > > [Non-text portions of this message have been removed]
        > > >
        > >
        > >
        > >
        > >
        > > ------------------------------------
        > >
        > > ---------------------------------------
        > > The Web Analytics Forum
        > > Founded by Eric T. Peterson
        > > (www.webanalyticsdemystified.com)
        > > Moderated by the Web Analytics Association
        > > (www.webanalyticsassociation.org)
        > > Email moderators at:
        > > webanalytics-moderators@...! Groups Links
        > >
        > >
        > >
        >
      • Andy Edmonds
        This has been a good dialogue so far, but I think it s worth reiterating that most statistical techniques assume equal variance in the compared samples.
        Message 3 of 10 , Oct 16, 2008
          This has been a good dialogue so far, but I think it's worth
          reiterating that most statistical techniques assume "equal variance"
          in the compared samples.

          Ronny Kohavi & the ExP team at Microsoft have a good discussion of
          this in the 7 Pitfalls paper
          (http://exp-platform.com/ExPpitfalls.aspx) though you might find the
          Pratical guide paper (http://exp-platform.com/hippo.aspx) or video
          (http://videolectures.net/kdd07_kohavi_pctce/) a better starting point.

          The term "heteroscadisticity" refers to the situation of unequal
          variances. While there are a lot of concerns here, the answer to the
          question "should my control be the same size as my treatment" is
          largely yes because you need the variances of the two samples to be
          the same for basic apples to apples type reasons.

          hth, Andy
          Related blog post:
          http://alwaysbetesting.com/abtest/index.cfm/2008/3/16/Getting-Serious-About-Testing-Learn-from-the-Pros
        Your message has been successfully submitted and would be delivered to recipients shortly.