- Hi,

For those of us that are wondering about this in A/B tests, there is a nice

short paragraph about assigning equal users to treatments.

It is located in section 6.2.4, of the paper located here:

http://exp-platform.com/hippo_long.aspx

This paper is a great read!

Thanks,

Dylan

(Moderator, Analyst@Intuit)

On Wed, Oct 15, 2008 at 8:47 AM, JT Buser <jbuser@...> wrote:

> Dave,

>

> Agree about bringing in someone to design, analyze and interperet the

> test. I always say, about 95% (I am a stats guy, I can make up

> numbers) of the work in doing an analysis lies within setting it up

> correctly (identifying factors, controlling for extraneous factors,

> etc.) so that you are getting the answer you want. Always comes down

> to Should I vs. Can I....Yes, I CAN always do an analysis. SHOULD I,

> is a whole different ball game. In order to do so, one must

> understand exactly what models are available, but letting the question

> dictate the model, not the other way around.

>

> JT

>

>

> --- In webanalytics@yahoogroups.com <webanalytics%40yahoogroups.com>,

> "\"Wandering\" Dave Rhee"

> <wdaveonline@...> wrote:

> >

> > Hey, JT -- awesome demo videos for your A/B calculators! I

> especially

> > like the power analysis setting in the advanced calculator, very

> nice,

> > as is the backwards-calculation of the actual confidence interval.

> > (Ophir -- I see your calculator also has this feature -- nice to

> have

> > it in an easy-to-bookmark web page!)

> >

> > JT -- good points about using the vendor's MVT tools -- one would

> hope

> > that they're already using an appropriate analysis for web data

> (e.g.,

> > one-tailed, appropriate distribution curve, etc.), but it's always

> > nice to have an independent pair of eyes validate the tool, if you

> > have access to such a resource. I think for something like MVT,

> it's

> > well worth hiring an outside consultant to help design, analyze, and

> > interpret the first few tests, just to help you step past some of

> the

> > most obvious mistakes. I think beyond that, it's relatively simple

> to

> > do "another test like the last one" on one's own.

> >

> > WDave

> >

>

>

>

[Non-text portions of this message have been removed] - Not a statitician either, but there is more to this than has been

stated - basically to be confident in the output you need to ensure

that the numbers are down to the difference.

Groups do NOT need to be the same size but you must get a reasonable

conversion in both for it to be significantly significant - i.e. if

your conversion is 2% and you test only 100 people the % will be

heavily influenced by one or two sales.

It is quite common for the test group to be sizably smaller - i.e. 10

or 20% of the population, this is assuming that this size group will

return large enough numbers and the conversion will be significant

enough to be a true representation, if the hard conversion isn't

giving you enough data, also try a softer conversion such as "add to

basket" or getting to a funnel point higher than the actual purchase,

if they go into the same funnel and the test has no relation to the

final steps in the funnel you might be able to test this point of

conversion and have far greater confidence in the outcome.

Split 50/50 are in some way ideal, but if early results indicate an

improvement in funnel B I would be tempted to go 80% funnel B and 20%

A if that makes sense.

G

--- In webanalytics@yahoogroups.com, "Michael D. Greenberg"

<mdgberg99@...> wrote:>

size: The size of your test group and your estimated lift in

> There are two factors that impact an appropriate control group

response in the test group vs. control.>

relation to the test group (and vice versa).

> The larger your test group, the smaller a control group can be in

> The smaller the difference in expected response between the test

and control groups, the larger a sample must be.

>

the test group vs. a typical 1.0% in the control group), you need a

> So if you expect very small differences in response (say 1.1% in

pretty large control group to detect a significant difference in

behavior.>

relationship above will still hold true.

> While it gets much more complicated in multivariate, the basic

>

group size match test size?

> --- On Wed, 10/15/08, fredeilam <prusak@...> wrote:

>

> > From: fredeilam <prusak@...>

> > Subject: [webanalytics] Re: Multivariate testing: must a control

> > To: webanalytics@yahoogroups.com

> > Date: Wednesday, October 15, 2008, 6:50 AM

> > There is a difference between could, should and must :)

> >

> > I'm not a statistics expert, but intuitively, the

> > control and test

> > groups *should* be the same size.

> >

> > So I ran some numbers to test my theory.

> >

> > A while back I created an online conversion rate confidence

> >

> > calculator:

> >

> > http://www.prusak.com/tools/conversion-confidence-calculator/

> >

> > I think the numbers speak for themselves:

> >

> > Lets say group B is currently showing a 15% improvement

> > over group A.

> >

> > Option 1:

> > Group A 3000 visits. 300 conversions

> > Group B 3000 visits. 345 conversions

> > Confidence Level: 93% <- highest

> >

> >

> > Option 2:

> > Group A 2000 visits. 200 conversions

> > Group B 4000 visits. 460 conversions

> > Confidence Level: 92% <- slightly lower

> >

> >

> > Option 3:

> > Group A 2000 visits. 200 conversions

> > Group B 3000 visits. 345 conversions

> > Confidence Level: 90% <- lowest

> >

> > - Ophir

> >

> >

> > --- In webanalytics@yahoogroups.com, "Kevin

> > Higgins"

> > <kevinjhiggins@> wrote:

> > >

> > > Hi everyone,

> > >

> > > I'm new to MTV and wonder if you could help with a

> > question. Must

> > the size

> > > of a control group always match testing groups? For

> > example, if each

> > of the

> > > testing groups are 5,000 should the control be of

> > relative same size

> > (5,000

> > > as well). Could there be a lesser number for the

> > control which

> > would still

> > > lend statistical relevance?

> > >

> > > Kevin

> > >

> > >

> > > [Non-text portions of this message have been removed]

> > >

> >

> >

> >

> >

> > ------------------------------------

> >

> > ---------------------------------------

> > The Web Analytics Forum

> > Founded by Eric T. Peterson

> > (www.webanalyticsdemystified.com)

> > Moderated by the Web Analytics Association

> > (www.webanalyticsassociation.org)

> > Email moderators at:

> > webanalytics-moderators@...! Groups Links

> >

> >

> >

> - This has been a good dialogue so far, but I think it's worth

reiterating that most statistical techniques assume "equal variance"

in the compared samples.

Ronny Kohavi & the ExP team at Microsoft have a good discussion of

this in the 7 Pitfalls paper

(http://exp-platform.com/ExPpitfalls.aspx) though you might find the

Pratical guide paper (http://exp-platform.com/hippo.aspx) or video

(http://videolectures.net/kdd07_kohavi_pctce/) a better starting point.

The term "heteroscadisticity" refers to the situation of unequal

variances. While there are a lot of concerns here, the answer to the

question "should my control be the same size as my treatment" is

largely yes because you need the variances of the two samples to be

the same for basic apples to apples type reasons.

hth, Andy

Related blog post:

http://alwaysbetesting.com/abtest/index.cfm/2008/3/16/Getting-Serious-About-Testing-Learn-from-the-Pros