## How long to run a test for?

Expand Messages
• OK, The answer to how long do I run this test depends on several factors: * How much traffic does this page or process get? * What confidence interval and by
Message 1 of 6 , May 1, 2008
OK,

The answer to 'how long do I run this test' depends on several factors:

* What confidence interval and by implication sample size do you
need?
* What is your cyclical traffic mix
* What is your promotional mix

Firstly, I'm not a statistician so don't flame me if this explanation
isn't detailed <grin>:

Traffic:

Are you getting enough traffic to get a statistically significant sample
from the page splits? You may have to run the test for long enough to
get a suitable sample. The lower the traffic, the longer the test. Use
your web analytics data to figure out what sort of traffic you get.

Sample Size & Confidence Intervals

http://en.wikipedia.org/wiki/Confidence_level) but really is all about
how you can be sure that you have enough data. If you have one sample
converting 8 people and another converting 10, you might assume that the
second version is better - however, there is clearly not enough data
here for you to be 'confident' that the difference between the two is
real. If you need to calculate your sample size, there is a calculator
here (http://www.surveysystem.com/sscalc.htm).

Cyclical traffic mix

All website traffic varies from hour-hour, day-day, weekly, monthly,
seasonally etc. Ensure that you capture at least one whole cycle here.
For example, if we know that conversion rates are different at weekends
rather than weekdays, then your test period should include at least one
weekend period.

Promotional mix

Imagine that halfway through a test, you run a TV advert campaign. You
may find that the people visiting after this time convert differently
for different creatives than people who didn't see the TV ad. You might
then choose the best performing variant but what happens when you stop
running the TV ads? You are now back with your old traffic mix and they
may have preferred a different creative.

You have to make sure that you understand the promotional mix and visit
sources of the people in your experiment. You might get an answer but
it may only be 'right' in the context of the traffic mix during the time
of the test. The example I've given here would be fine if you are
running TV ads all the time but potentially skewed if this is a limited

Regards,

<http://www.lovefilm.com/>

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/>

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@...

________________________________

From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
On Behalf Of sunitha js
Sent: 30 April 2008 15:31
To: webanalytics@yahoogroups.com
Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

Hi ,
I am new to the Web Analytics field and we are having a project on
Landing page optimization .Our's is a financial services company and our
conversion is applying for loan, but for now our conversion would be to
customer login. We are trying to do ABC test with Test and Target from
Omniture. I would like to know how long do we have to run this test,
that is do we have to leave the test for 7-10 days or longer. Before we
push the winner what is the data we test.
Thanks
Sunitha.

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

LOVEFiLM International Limited is a company registered in England and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, please delete it from your system and notify the sender immediately.

This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com

[Non-text portions of this message have been removed]
• 2 week minimum 100 successes in each test group ... and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United
Message 2 of 6 , May 1, 2008
2 week minimum

100 successes in each test group

--- In webanalytics@yahoogroups.com, "Craig Sullivan"
<craig.sullivan@...> wrote:
>
> OK,
>
>
>
> The answer to 'how long do I run this test' depends on several factors:
>
>
>
> * What confidence interval and by implication sample size do you
> need?
> * What is your cyclical traffic mix
> * What is your promotional mix
>
>
>
>
>
>
> Firstly, I'm not a statistician so don't flame me if this explanation
> isn't detailed <grin>:
>
>
>
> Traffic:
>
>
>
> Are you getting enough traffic to get a statistically significant sample
> from the page splits? You may have to run the test for long enough to
> get a suitable sample. The lower the traffic, the longer the test. Use
> your web analytics data to figure out what sort of traffic you get.
>
>
>
> Sample Size & Confidence Intervals
>
>
>
> http://en.wikipedia.org/wiki/Confidence_level) but really is all about
> how you can be sure that you have enough data. If you have one sample
> converting 8 people and another converting 10, you might assume that the
> second version is better - however, there is clearly not enough data
> here for you to be 'confident' that the difference between the two is
> real. If you need to calculate your sample size, there is a calculator
> here (http://www.surveysystem.com/sscalc.htm).
>
>
>
> Cyclical traffic mix
>
>
>
> All website traffic varies from hour-hour, day-day, weekly, monthly,
> seasonally etc. Ensure that you capture at least one whole cycle here.
> For example, if we know that conversion rates are different at weekends
> rather than weekdays, then your test period should include at least one
> weekend period.
>
>
>
> Promotional mix
>
>
>
> Imagine that halfway through a test, you run a TV advert campaign. You
> may find that the people visiting after this time convert differently
> for different creatives than people who didn't see the TV ad. You might
> then choose the best performing variant but what happens when you stop
> running the TV ads? You are now back with your old traffic mix and they
> may have preferred a different creative.
>
>
>
> You have to make sure that you understand the promotional mix and visit
> sources of the people in your experiment. You might get an answer but
> it may only be 'right' in the context of the traffic mix during the time
> of the test. The example I've given here would be fine if you are
> running TV ads all the time but potentially skewed if this is a limited
>
>
>
> Regards,
>
>
>
>
>
> <http://www.lovefilm.com/>
>
> Craig Sullivan
>
> Product Manager - Digital and Usability
> LOVEFiLM.com <http://www.lovefilm.com/>
>
> No.9 | 6 Portal Way | London | W3 6RU
> T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
> craig.sullivan@...
>
>
>
>
>
> ________________________________
>
> From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
> On Behalf Of sunitha js
> Sent: 30 April 2008 15:31
> To: webanalytics@yahoogroups.com
> Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing
>
>
>
> Hi ,
> I am new to the Web Analytics field and we are having a project on
> Landing page optimization .Our's is a financial services company and our
> conversion is applying for loan, but for now our conversion would be to
> customer login. We are trying to do ABC test with Test and Target from
> Omniture. I would like to know how long do we have to run this test,
> that is do we have to leave the test for 7-10 days or longer. Before we
> push the winner what is the data we test.
> Any suggestions would be helpful.
> Thanks
> Sunitha.
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it now.
>
> [Non-text portions of this message have been removed]
>
>
>
> <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
> BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
> EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
> PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.
>
>
>
> LOVEFiLM International Limited is a company registered in England
and Wales. Registered Number: 04392195. Registered Office: No.9, 6
Portal Way, London W3 6RU, United Kingdom.
>
> This e-mail is confidential to the ordinary user of the e-mail
>
> This message has been scanned for viruses by BlackSpider MailControl
- www.blackspider.com
>
>
> [Non-text portions of this message have been removed]
>
• 2 week minimum and 100 successes in each test group? How did you work that one out then? C. Craig Sullivan Product Manager - Digital
Message 3 of 6 , May 2, 2008
2 week minimum and 100 successes in each test group?

How did you work that one out then?

C.

<http://www.lovefilm.com/>

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/>

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@...

________________________________

From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
On Behalf Of mbchoe
Sent: 01 May 2008 17:06
To: webanalytics@yahoogroups.com
Subject: [webanalytics] Re: How long to run a test for?

2 week minimum

100 successes in each test group

--- In webanalytics@yahoogroups.com
<mailto:webanalytics%40yahoogroups.com> , "Craig Sullivan"
<craig.sullivan@...> wrote:
>
> OK,
>
>
>
> The answer to 'how long do I run this test' depends on several
factors:
>
>
>
> * What confidence interval and by implication sample size do you
> need?
> * What is your cyclical traffic mix
> * What is your promotional mix
>
>
>
>
>
>
> Firstly, I'm not a statistician so don't flame me if this explanation
> isn't detailed <grin>:
>
>
>
> Traffic:
>
>
>
> Are you getting enough traffic to get a statistically significant
sample
> from the page splits? You may have to run the test for long enough to
> get a suitable sample. The lower the traffic, the longer the test. Use
> your web analytics data to figure out what sort of traffic you get.
>
>
>
> Sample Size & Confidence Intervals
>
>
>
> http://en.wikipedia.org/wiki/Confidence_level
<http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
> how you can be sure that you have enough data. If you have one sample
> converting 8 people and another converting 10, you might assume that
the
> second version is better - however, there is clearly not enough data
> here for you to be 'confident' that the difference between the two is
> real. If you need to calculate your sample size, there is a calculator
> here (http://www.surveysystem.com/sscalc.htm
<http://www.surveysystem.com/sscalc.htm> ).
>
>
>
> Cyclical traffic mix
>
>
>
> All website traffic varies from hour-hour, day-day, weekly, monthly,
> seasonally etc. Ensure that you capture at least one whole cycle here.
> For example, if we know that conversion rates are different at
weekends
> rather than weekdays, then your test period should include at least
one
> weekend period.
>
>
>
> Promotional mix
>
>
>
> Imagine that halfway through a test, you run a TV advert campaign. You
> may find that the people visiting after this time convert differently
> for different creatives than people who didn't see the TV ad. You
might
> then choose the best performing variant but what happens when you stop
> running the TV ads? You are now back with your old traffic mix and
they
> may have preferred a different creative.
>
>
>
> You have to make sure that you understand the promotional mix and
visit
> sources of the people in your experiment. You might get an answer but
> it may only be 'right' in the context of the traffic mix during the
time
> of the test. The example I've given here would be fine if you are
> running TV ads all the time but potentially skewed if this is a
limited
>
>
>
> Regards,
>
>
>
>
>
> <http://www.lovefilm.com/ <http://www.lovefilm.com/> >
>
> Craig Sullivan
>
> Product Manager - Digital and Usability
> LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >
>
> No.9 | 6 Portal Way | London | W3 6RU
> T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
> craig.sullivan@...
>
>
>
>
>
> ________________________________
>
> From: webanalytics@yahoogroups.com
<mailto:webanalytics%40yahoogroups.com> [mailto:
webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
> On Behalf Of sunitha js
> Sent: 30 April 2008 15:31
> To: webanalytics@yahoogroups.com
<mailto:webanalytics%40yahoogroups.com>
> Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing
>
>
>
> Hi ,
> I am new to the Web Analytics field and we are having a project on
> Landing page optimization .Our's is a financial services company and
our
> conversion is applying for loan, but for now our conversion would be
to
> customer login. We are trying to do ABC test with Test and Target from
> Omniture. I would like to know how long do we have to run this test,
> that is do we have to leave the test for 7-10 days or longer. Before
we
> push the winner what is the data we test.
> Any suggestions would be helpful.
> Thanks
> Sunitha.
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> it now.
>
> [Non-text portions of this message have been removed]
>
>
>
> <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
>
BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
>
EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
> PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as
spam.
>
>
>
> LOVEFiLM International Limited is a company registered in England
and Wales. Registered Number: 04392195. Registered Office: No.9, 6
Portal Way, London W3 6RU, United Kingdom.
>
> This e-mail is confidential to the ordinary user of the e-mail
>
> This message has been scanned for viruses by BlackSpider MailControl
- www.blackspider.com
>
>
> [Non-text portions of this message have been removed]
>

[Non-text portions of this message have been removed]
• Hi All, Thanks for the suggestions. Our Daily unique visitors are around 15,000 everyday. Our company is a new , we do advertising only online(no ads on TV or
Message 4 of 6 , May 2, 2008
Hi All,
Thanks for the suggestions. Our Daily unique visitors are around 15,000 everyday. Our company is a new , we do advertising only online(no ads on TV or Radio or any other mode of advertising). We have omniture track the results for us. One of the Representative from the SEM did AB test on Landing page. Our conversion vary from days of the week to weeks of the month,like on Monday more people apply for loan than on Sunday and first and third week does better than second and fourth week. In case of Landing pages they tested for 2 days and when they had 97% confidence interval they pushed the winner. I thought they should have let the test run for atleast 2 weeks. I am new this company and still getting to know more about the company.
Thanks for the suggestions again and anyone would want to coment on the above please do.
Sunitha

Craig Sullivan <craig.sullivan@...> wrote:
OK,

The answer to 'how long do I run this test' depends on several factors:

* What confidence interval and by implication sample size do you
need?
* What is your cyclical traffic mix
* What is your promotional mix

Firstly, I'm not a statistician so don't flame me if this explanation
isn't detailed <grin>:

Traffic:

Are you getting enough traffic to get a statistically significant sample
from the page splits? You may have to run the test for long enough to
get a suitable sample. The lower the traffic, the longer the test. Use
your web analytics data to figure out what sort of traffic you get.

Sample Size & Confidence Intervals

http://en.wikipedia.org/wiki/Confidence_level) but really is all about
how you can be sure that you have enough data. If you have one sample
converting 8 people and another converting 10, you might assume that the
second version is better - however, there is clearly not enough data
here for you to be 'confident' that the difference between the two is
real. If you need to calculate your sample size, there is a calculator
here (http://www.surveysystem.com/sscalc.htm).

Cyclical traffic mix

All website traffic varies from hour-hour, day-day, weekly, monthly,
seasonally etc. Ensure that you capture at least one whole cycle here.
For example, if we know that conversion rates are different at weekends
rather than weekdays, then your test period should include at least one
weekend period.

Promotional mix

Imagine that halfway through a test, you run a TV advert campaign. You
may find that the people visiting after this time convert differently
for different creatives than people who didn't see the TV ad. You might
then choose the best performing variant but what happens when you stop
running the TV ads? You are now back with your old traffic mix and they
may have preferred a different creative.

You have to make sure that you understand the promotional mix and visit
sources of the people in your experiment. You might get an answer but
it may only be 'right' in the context of the traffic mix during the time
of the test. The example I've given here would be fine if you are
running TV ads all the time but potentially skewed if this is a limited

Regards,

<http://www.lovefilm.com/>

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/>

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@...

________________________________

From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
On Behalf Of sunitha js
Sent: 30 April 2008 15:31
To: webanalytics@yahoogroups.com
Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

Hi ,
I am new to the Web Analytics field and we are having a project on
Landing page optimization .Our's is a financial services company and our
conversion is applying for loan, but for now our conversion would be to
customer login. We are trying to do ABC test with Test and Target from
Omniture. I would like to know how long do we have to run this test,
that is do we have to leave the test for 7-10 days or longer. Before we
push the winner what is the data we test.
Thanks
Sunitha.

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

LOVEFiLM International Limited is a company registered in England and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, please delete it from your system and notify the sender immediately.

This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com

[Non-text portions of this message have been removed]

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.

[Non-text portions of this message have been removed]
• Hmmm, 2 days with that traffic does sound a bit short.... If you were a/b testing though, you would probably be able to get 97% confidence fairly quickly. I
Message 5 of 6 , May 6, 2008
Hmmm,

2 days with that traffic does sound a bit short.... If you were a/b
testing though, you would probably be able to get 97% confidence fairly
quickly.

I guess the question to ask the person conducting the test is by showing
them the sample size calculator, get them to show you how the traffic
they got exceeded what the sample size calculator will show.

If the vendor isn't calculating their sample size in the same way as the
calculator I showed, I'd be worried as these are standard formulas.....

Regards,

<http://www.lovefilm.com/>

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/>

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@...

________________________________

From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
On Behalf Of sunitha js
Sent: 02 May 2008 19:14
To: webanalytics@yahoogroups.com
Subject: [webanalytics] How long to run a test for?

Hi All,
Thanks for the suggestions. Our Daily unique visitors are around 15,000
everyday. Our company is a new , we do advertising only online(no ads on
TV or Radio or any other mode of advertising). We have omniture track
the results for us. One of the Representative from the SEM did AB test
on Landing page. Our conversion vary from days of the week to weeks of
the month,like on Monday more people apply for loan than on Sunday and
first and third week does better than second and fourth week. In case of
Landing pages they tested for 2 days and when they had 97% confidence
interval they pushed the winner. I thought they should have let the test
run for atleast 2 weeks. I am new this company and still getting to know
Thanks for the suggestions again and anyone would want to coment on the
Sunitha

Craig Sullivan <craig.sullivan@...
<mailto:craig.sullivan%40lovefilm.com> > wrote:
OK,

The answer to 'how long do I run this test' depends on several factors:

* What confidence interval and by implication sample size do you
need?
* What is your cyclical traffic mix
* What is your promotional mix

Firstly, I'm not a statistician so don't flame me if this explanation
isn't detailed <grin>:

Traffic:

Are you getting enough traffic to get a statistically significant sample
from the page splits? You may have to run the test for long enough to
get a suitable sample. The lower the traffic, the longer the test. Use
your web analytics data to figure out what sort of traffic you get.

Sample Size & Confidence Intervals

http://en.wikipedia.org/wiki/Confidence_level
<http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
how you can be sure that you have enough data. If you have one sample
converting 8 people and another converting 10, you might assume that the
second version is better - however, there is clearly not enough data
here for you to be 'confident' that the difference between the two is
real. If you need to calculate your sample size, there is a calculator
here (http://www.surveysystem.com/sscalc.htm
<http://www.surveysystem.com/sscalc.htm> ).

Cyclical traffic mix

All website traffic varies from hour-hour, day-day, weekly, monthly,
seasonally etc. Ensure that you capture at least one whole cycle here.
For example, if we know that conversion rates are different at weekends
rather than weekdays, then your test period should include at least one
weekend period.

Promotional mix

Imagine that halfway through a test, you run a TV advert campaign. You
may find that the people visiting after this time convert differently
for different creatives than people who didn't see the TV ad. You might
then choose the best performing variant but what happens when you stop
running the TV ads? You are now back with your old traffic mix and they
may have preferred a different creative.

You have to make sure that you understand the promotional mix and visit
sources of the people in your experiment. You might get an answer but
it may only be 'right' in the context of the traffic mix during the time
of the test. The example I've given here would be fine if you are
running TV ads all the time but potentially skewed if this is a limited

Regards,

<http://www.lovefilm.com/ <http://www.lovefilm.com/> >

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@... <mailto:craig.sullivan%40lovefilm.com>

________________________________

From: webanalytics@yahoogroups.com
<mailto:webanalytics%40yahoogroups.com> [mailto:
webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
On Behalf Of sunitha js
Sent: 30 April 2008 15:31
To: webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com>

Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

Hi ,
I am new to the Web Analytics field and we are having a project on
Landing page optimization .Our's is a financial services company and our
conversion is applying for loan, but for now our conversion would be to
customer login. We are trying to do ABC test with Test and Target from
Omniture. I would like to know how long do we have to run this test,
that is do we have to leave the test for 7-10 days or longer. Before we
push the winner what is the data we test.
Thanks
Sunitha.

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

LOVEFiLM International Limited is a company registered in England and
Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal
Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address
delete it from your system and notify the sender immediately.

This message has been scanned for viruses by BlackSpider MailControl -
www.blackspider.com

[Non-text portions of this message have been removed]

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

[Non-text portions of this message have been removed]
• Hey guys, Can you see the following link http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/ I think it will help you definitely.
Message 6 of 6 , May 6, 2008
Hey guys,

Can you see the following link

http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/

Thanks,
Bhagawat.
Senior Executive Analytics
Position2,Bangalore
India.

Craig Sullivan <craig.sullivan@...> wrote: Hmmm,

2 days with that traffic does sound a bit short.... If you were a/b
testing though, you would probably be able to get 97% confidence fairly
quickly.

I guess the question to ask the person conducting the test is by showing
them the sample size calculator, get them to show you how the traffic
they got exceeded what the sample size calculator will show.

If the vendor isn't calculating their sample size in the same way as the
calculator I showed, I'd be worried as these are standard formulas.....

Regards,

<http://www.lovefilm.com/>

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/>

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@...

________________________________

From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
On Behalf Of sunitha js
Sent: 02 May 2008 19:14
To: webanalytics@yahoogroups.com
Subject: [webanalytics] How long to run a test for?

Hi All,
Thanks for the suggestions. Our Daily unique visitors are around 15,000
everyday. Our company is a new , we do advertising only online(no ads on
TV or Radio or any other mode of advertising). We have omniture track
the results for us. One of the Representative from the SEM did AB test
on Landing page. Our conversion vary from days of the week to weeks of
the month,like on Monday more people apply for loan than on Sunday and
first and third week does better than second and fourth week. In case of
Landing pages they tested for 2 days and when they had 97% confidence
interval they pushed the winner. I thought they should have let the test
run for atleast 2 weeks. I am new this company and still getting to know
Thanks for the suggestions again and anyone would want to coment on the
Sunitha

Craig Sullivan <craig.sullivan@...
<mailto:craig.sullivan%40lovefilm.com> > wrote:
OK,

The answer to 'how long do I run this test' depends on several factors:

* What confidence interval and by implication sample size do you
need?
* What is your cyclical traffic mix
* What is your promotional mix

Firstly, I'm not a statistician so don't flame me if this explanation
isn't detailed <grin>:

Traffic:

Are you getting enough traffic to get a statistically significant sample
from the page splits? You may have to run the test for long enough to
get a suitable sample. The lower the traffic, the longer the test. Use
your web analytics data to figure out what sort of traffic you get.

Sample Size & Confidence Intervals

http://en.wikipedia.org/wiki/Confidence_level
<http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
how you can be sure that you have enough data. If you have one sample
converting 8 people and another converting 10, you might assume that the
second version is better - however, there is clearly not enough data
here for you to be 'confident' that the difference between the two is
real. If you need to calculate your sample size, there is a calculator
here (http://www.surveysystem.com/sscalc.htm
<http://www.surveysystem.com/sscalc.htm> ).

Cyclical traffic mix

All website traffic varies from hour-hour, day-day, weekly, monthly,
seasonally etc. Ensure that you capture at least one whole cycle here.
For example, if we know that conversion rates are different at weekends
rather than weekdays, then your test period should include at least one
weekend period.

Promotional mix

Imagine that halfway through a test, you run a TV advert campaign. You
may find that the people visiting after this time convert differently
for different creatives than people who didn't see the TV ad. You might
then choose the best performing variant but what happens when you stop
running the TV ads? You are now back with your old traffic mix and they
may have preferred a different creative.

You have to make sure that you understand the promotional mix and visit
sources of the people in your experiment. You might get an answer but
it may only be 'right' in the context of the traffic mix during the time
of the test. The example I've given here would be fine if you are
running TV ads all the time but potentially skewed if this is a limited

Regards,

<http://www.lovefilm.com/ <http://www.lovefilm.com/> >

Craig Sullivan

Product Manager - Digital and Usability
LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

No.9 | 6 Portal Way | London | W3 6RU
T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
craig.sullivan@... <mailto:craig.sullivan%40lovefilm.com>

________________________________

From: webanalytics@yahoogroups.com
<mailto:webanalytics%40yahoogroups.com> [mailto:
webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
On Behalf Of sunitha js
Sent: 30 April 2008 15:31
To: webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com>

Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

Hi ,
I am new to the Web Analytics field and we are having a project on
Landing page optimization .Our's is a financial services company and our
conversion is applying for loan, but for now our conversion would be to
customer login. We are trying to do ABC test with Test and Target from
Omniture. I would like to know how long do we have to run this test,
that is do we have to leave the test for 7-10 days or longer. Before we
push the winner what is the data we test.
Thanks
Sunitha.

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

LOVEFiLM International Limited is a company registered in England and
Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal
Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address
delete it from your system and notify the sender immediately.

This message has been scanned for viruses by BlackSpider MailControl -
www.blackspider.com

[Non-text portions of this message have been removed]

---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
it now.

[Non-text portions of this message have been removed]

[Non-text portions of this message have been removed]

Thanks & Regards,
Bhagawat J.

Position2.com
+91 9986508042
skype id: bhagawat23

---------------------------------
Meet people who discuss and share your passions. Join them now.

[Non-text portions of this message have been removed]
Your message has been successfully submitted and would be delivered to recipients shortly.