Loading ...
Sorry, an error occurred while loading the content.

How long to run a test for?

Expand Messages
  • Craig Sullivan
    OK, The answer to how long do I run this test depends on several factors: * How much traffic does this page or process get? * What confidence interval and by
    Message 1 of 6 , May 1, 2008
    • 0 Attachment
      OK,



      The answer to 'how long do I run this test' depends on several factors:



      * How much traffic does this page or process get?
      * What confidence interval and by implication sample size do you
      need?
      * What is your cyclical traffic mix
      * What is your promotional mix



      This will help you determine time and sample size.



      Firstly, I'm not a statistician so don't flame me if this explanation
      isn't detailed <grin>:



      Traffic:



      Are you getting enough traffic to get a statistically significant sample
      from the page splits? You may have to run the test for long enough to
      get a suitable sample. The lower the traffic, the longer the test. Use
      your web analytics data to figure out what sort of traffic you get.



      Sample Size & Confidence Intervals



      You can read about this here (
      http://en.wikipedia.org/wiki/Confidence_level) but really is all about
      how you can be sure that you have enough data. If you have one sample
      converting 8 people and another converting 10, you might assume that the
      second version is better - however, there is clearly not enough data
      here for you to be 'confident' that the difference between the two is
      real. If you need to calculate your sample size, there is a calculator
      here (http://www.surveysystem.com/sscalc.htm).



      Cyclical traffic mix



      All website traffic varies from hour-hour, day-day, weekly, monthly,
      seasonally etc. Ensure that you capture at least one whole cycle here.
      For example, if we know that conversion rates are different at weekends
      rather than weekdays, then your test period should include at least one
      weekend period.



      Promotional mix



      Imagine that halfway through a test, you run a TV advert campaign. You
      may find that the people visiting after this time convert differently
      for different creatives than people who didn't see the TV ad. You might
      then choose the best performing variant but what happens when you stop
      running the TV ads? You are now back with your old traffic mix and they
      may have preferred a different creative.



      You have to make sure that you understand the promotional mix and visit
      sources of the people in your experiment. You might get an answer but
      it may only be 'right' in the context of the traffic mix during the time
      of the test. The example I've given here would be fine if you are
      running TV ads all the time but potentially skewed if this is a limited
      advertising promotion.



      Regards,





      <http://www.lovefilm.com/>

      Craig Sullivan

      Product Manager - Digital and Usability
      LOVEFiLM.com <http://www.lovefilm.com/>

      No.9 | 6 Portal Way | London | W3 6RU
      T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
      craig.sullivan@...





      ________________________________

      From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
      On Behalf Of sunitha js
      Sent: 30 April 2008 15:31
      To: webanalytics@yahoogroups.com
      Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing



      Hi ,
      I am new to the Web Analytics field and we are having a project on
      Landing page optimization .Our's is a financial services company and our
      conversion is applying for loan, but for now our conversion would be to
      customer login. We are trying to do ABC test with Test and Target from
      Omniture. I would like to know how long do we have to run this test,
      that is do we have to leave the test for 7-10 days or longer. Before we
      push the winner what is the data we test.
      Any suggestions would be helpful.
      Thanks
      Sunitha.

      ---------------------------------
      Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
      it now.

      [Non-text portions of this message have been removed]



      Click here
      <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
      BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
      EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
      PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.



      LOVEFiLM International Limited is a company registered in England and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

      This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, please delete it from your system and notify the sender immediately.

      This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com


      [Non-text portions of this message have been removed]
    • mbchoe
      2 week minimum 100 successes in each test group ... and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United
      Message 2 of 6 , May 1, 2008
      • 0 Attachment
        2 week minimum

        100 successes in each test group




        --- In webanalytics@yahoogroups.com, "Craig Sullivan"
        <craig.sullivan@...> wrote:
        >
        > OK,
        >
        >
        >
        > The answer to 'how long do I run this test' depends on several factors:
        >
        >
        >
        > * How much traffic does this page or process get?
        > * What confidence interval and by implication sample size do you
        > need?
        > * What is your cyclical traffic mix
        > * What is your promotional mix
        >
        >
        >
        > This will help you determine time and sample size.
        >
        >
        >
        > Firstly, I'm not a statistician so don't flame me if this explanation
        > isn't detailed <grin>:
        >
        >
        >
        > Traffic:
        >
        >
        >
        > Are you getting enough traffic to get a statistically significant sample
        > from the page splits? You may have to run the test for long enough to
        > get a suitable sample. The lower the traffic, the longer the test. Use
        > your web analytics data to figure out what sort of traffic you get.
        >
        >
        >
        > Sample Size & Confidence Intervals
        >
        >
        >
        > You can read about this here (
        > http://en.wikipedia.org/wiki/Confidence_level) but really is all about
        > how you can be sure that you have enough data. If you have one sample
        > converting 8 people and another converting 10, you might assume that the
        > second version is better - however, there is clearly not enough data
        > here for you to be 'confident' that the difference between the two is
        > real. If you need to calculate your sample size, there is a calculator
        > here (http://www.surveysystem.com/sscalc.htm).
        >
        >
        >
        > Cyclical traffic mix
        >
        >
        >
        > All website traffic varies from hour-hour, day-day, weekly, monthly,
        > seasonally etc. Ensure that you capture at least one whole cycle here.
        > For example, if we know that conversion rates are different at weekends
        > rather than weekdays, then your test period should include at least one
        > weekend period.
        >
        >
        >
        > Promotional mix
        >
        >
        >
        > Imagine that halfway through a test, you run a TV advert campaign. You
        > may find that the people visiting after this time convert differently
        > for different creatives than people who didn't see the TV ad. You might
        > then choose the best performing variant but what happens when you stop
        > running the TV ads? You are now back with your old traffic mix and they
        > may have preferred a different creative.
        >
        >
        >
        > You have to make sure that you understand the promotional mix and visit
        > sources of the people in your experiment. You might get an answer but
        > it may only be 'right' in the context of the traffic mix during the time
        > of the test. The example I've given here would be fine if you are
        > running TV ads all the time but potentially skewed if this is a limited
        > advertising promotion.
        >
        >
        >
        > Regards,
        >
        >
        >
        >
        >
        > <http://www.lovefilm.com/>
        >
        > Craig Sullivan
        >
        > Product Manager - Digital and Usability
        > LOVEFiLM.com <http://www.lovefilm.com/>
        >
        > No.9 | 6 Portal Way | London | W3 6RU
        > T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
        > craig.sullivan@...
        >
        >
        >
        >
        >
        > ________________________________
        >
        > From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
        > On Behalf Of sunitha js
        > Sent: 30 April 2008 15:31
        > To: webanalytics@yahoogroups.com
        > Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing
        >
        >
        >
        > Hi ,
        > I am new to the Web Analytics field and we are having a project on
        > Landing page optimization .Our's is a financial services company and our
        > conversion is applying for loan, but for now our conversion would be to
        > customer login. We are trying to do ABC test with Test and Target from
        > Omniture. I would like to know how long do we have to run this test,
        > that is do we have to leave the test for 7-10 days or longer. Before we
        > push the winner what is the data we test.
        > Any suggestions would be helpful.
        > Thanks
        > Sunitha.
        >
        > ---------------------------------
        > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
        > it now.
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >
        > Click here
        > <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
        > BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
        > EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
        > PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.
        >
        >
        >
        > LOVEFiLM International Limited is a company registered in England
        and Wales. Registered Number: 04392195. Registered Office: No.9, 6
        Portal Way, London W3 6RU, United Kingdom.
        >
        > This e-mail is confidential to the ordinary user of the e-mail
        address to which it was addressed. If you have received it in error,
        please delete it from your system and notify the sender immediately.
        >
        > This message has been scanned for viruses by BlackSpider MailControl
        - www.blackspider.com
        >
        >
        > [Non-text portions of this message have been removed]
        >
      • Craig Sullivan
        2 week minimum and 100 successes in each test group? How did you work that one out then? C. Craig Sullivan Product Manager - Digital
        Message 3 of 6 , May 2, 2008
        • 0 Attachment
          2 week minimum and 100 successes in each test group?



          How did you work that one out then?



          C.



          <http://www.lovefilm.com/>

          Craig Sullivan

          Product Manager - Digital and Usability
          LOVEFiLM.com <http://www.lovefilm.com/>

          No.9 | 6 Portal Way | London | W3 6RU
          T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
          craig.sullivan@...





          ________________________________

          From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
          On Behalf Of mbchoe
          Sent: 01 May 2008 17:06
          To: webanalytics@yahoogroups.com
          Subject: [webanalytics] Re: How long to run a test for?




          2 week minimum

          100 successes in each test group

          --- In webanalytics@yahoogroups.com
          <mailto:webanalytics%40yahoogroups.com> , "Craig Sullivan"
          <craig.sullivan@...> wrote:
          >
          > OK,
          >
          >
          >
          > The answer to 'how long do I run this test' depends on several
          factors:
          >
          >
          >
          > * How much traffic does this page or process get?
          > * What confidence interval and by implication sample size do you
          > need?
          > * What is your cyclical traffic mix
          > * What is your promotional mix
          >
          >
          >
          > This will help you determine time and sample size.
          >
          >
          >
          > Firstly, I'm not a statistician so don't flame me if this explanation
          > isn't detailed <grin>:
          >
          >
          >
          > Traffic:
          >
          >
          >
          > Are you getting enough traffic to get a statistically significant
          sample
          > from the page splits? You may have to run the test for long enough to
          > get a suitable sample. The lower the traffic, the longer the test. Use
          > your web analytics data to figure out what sort of traffic you get.
          >
          >
          >
          > Sample Size & Confidence Intervals
          >
          >
          >
          > You can read about this here (
          > http://en.wikipedia.org/wiki/Confidence_level
          <http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
          about
          > how you can be sure that you have enough data. If you have one sample
          > converting 8 people and another converting 10, you might assume that
          the
          > second version is better - however, there is clearly not enough data
          > here for you to be 'confident' that the difference between the two is
          > real. If you need to calculate your sample size, there is a calculator
          > here (http://www.surveysystem.com/sscalc.htm
          <http://www.surveysystem.com/sscalc.htm> ).
          >
          >
          >
          > Cyclical traffic mix
          >
          >
          >
          > All website traffic varies from hour-hour, day-day, weekly, monthly,
          > seasonally etc. Ensure that you capture at least one whole cycle here.
          > For example, if we know that conversion rates are different at
          weekends
          > rather than weekdays, then your test period should include at least
          one
          > weekend period.
          >
          >
          >
          > Promotional mix
          >
          >
          >
          > Imagine that halfway through a test, you run a TV advert campaign. You
          > may find that the people visiting after this time convert differently
          > for different creatives than people who didn't see the TV ad. You
          might
          > then choose the best performing variant but what happens when you stop
          > running the TV ads? You are now back with your old traffic mix and
          they
          > may have preferred a different creative.
          >
          >
          >
          > You have to make sure that you understand the promotional mix and
          visit
          > sources of the people in your experiment. You might get an answer but
          > it may only be 'right' in the context of the traffic mix during the
          time
          > of the test. The example I've given here would be fine if you are
          > running TV ads all the time but potentially skewed if this is a
          limited
          > advertising promotion.
          >
          >
          >
          > Regards,
          >
          >
          >
          >
          >
          > <http://www.lovefilm.com/ <http://www.lovefilm.com/> >
          >
          > Craig Sullivan
          >
          > Product Manager - Digital and Usability
          > LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >
          >
          > No.9 | 6 Portal Way | London | W3 6RU
          > T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
          > craig.sullivan@...
          >
          >
          >
          >
          >
          > ________________________________
          >
          > From: webanalytics@yahoogroups.com
          <mailto:webanalytics%40yahoogroups.com> [mailto:
          webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
          > On Behalf Of sunitha js
          > Sent: 30 April 2008 15:31
          > To: webanalytics@yahoogroups.com
          <mailto:webanalytics%40yahoogroups.com>
          > Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing
          >
          >
          >
          > Hi ,
          > I am new to the Web Analytics field and we are having a project on
          > Landing page optimization .Our's is a financial services company and
          our
          > conversion is applying for loan, but for now our conversion would be
          to
          > customer login. We are trying to do ABC test with Test and Target from
          > Omniture. I would like to know how long do we have to run this test,
          > that is do we have to leave the test for 7-10 days or longer. Before
          we
          > push the winner what is the data we test.
          > Any suggestions would be helpful.
          > Thanks
          > Sunitha.
          >
          > ---------------------------------
          > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
          > it now.
          >
          > [Non-text portions of this message have been removed]
          >
          >
          >
          > Click here
          > <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
          <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
          >
          BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
          >
          EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
          > PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as
          spam.
          >
          >
          >
          > LOVEFiLM International Limited is a company registered in England
          and Wales. Registered Number: 04392195. Registered Office: No.9, 6
          Portal Way, London W3 6RU, United Kingdom.
          >
          > This e-mail is confidential to the ordinary user of the e-mail
          address to which it was addressed. If you have received it in error,
          please delete it from your system and notify the sender immediately.
          >
          > This message has been scanned for viruses by BlackSpider MailControl
          - www.blackspider.com
          >
          >
          > [Non-text portions of this message have been removed]
          >





          [Non-text portions of this message have been removed]
        • sunitha js
          Hi All, Thanks for the suggestions. Our Daily unique visitors are around 15,000 everyday. Our company is a new , we do advertising only online(no ads on TV or
          Message 4 of 6 , May 2, 2008
          • 0 Attachment
            Hi All,
            Thanks for the suggestions. Our Daily unique visitors are around 15,000 everyday. Our company is a new , we do advertising only online(no ads on TV or Radio or any other mode of advertising). We have omniture track the results for us. One of the Representative from the SEM did AB test on Landing page. Our conversion vary from days of the week to weeks of the month,like on Monday more people apply for loan than on Sunday and first and third week does better than second and fourth week. In case of Landing pages they tested for 2 days and when they had 97% confidence interval they pushed the winner. I thought they should have let the test run for atleast 2 weeks. I am new this company and still getting to know more about the company.
            Thanks for the suggestions again and anyone would want to coment on the above please do.
            Sunitha

            Craig Sullivan <craig.sullivan@...> wrote:
            OK,

            The answer to 'how long do I run this test' depends on several factors:

            * How much traffic does this page or process get?
            * What confidence interval and by implication sample size do you
            need?
            * What is your cyclical traffic mix
            * What is your promotional mix

            This will help you determine time and sample size.

            Firstly, I'm not a statistician so don't flame me if this explanation
            isn't detailed <grin>:

            Traffic:

            Are you getting enough traffic to get a statistically significant sample
            from the page splits? You may have to run the test for long enough to
            get a suitable sample. The lower the traffic, the longer the test. Use
            your web analytics data to figure out what sort of traffic you get.

            Sample Size & Confidence Intervals

            You can read about this here (
            http://en.wikipedia.org/wiki/Confidence_level) but really is all about
            how you can be sure that you have enough data. If you have one sample
            converting 8 people and another converting 10, you might assume that the
            second version is better - however, there is clearly not enough data
            here for you to be 'confident' that the difference between the two is
            real. If you need to calculate your sample size, there is a calculator
            here (http://www.surveysystem.com/sscalc.htm).

            Cyclical traffic mix

            All website traffic varies from hour-hour, day-day, weekly, monthly,
            seasonally etc. Ensure that you capture at least one whole cycle here.
            For example, if we know that conversion rates are different at weekends
            rather than weekdays, then your test period should include at least one
            weekend period.

            Promotional mix

            Imagine that halfway through a test, you run a TV advert campaign. You
            may find that the people visiting after this time convert differently
            for different creatives than people who didn't see the TV ad. You might
            then choose the best performing variant but what happens when you stop
            running the TV ads? You are now back with your old traffic mix and they
            may have preferred a different creative.

            You have to make sure that you understand the promotional mix and visit
            sources of the people in your experiment. You might get an answer but
            it may only be 'right' in the context of the traffic mix during the time
            of the test. The example I've given here would be fine if you are
            running TV ads all the time but potentially skewed if this is a limited
            advertising promotion.

            Regards,

            <http://www.lovefilm.com/>

            Craig Sullivan

            Product Manager - Digital and Usability
            LOVEFiLM.com <http://www.lovefilm.com/>

            No.9 | 6 Portal Way | London | W3 6RU
            T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
            craig.sullivan@...

            ________________________________

            From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
            On Behalf Of sunitha js
            Sent: 30 April 2008 15:31
            To: webanalytics@yahoogroups.com
            Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

            Hi ,
            I am new to the Web Analytics field and we are having a project on
            Landing page optimization .Our's is a financial services company and our
            conversion is applying for loan, but for now our conversion would be to
            customer login. We are trying to do ABC test with Test and Target from
            Omniture. I would like to know how long do we have to run this test,
            that is do we have to leave the test for 7-10 days or longer. Before we
            push the winner what is the data we test.
            Any suggestions would be helpful.
            Thanks
            Sunitha.

            ---------------------------------
            Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
            it now.

            [Non-text portions of this message have been removed]

            Click here
            <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
            BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
            EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
            PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

            LOVEFiLM International Limited is a company registered in England and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

            This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, please delete it from your system and notify the sender immediately.

            This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com

            [Non-text portions of this message have been removed]






            ---------------------------------
            Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.

            [Non-text portions of this message have been removed]
          • Craig Sullivan
            Hmmm, 2 days with that traffic does sound a bit short.... If you were a/b testing though, you would probably be able to get 97% confidence fairly quickly. I
            Message 5 of 6 , May 6, 2008
            • 0 Attachment
              Hmmm,



              2 days with that traffic does sound a bit short.... If you were a/b
              testing though, you would probably be able to get 97% confidence fairly
              quickly.



              I guess the question to ask the person conducting the test is by showing
              them the sample size calculator, get them to show you how the traffic
              they got exceeded what the sample size calculator will show.



              If the vendor isn't calculating their sample size in the same way as the
              calculator I showed, I'd be worried as these are standard formulas.....



              Regards,



              <http://www.lovefilm.com/>

              Craig Sullivan

              Product Manager - Digital and Usability
              LOVEFiLM.com <http://www.lovefilm.com/>

              No.9 | 6 Portal Way | London | W3 6RU
              T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
              craig.sullivan@...





              ________________________________

              From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
              On Behalf Of sunitha js
              Sent: 02 May 2008 19:14
              To: webanalytics@yahoogroups.com
              Subject: [webanalytics] How long to run a test for?



              Hi All,
              Thanks for the suggestions. Our Daily unique visitors are around 15,000
              everyday. Our company is a new , we do advertising only online(no ads on
              TV or Radio or any other mode of advertising). We have omniture track
              the results for us. One of the Representative from the SEM did AB test
              on Landing page. Our conversion vary from days of the week to weeks of
              the month,like on Monday more people apply for loan than on Sunday and
              first and third week does better than second and fourth week. In case of
              Landing pages they tested for 2 days and when they had 97% confidence
              interval they pushed the winner. I thought they should have let the test
              run for atleast 2 weeks. I am new this company and still getting to know
              more about the company.
              Thanks for the suggestions again and anyone would want to coment on the
              above please do.
              Sunitha

              Craig Sullivan <craig.sullivan@...
              <mailto:craig.sullivan%40lovefilm.com> > wrote:
              OK,

              The answer to 'how long do I run this test' depends on several factors:

              * How much traffic does this page or process get?
              * What confidence interval and by implication sample size do you
              need?
              * What is your cyclical traffic mix
              * What is your promotional mix

              This will help you determine time and sample size.

              Firstly, I'm not a statistician so don't flame me if this explanation
              isn't detailed <grin>:

              Traffic:

              Are you getting enough traffic to get a statistically significant sample
              from the page splits? You may have to run the test for long enough to
              get a suitable sample. The lower the traffic, the longer the test. Use
              your web analytics data to figure out what sort of traffic you get.

              Sample Size & Confidence Intervals

              You can read about this here (
              http://en.wikipedia.org/wiki/Confidence_level
              <http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
              about
              how you can be sure that you have enough data. If you have one sample
              converting 8 people and another converting 10, you might assume that the
              second version is better - however, there is clearly not enough data
              here for you to be 'confident' that the difference between the two is
              real. If you need to calculate your sample size, there is a calculator
              here (http://www.surveysystem.com/sscalc.htm
              <http://www.surveysystem.com/sscalc.htm> ).

              Cyclical traffic mix

              All website traffic varies from hour-hour, day-day, weekly, monthly,
              seasonally etc. Ensure that you capture at least one whole cycle here.
              For example, if we know that conversion rates are different at weekends
              rather than weekdays, then your test period should include at least one
              weekend period.

              Promotional mix

              Imagine that halfway through a test, you run a TV advert campaign. You
              may find that the people visiting after this time convert differently
              for different creatives than people who didn't see the TV ad. You might
              then choose the best performing variant but what happens when you stop
              running the TV ads? You are now back with your old traffic mix and they
              may have preferred a different creative.

              You have to make sure that you understand the promotional mix and visit
              sources of the people in your experiment. You might get an answer but
              it may only be 'right' in the context of the traffic mix during the time
              of the test. The example I've given here would be fine if you are
              running TV ads all the time but potentially skewed if this is a limited
              advertising promotion.

              Regards,

              <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

              Craig Sullivan

              Product Manager - Digital and Usability
              LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

              No.9 | 6 Portal Way | London | W3 6RU
              T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
              craig.sullivan@... <mailto:craig.sullivan%40lovefilm.com>

              ________________________________

              From: webanalytics@yahoogroups.com
              <mailto:webanalytics%40yahoogroups.com> [mailto:
              webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
              On Behalf Of sunitha js
              Sent: 30 April 2008 15:31
              To: webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com>

              Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

              Hi ,
              I am new to the Web Analytics field and we are having a project on
              Landing page optimization .Our's is a financial services company and our
              conversion is applying for loan, but for now our conversion would be to
              customer login. We are trying to do ABC test with Test and Target from
              Omniture. I would like to know how long do we have to run this test,
              that is do we have to leave the test for 7-10 days or longer. Before we
              push the winner what is the data we test.
              Any suggestions would be helpful.
              Thanks
              Sunitha.

              ---------------------------------
              Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
              it now.

              [Non-text portions of this message have been removed]

              Click here
              <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
              <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
              BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
              EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
              PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

              LOVEFiLM International Limited is a company registered in England and
              Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal
              Way, London W3 6RU, United Kingdom.

              This e-mail is confidential to the ordinary user of the e-mail address
              to which it was addressed. If you have received it in error, please
              delete it from your system and notify the sender immediately.

              This message has been scanned for viruses by BlackSpider MailControl -
              www.blackspider.com

              [Non-text portions of this message have been removed]

              ---------------------------------
              Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
              it now.

              [Non-text portions of this message have been removed]





              [Non-text portions of this message have been removed]
            • bhagawat jadhav
              Hey guys, Can you see the following link http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/ I think it will help you definitely.
              Message 6 of 6 , May 6, 2008
              • 0 Attachment
                Hey guys,

                Can you see the following link

                http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/

                I think it will help you definitely.

                Thanks,
                Bhagawat.
                Senior Executive Analytics
                Position2,Bangalore
                India.

                Craig Sullivan <craig.sullivan@...> wrote: Hmmm,

                2 days with that traffic does sound a bit short.... If you were a/b
                testing though, you would probably be able to get 97% confidence fairly
                quickly.

                I guess the question to ask the person conducting the test is by showing
                them the sample size calculator, get them to show you how the traffic
                they got exceeded what the sample size calculator will show.

                If the vendor isn't calculating their sample size in the same way as the
                calculator I showed, I'd be worried as these are standard formulas.....

                Regards,

                <http://www.lovefilm.com/>

                Craig Sullivan

                Product Manager - Digital and Usability
                LOVEFiLM.com <http://www.lovefilm.com/>

                No.9 | 6 Portal Way | London | W3 6RU
                T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
                craig.sullivan@...

                ________________________________

                From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
                On Behalf Of sunitha js
                Sent: 02 May 2008 19:14
                To: webanalytics@yahoogroups.com
                Subject: [webanalytics] How long to run a test for?

                Hi All,
                Thanks for the suggestions. Our Daily unique visitors are around 15,000
                everyday. Our company is a new , we do advertising only online(no ads on
                TV or Radio or any other mode of advertising). We have omniture track
                the results for us. One of the Representative from the SEM did AB test
                on Landing page. Our conversion vary from days of the week to weeks of
                the month,like on Monday more people apply for loan than on Sunday and
                first and third week does better than second and fourth week. In case of
                Landing pages they tested for 2 days and when they had 97% confidence
                interval they pushed the winner. I thought they should have let the test
                run for atleast 2 weeks. I am new this company and still getting to know
                more about the company.
                Thanks for the suggestions again and anyone would want to coment on the
                above please do.
                Sunitha

                Craig Sullivan <craig.sullivan@...
                <mailto:craig.sullivan%40lovefilm.com> > wrote:
                OK,

                The answer to 'how long do I run this test' depends on several factors:

                * How much traffic does this page or process get?
                * What confidence interval and by implication sample size do you
                need?
                * What is your cyclical traffic mix
                * What is your promotional mix

                This will help you determine time and sample size.

                Firstly, I'm not a statistician so don't flame me if this explanation
                isn't detailed <grin>:

                Traffic:

                Are you getting enough traffic to get a statistically significant sample
                from the page splits? You may have to run the test for long enough to
                get a suitable sample. The lower the traffic, the longer the test. Use
                your web analytics data to figure out what sort of traffic you get.

                Sample Size & Confidence Intervals

                You can read about this here (
                http://en.wikipedia.org/wiki/Confidence_level
                <http://en.wikipedia.org/wiki/Confidence_level> ) but really is all
                about
                how you can be sure that you have enough data. If you have one sample
                converting 8 people and another converting 10, you might assume that the
                second version is better - however, there is clearly not enough data
                here for you to be 'confident' that the difference between the two is
                real. If you need to calculate your sample size, there is a calculator
                here (http://www.surveysystem.com/sscalc.htm
                <http://www.surveysystem.com/sscalc.htm> ).

                Cyclical traffic mix

                All website traffic varies from hour-hour, day-day, weekly, monthly,
                seasonally etc. Ensure that you capture at least one whole cycle here.
                For example, if we know that conversion rates are different at weekends
                rather than weekdays, then your test period should include at least one
                weekend period.

                Promotional mix

                Imagine that halfway through a test, you run a TV advert campaign. You
                may find that the people visiting after this time convert differently
                for different creatives than people who didn't see the TV ad. You might
                then choose the best performing variant but what happens when you stop
                running the TV ads? You are now back with your old traffic mix and they
                may have preferred a different creative.

                You have to make sure that you understand the promotional mix and visit
                sources of the people in your experiment. You might get an answer but
                it may only be 'right' in the context of the traffic mix during the time
                of the test. The example I've given here would be fine if you are
                running TV ads all the time but potentially skewed if this is a limited
                advertising promotion.

                Regards,

                <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

                Craig Sullivan

                Product Manager - Digital and Usability
                LOVEFiLM.com <http://www.lovefilm.com/ <http://www.lovefilm.com/> >

                No.9 | 6 Portal Way | London | W3 6RU
                T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
                craig.sullivan@... <mailto:craig.sullivan%40lovefilm.com>

                ________________________________

                From: webanalytics@yahoogroups.com
                <mailto:webanalytics%40yahoogroups.com> [mailto:
                webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com> ]
                On Behalf Of sunitha js
                Sent: 30 April 2008 15:31
                To: webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com>

                Subject: Re: [webanalytics] Tool to track clicks and lead A/B testing

                Hi ,
                I am new to the Web Analytics field and we are having a project on
                Landing page optimization .Our's is a financial services company and our
                conversion is applying for loan, but for now our conversion would be to
                customer login. We are trying to do ABC test with Test and Target from
                Omniture. I would like to know how long do we have to run this test,
                that is do we have to leave the test for 7-10 days or longer. Before we
                push the winner what is the data we test.
                Any suggestions would be helpful.
                Thanks
                Sunitha.

                ---------------------------------
                Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
                it now.

                [Non-text portions of this message have been removed]

                Click here
                <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
                <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==>
                BIFl!QsU4cvexBZHAnUk0nZxA3KI97doM1iyfXHLcYYWaIny2gsKCmCuIfV+LH3aEyiQygYV
                EqqlDLQlQ1SeUWcqIo08WplUy0lhUk3t6e7nUmM7NW40i4aFJz081dc1hbWfEhdO6BZokepw
                PUMXUHImsKaXtZ5fyHfFT1kIm+0mydkzRWvZw8j!> to report this email as spam.

                LOVEFiLM International Limited is a company registered in England and
                Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal
                Way, London W3 6RU, United Kingdom.

                This e-mail is confidential to the ordinary user of the e-mail address
                to which it was addressed. If you have received it in error, please
                delete it from your system and notify the sender immediately.

                This message has been scanned for viruses by BlackSpider MailControl -
                www.blackspider.com

                [Non-text portions of this message have been removed]

                ---------------------------------
                Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
                it now.

                [Non-text portions of this message have been removed]

                [Non-text portions of this message have been removed]






                Thanks & Regards,
                Bhagawat J.

                Position2.com
                +91 9986508042
                skype id: bhagawat23

                ---------------------------------
                Meet people who discuss and share your passions. Join them now.

                [Non-text portions of this message have been removed]
              Your message has been successfully submitted and would be delivered to recipients shortly.