Loading ...
Sorry, an error occurred while loading the content.

A/B testing

Expand Messages
  • ebsavov
    Hello everyone, we are starting to implement A/B testing in our company and I have a real beginner question: Do you serve the variations to the same amount of
    Message 1 of 24 , Dec 7, 2010
    • 0 Attachment
      Hello everyone,

      we are starting to implement A/B testing in our company and I have a real beginner question: Do you serve the variations to the same amount of visitors or are small deviations unproblematic (e.g. version A was viewed by 200 and version B by 230 users)? To what extent are deviations in the number of visitors exposed to the different variants acceptable?

      Thanks a lot!
    • Matt Curtis
      ebsavov, Generally, you shouldn t expect to see the exact same number of visitors be placed in each of your test groups. There will be some natural variation
      Message 2 of 24 , Dec 7, 2010
      • 0 Attachment
        ebsavov,

        Generally, you shouldn't expect to see the exact same number of visitors be
        placed in each of your test groups. There will be some natural variation due
        to the way the tool calculates which group a visitor fits into (each visitor
        is randomly assigned to a group 1...n and has a probably of 1/n of being
        assigned to each group).

        That being said, in the example you noted, there is about a
        15% discrepancy between your two groups, which is rather large. I think
        something in the 1-3 or 1-5% might be more acceptable. Part of the problem
        is that you seem to be testing pages that don't receive enough traffic. I
        don't know what the time frame or customer base are for your test, but I
        would want to see at least a few thousand visitors (and the more the
        merrier!) before I was comfortable using any test insight to drive usability
        or conversion decisions.

        --
        Matt Curtis
        matt.a.curtis@...
        http://www.linkedin.com/in/mattacurtis



        On Tue, Dec 7, 2010 at 5:26 AM, ebsavov <elisavova@...> wrote:

        >
        >
        > Hello everyone,
        >
        > we are starting to implement A/B testing in our company and I have a real
        > beginner question: Do you serve the variations to the same amount of
        > visitors or are small deviations unproblematic (e.g. version A was viewed by
        > 200 and version B by 230 users)? To what extent are deviations in the number
        > of visitors exposed to the different variants acceptable?
        >
        > Thanks a lot!
        >
        >
        >


        [Non-text portions of this message have been removed]
      • CG
        There is no reason to try to get equal-size groups. In fact, a good A/B serving tool will actually track the results from the start and adjust the sizes of
        Message 3 of 24 , Dec 8, 2010
        • 0 Attachment
          There is no reason to try to get equal-size groups. In fact, a good A/B serving tool will actually track the results from the start and adjust the sizes of the groups according to the amount of variability that it is seeing in the results of each group. If you're tracking a quantity (size of purchase, quality of visit measured on a scale) then the variability would be calculated for the entire group to date. If you're tracking a percentage of yes/no events (% of visits that made a purchased) then the calculations should happen per day or week or whatever, and the different time periods compared to each other looking for variability. There are other methods for checking variability involving splitting and resplitting the group ... but the bottom line is, if one group is showing huge variability (results are varying all over the map from one day to the next or one purchase to the next) then that group should be larger in order to have more confidence in the results.

          A good stats program will compensate for the size of groups. It should be built in as one of the questions it asks you when analyzing.
        • Zakrajsek, Scott
          I completely agree with the points made by Matt. I would say that the discrepancy (~15%) looks fairly large but that could be a factor of having a very small
          Message 4 of 24 , Dec 8, 2010
          • 0 Attachment
            I completely agree with the points made by Matt. I would say that the
            discrepancy (~15%) looks fairly large but that could be a factor of
            having a very small sample size. I have used a couple of enterprise
            level tools (Sitespect and TnT) on high-traffic sites and typically see
            visitor assignment variances <1% between groups. Some other things for
            you to think about...



            -When you say that X "users" viewed a version of your test, are you
            referring to visits or unique visitors? You should be looking at
            visitor counts. I've seen that certain tests can affect the visit per
            unique visitor metric.



            -Try testing your tool's splitting capability w/ an A:A test. Pick a
            high traffic page (perhaps your homepage) and setup a simple split test
            where each version displays identical content. This should allow you to
            get a baseline for whether or not your tool splits traffic correctly.



            -More of a question, which tool are you using? Most of the
            enterprise-level tools should have robust splitting logic. In the past I
            have seen strange things from home-grown AB test tools due to poor
            randomization methods and inability to persist versions for a unique
            visitor across multiple visits. If this is a home-grown tool make sure
            you check and double-check the randomization logic.



            Good luck!



            -Scott





            From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
            On Behalf Of Matt Curtis
            Sent: Tuesday, December 07, 2010 5:53 PM
            To: webanalytics@yahoogroups.com
            Subject: Re: [webanalytics] A/B testing





            ebsavov,

            Generally, you shouldn't expect to see the exact same number of visitors
            be
            placed in each of your test groups. There will be some natural variation
            due
            to the way the tool calculates which group a visitor fits into (each
            visitor
            is randomly assigned to a group 1...n and has a probably of 1/n of being
            assigned to each group).

            That being said, in the example you noted, there is about a
            15% discrepancy between your two groups, which is rather large. I think
            something in the 1-3 or 1-5% might be more acceptable. Part of the
            problem
            is that you seem to be testing pages that don't receive enough traffic.
            I
            don't know what the time frame or customer base are for your test, but I
            would want to see at least a few thousand visitors (and the more the
            merrier!) before I was comfortable using any test insight to drive
            usability
            or conversion decisions.

            --
            Matt Curtis
            matt.a.curtis@... <mailto:matt.a.curtis%40gmail.com>
            http://www.linkedin.com/in/mattacurtis

            On Tue, Dec 7, 2010 at 5:26 AM, ebsavov <elisavova@...
            <mailto:elisavova%40gmail.com> > wrote:

            >
            >
            > Hello everyone,
            >
            > we are starting to implement A/B testing in our company and I have a
            real
            > beginner question: Do you serve the variations to the same amount of
            > visitors or are small deviations unproblematic (e.g. version A was
            viewed by
            > 200 and version B by 230 users)? To what extent are deviations in the
            number
            > of visitors exposed to the different variants acceptable?
            >
            > Thanks a lot!
            >
            >
            >

            [Non-text portions of this message have been removed]





            [Non-text portions of this message have been removed]
          • Craig Sullivan
            Hi, I agree with Matt. Another point as well - lets say that variant A converted at 10%. That would mean that one more conversion could add 5% to your
            Message 5 of 24 , Dec 8, 2010
            • 0 Attachment
              Hi,

              I agree with Matt. Another point as well - lets say that variant A
              converted at 10%. That would mean that one more conversion could add 5% to
              your figures. See where I'm heading?

              If your sample size is small (say 250) and then the conversion event is even
              smaller in percentage terms, you are playing with numbers that are tiny.
              You will make bad decisions with very small numbers.

              Always remember to base a decent sample target on the goal, not the start of
              any funnel. I personally don't use anything less than 250 people in a
              sample and since we're testing 3M things a month, we tend to use as large as
              necessary, to have statistical confidence. Once we reach numbers that give
              us at least 95% confidence, this is normally enough to smooth out any
              differences in the numbers of people for each creative.

              C.

              On Tue, Dec 7, 2010 at 10:26 AM, ebsavov <elisavova@...> wrote:

              >
              >
              > Hello everyone,
              >
              > we are starting to implement A/B testing in our company and I have a real
              > beginner question: Do you serve the variations to the same amount of
              > visitors or are small deviations unproblematic (e.g. version A was viewed by
              > 200 and version B by 230 users)? To what extent are deviations in the number
              > of visitors exposed to the different variants acceptable?
              >
              > Thanks a lot!
              >
              >
              >



              --
              Craig Sullivan
              http://www.linkedin.com/in/craigsullivan
              +44-(0)7711-657315
              +44-(0)208-318-9290

              Not sent from my blackberry <grin>


              [Non-text portions of this message have been removed]
            • ebsavov
              Thanks to everyone for the quick and informative replies! We are planning to develop a testing tool internally. I will keep in mind the advice given here.
              Message 6 of 24 , Dec 9, 2010
              • 0 Attachment
                Thanks to everyone for the quick and informative replies!

                We are planning to develop a testing tool internally. I will keep in mind the advice given here.


                --- In webanalytics@yahoogroups.com, "ebsavov" <elisavova@...> wrote:
                >
                > Hello everyone,
                >
                > we are starting to implement A/B testing in our company and I have a real beginner question: Do you serve the variations to the same amount of visitors or are small deviations unproblematic (e.g. version A was viewed by 200 and version B by 230 users)? To what extent are deviations in the number of visitors exposed to the different variants acceptable?
                >
                > Thanks a lot!
                >
              • Jay Tkachuk
                As long as the results are statistically significant, you should be ok. ... [Non-text portions of this message have been removed]
                Message 7 of 24 , Dec 10, 2010
                • 0 Attachment
                  As long as the results are statistically significant, you should be ok.

                  On Tue, Dec 7, 2010 at 4:26 AM, ebsavov <elisavova@...> wrote:

                  >
                  >
                  > Hello everyone,
                  >
                  > we are starting to implement A/B testing in our company and I have a real
                  > beginner question: Do you serve the variations to the same amount of
                  > visitors or are small deviations unproblematic (e.g. version A was viewed by
                  > 200 and version B by 230 users)? To what extent are deviations in the number
                  > of visitors exposed to the different variants acceptable?
                  >
                  > Thanks a lot!
                  >
                  >
                  >


                  [Non-text portions of this message have been removed]
                Your message has been successfully submitted and would be delivered to recipients shortly.