Loading ...
Sorry, an error occurred while loading the content.
 

Re: A/B Test Question

Expand Messages
  • alexbrasil489
    ... Thanks for that article Barry-- am reading through it now and it is indeed a great resource. Timely too since I m going through a complete tear
    Message 1 of 17 , Apr 13, 2008
      --- In webanalytics@yahoogroups.com, "johnnydeptford"
      <barrysholloway@...> wrote:
      >
      > Hi Adam
      >
      > There's an excellent article on A/B and multivariate testing here -
      > http://exp-platform.com/hippo_long.aspx.
      >
      > Hope that helps
      >
      > Barry
      >

      Thanks for that article Barry-- am reading through it now and it is
      indeed a great resource. Timely too since I'm going through a
      complete tear down/rebuild of an ecommerce site.

      FYI the period got caught in the URL.
      http://exp-platform.com/hippo_long.aspx should work.

      Alex
    • bhagawat jadhav
      Hi Kevin, How will you decide that an increase in conversions by a statistical significant amount? Is it by proportion test? Thanks, Bhagawat. Kevin Rogers
      Message 2 of 17 , Apr 13, 2008
        Hi Kevin,

        How will you decide that an increase in conversions by a statistical significant amount?

        Is it by proportion test?

        Thanks,
        Bhagawat.

        Kevin Rogers <kevin.rogers@...> wrote: If the test led to an increase in conversions by a statistical significant amount then I would call the test a success.

        Sent via BlackBerry from T-Mobile

        -----Original Message-----
        From: Adam Berlinger

        Date: Sat, 12 Apr 2008 16:46:17
        To:webanalytics@yahoogroups.com
        Subject: [webanalytics] A/B Test Question


        Hello everyone. I'm curious to see how you gauge the success of an A/B or multivariate test? What are the metrics that are measured? What are the things that you look for in these types of initiatives? I appreciate your insight & feedback.

        Thanks,
        Adam
        http://analyticsbya dam.blogspot.com

        __________________________________________________
        Do You Yahoo!?
        Tired of spam? Yahoo! Mail has the best spam protection around
        http://mail. yahoo.com

        [Non-text portions of this message have been removed]


        ------------------------------------

        ---------------------------------------
        The Web Analytics Forum
        Founded by Eric T. Peterson (www.webanalyticsdemystified.com)
        Moderated by the Web Analytics Association (www.webanalyticsassociation.org)
        Email moderators at: webanalytics-moderators@yahoogroups.comYahoo! Groups Links






        Thanks & Regards,

        Bhagawat J.

        Position2.com

        +91 98447 13874

        ---------------------------------
        Best Jokes, Best Friends, Best Food. Get all this and more on Best of Yahoo! Groups.

        [Non-text portions of this message have been removed]
      • Iain Murphy
        Hello Adam, You are best using the metrics that matter most to your business! Lots of businesses use Visits, Page Views or (dare I use it) the dreaded metric,
        Message 3 of 17 , Apr 14, 2008
          Hello Adam,

          You are best using the metrics that matter most to your business! Lots of
          businesses use Visits, Page Views or (dare I use it) the dreaded metric,
          H.I.T.S, to measure conversion in a given engagement process. However, none
          of these metrics really mean anything to an Insurance, Finance, Commerce and
          other types of sites.
          You are best sitting down and brainstorming what metrics are best for your
          business; they could be Applications, Submissions, Downloads, Registrations,
          Orders and so on... measuring any of those (or others) in a A/B test
          environment would be the indicators of choice to measure any uplift and
          improvement.

          We can help you work out your KPI's if interested. This is something we talk
          about everyday with our clients as it is the essence of Web Analytics. Have
          a bad KPI, have meaningless reports. Have a good KPI however and have
          powerful, indicative reports from which you can make well educated and
          objective decisions rather than relying on instinct.

          Which A/B Test tool are you using or looking to use? Google's Website
          Optimizer is a good (and free) place to start:
          http://services.google.com/websiteoptimizer

          Hope this helps,
          Iain

          SPIWEB Consultancy Limited
          www.spiweb-consultancy.com


          [Non-text portions of this message have been removed]
        • vabeachkevin
          I am a fan of the Teasley Statistical significance calculator. http://www.teasley.net/statcalc.xls ... statistical significant amount? ... increase in
          Message 4 of 17 , Apr 14, 2008
            I am a fan of the Teasley Statistical significance calculator.

            http://www.teasley.net/statcalc.xls


            --- In webanalytics@yahoogroups.com, bhagawat jadhav
            <bhagawat_jadhav@...> wrote:
            >
            > Hi Kevin,
            >
            > How will you decide that an increase in conversions by a
            statistical significant amount?
            >
            > Is it by proportion test?
            >
            > Thanks,
            > Bhagawat.
            >
            > Kevin Rogers <kevin.rogers@...> wrote: If the test led to an
            increase in conversions by a statistical significant amount then I
            would call the test a success.
            >
            > Sent via BlackBerry from T-Mobile
            >
            > -----Original Message-----
            > From: Adam Berlinger
            >
            > Date: Sat, 12 Apr 2008 16:46:17
            > To:webanalytics@yahoogroups.com
            > Subject: [webanalytics] A/B Test Question
            >
            >
            > Hello everyone. I'm curious to see how you gauge the success of an
            A/B or multivariate test? What are the metrics that are measured?
            What are the things that you look for in these types of initiatives?
            I appreciate your insight & feedback.
            >
            > Thanks,
            > Adam
            > http://analyticsbya dam.blogspot.com
            >
            > __________________________________________________
            > Do You Yahoo!?
            > Tired of spam? Yahoo! Mail has the best spam protection around
            > http://mail. yahoo.com
            >
            > [Non-text portions of this message have been removed]
            >
            >
            > ------------------------------------
            >
            > ---------------------------------------
            > The Web Analytics Forum
            > Founded by Eric T. Peterson (www.webanalyticsdemystified.com)
            > Moderated by the Web Analytics Association
            (www.webanalyticsassociation.org)
            > Email moderators at: webanalytics-moderators@...! Groups Links
            >
            >
            >
            >
            >
            >
            > Thanks & Regards,
            >
            > Bhagawat J.
            >
            > Position2.com
            >
            > +91 98447 13874
            >
            > ---------------------------------
            > Best Jokes, Best Friends, Best Food. Get all this and more on
            Best of Yahoo! Groups.
            >
            > [Non-text portions of this message have been removed]
            >
          • Tim Wilson
            Avinash tackled this subject a few months ago, so it s worth taking a look at his thoughts: http://tinyurl.com/2tvkoe. His post actually inspired our senior
            Message 5 of 17 , Apr 14, 2008
              Avinash tackled this subject a few months ago, so it's worth taking a
              look at his thoughts: http://tinyurl.com/2tvkoe.

              His post actually inspired our senior analyst, JT Buser, to develop a
              couple of Excel-based tools for assessing the results of an A/B test.
              JT's working on getting some supporting documentation and material
              together and posted on our web site, but it seemed sufficiently salient
              to this discussion that I've gone ahead and uploaded both files to the
              "Files" section of this group.

              The simple calculator uses a 90% confidence interval as the default.
              It's posted here: http://tinyurl.com/5v2e84

              The advanced calculator allows you to specify a confidence interval.
              And, it includes a second worksheet for helping you estimate the sample
              size you need to use for additional tests. It's posted here:
              http://tinyurl.com/68zuek

              Both calculators provide information beyond simply "the answer" -- how
              to actually interpret the results. I can't speak to the formulas behind
              the calculator...but that's one of the things we hope to get up on our
              web site fairly soon (the link in the spreadsheet will, ultimately, get
              redirected to that information).


              Tim Wilson
              http://www.gilliganondata.com
              http://www.bulldogsolutions.com


              ________________________________

              From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
              On Behalf Of Adam Berlinger
              Sent: Saturday, April 12, 2008 7:46 PM
              To: webanalytics@yahoogroups.com
              Subject: [webanalytics] A/B Test Question



              Hello everyone. I'm curious to see how you gauge the success of an A/B
              or multivariate test? What are the metrics that are measured? What are
              the things that you look for in these types of initiatives? I appreciate
              your insight & feedback.

              Thanks,
              Adam
              http://analyticsbyadam.blogspot.com
              <http://analyticsbyadam.blogspot.com>

              __________________________________________________
              Do You Yahoo!?
              Tired of spam? Yahoo! Mail has the best spam protection around
              http://mail.yahoo.com <http://mail.yahoo.com>

              [Non-text portions of this message have been removed]
            • Adam Berlinger
              How do most organizations measure the success of site A vs. B. Thanks, Adam http://analyticsbyadam.blogspot.com ... From: mbchoe To:
              Message 6 of 17 , Apr 14, 2008
                How do most organizations measure the success of site "A" vs. "B."

                Thanks,
                Adam
                http://analyticsbyadam.blogspot.com



                ----- Original Message ----
                From: mbchoe <mbchoe@...>
                To: webanalytics@yahoogroups.com
                Sent: Sunday, April 13, 2008 2:54:04 AM
                Subject: [webanalytics] Re: A/B Test Question


                adam,

                can you be more specific or elaborate with an example? i'm not clear
                on what you're asking...

                michael


                --- In webanalytics@ yahoogroups. com, Adam Berlinger <aberlinger1@ ...>
                wrote:
                >
                > Hello everyone. I'm curious to see how you gauge the success of an
                A/B or multivariate test? What are the metrics that are measured? What
                are the things that you look for in these types of initiatives? I
                appreciate your insight & feedback.
                >
                > Thanks,
                > Adam
                > http://analyticsbya dam.blogspot. com
                >
                > ____________ _________ _________ _________ _________ __
                > Do You Yahoo!?
                > Tired of spam? Yahoo! Mail has the best spam protection around
                > http://mail. yahoo.com
                >
                > [Non-text portions of this message have been removed]
                >





                ____________________________________________________________________________________
                Be a better friend, newshound, and
                know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

                [Non-text portions of this message have been removed]
              • Tim Wilson
                Daniel S. just pointed out to me that the tinyurl links are busted. Ugh. The link to Avinash s post is because it wound up with a . tacked on. This should
                Message 7 of 17 , Apr 14, 2008
                  Daniel S. just pointed out to me that the tinyurl links are busted. Ugh.

                  The link to Avinash's post is because it wound up with a "." tacked on.
                  This should work: http://tinyurl.com/2tvkoe

                  As for the links to the spreadsheets, apparently the TinyURLing of the
                  links as I navigated to them don't work.

                  So, go to the Files section for this group:
                  http://tech.groups.yahoo.com/group/webanalytics/files/

                  The "simple" calculator is called: Simple A_B Calculator.xls

                  The "advanced" calculator is called: Advanced A_B Calculator.xls

                  To properly credit the inspiration -- Brian Teasley's spreadsheet
                  (linked to in another response to this thread) was what Avinash's post
                  pointed to, and this was an exercise to enhance what he had done.

                  Tim


                  Tim Wilson
                  Vice President, Services
                  Bulldog Solutions, Inc.
                  5608 Parkcrest Drive, Suite 300
                  Austin, TX 78731
                  Tel: 512.538.0371
                  Mobile: 512.431.8502
                  http://www.bulldogsolutions.com <http://www.bulldogsolutions.com/>


                  ________________________________

                  From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
                  On Behalf Of Tim Wilson
                  Sent: Monday, April 14, 2008 11:15 AM
                  To: webanalytics@yahoogroups.com
                  Subject: RE: [webanalytics] A/B Test Question




                  Avinash tackled this subject a few months ago, so it's worth taking a
                  look at his thoughts: http://tinyurl.com/2tvkoe.
                  <http://tinyurl.com/2tvkoe.>

                  His post actually inspired our senior analyst, JT Buser, to develop a
                  couple of Excel-based tools for assessing the results of an A/B test.
                  JT's working on getting some supporting documentation and material
                  together and posted on our web site, but it seemed sufficiently salient
                  to this discussion that I've gone ahead and uploaded both files to the
                  "Files" section of this group.

                  The simple calculator uses a 90% confidence interval as the default.
                  It's posted here: http://tinyurl.com/5v2e84 <http://tinyurl.com/5v2e84>

                  The advanced calculator allows you to specify a confidence interval.
                  And, it includes a second worksheet for helping you estimate the sample
                  size you need to use for additional tests. It's posted here:
                  http://tinyurl.com/68zuek <http://tinyurl.com/68zuek>

                  Both calculators provide information beyond simply "the answer" -- how
                  to actually interpret the results. I can't speak to the formulas behind
                  the calculator...but that's one of the things we hope to get up on our
                  web site fairly soon (the link in the spreadsheet will, ultimately, get
                  redirected to that information).

                  Tim Wilson
                  http://www.gilliganondata.com <http://www.gilliganondata.com>
                  http://www.bulldogsolutions.com <http://www.bulldogsolutions.com>


                  ________________________________

                  From: webanalytics@yahoogroups.com
                  <mailto:webanalytics%40yahoogroups.com>
                  [mailto:webanalytics@yahoogroups.com
                  <mailto:webanalytics%40yahoogroups.com> ]
                  On Behalf Of Adam Berlinger
                  Sent: Saturday, April 12, 2008 7:46 PM
                  To: webanalytics@yahoogroups.com <mailto:webanalytics%40yahoogroups.com>

                  Subject: [webanalytics] A/B Test Question

                  Hello everyone. I'm curious to see how you gauge the success of an A/B
                  or multivariate test? What are the metrics that are measured? What are
                  the things that you look for in these types of initiatives? I appreciate
                  your insight & feedback.

                  Thanks,
                  Adam
                  http://analyticsbyadam.blogspot.com
                  <http://analyticsbyadam.blogspot.com>
                  <http://analyticsbyadam.blogspot.com
                  <http://analyticsbyadam.blogspot.com> >

                  __________________________________________________
                  Do You Yahoo!?
                  Tired of spam? Yahoo! Mail has the best spam protection around
                  http://mail.yahoo.com <http://mail.yahoo.com> <http://mail.yahoo.com
                  <http://mail.yahoo.com> >

                  [Non-text portions of this message have been removed]






                  [Non-text portions of this message have been removed]
                • jeffcarolinecampbell
                  Here is a good how to on determining when/how to A/B test. Apologies for promoting my own content, but should be relevant:
                  Message 8 of 17 , Apr 14, 2008
                    Here is a good "how to" on determining when/how to A/B test.
                    Apologies for promoting my own content, but should be relevant:

                    http://www.findresolution.com/2008/01/with-ab-testing-you-dont-think-you-know.html

                    I'm reading Tim Ash's Guide to Landing Page Optimization now, a very
                    good primer that answers your question in-depth. It's on Amazon for
                    like $20.
                  • Adam Berlinger
                    Actually, let me re-phrase. Once you have performaned an A/B or multivarite test...how do you determine that your results have not been influenced by a
                    Message 9 of 17 , Apr 14, 2008
                      Actually, let me re-phrase. Once you have performaned an A/B or multivarite test...how do you determine that your results have not been influenced by a performance issue. For example, perhaps an object did not load adequately in Site "B" vs. "A" or the average response time of your application differed across your geographic points of interest.

                      So how deep do most organizations dig into the data they're presented after such initiatives. Conversions (quantity) might be higher in "A" or "B" but why (quality)? That's the substance to my question and where I am trying to educate myself.

                      Thanks,
                      Adam
                      http://analyticsbyadam.blogspot.com



                      ----- Original Message ----
                      From: mbchoe <mbchoe@...>
                      To: webanalytics@yahoogroups.com
                      Sent: Sunday, April 13, 2008 2:54:04 AM
                      Subject: [webanalytics] Re: A/B Test Question


                      adam,

                      can you be more specific or elaborate with an example? i'm not clear
                      on what you're asking...

                      michael


                      --- In webanalytics@ yahoogroups. com, Adam Berlinger <aberlinger1@ ...>
                      wrote:
                      >
                      > Hello everyone. I'm curious to see how you gauge the success of an
                      A/B or multivariate test? What are the metrics that are measured? What
                      are the things that you look for in these types of initiatives? I
                      appreciate your insight & feedback.
                      >
                      > Thanks,
                      > Adam
                      > http://analyticsbya dam.blogspot. com
                      >
                      > ____________ _________ _________ _________ _________ __
                      > Do You Yahoo!?
                      > Tired of spam? Yahoo! Mail has the best spam protection around
                      > http://mail. yahoo.com
                      >
                      > [Non-text portions of this message have been removed]
                      >





                      ____________________________________________________________________________________
                      Be a better friend, newshound, and
                      know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

                      [Non-text portions of this message have been removed]
                    • widemilebs
                      If you are doing a simultaneous A/B test, what I call a split test, then performance issues should impact both pages equally and so should not be a factor of
                      Message 10 of 17 , Apr 14, 2008
                        If you are doing a simultaneous A/B test, what I call a split test,
                        then performance issues should impact both pages equally and so should
                        not be a factor of influence. However if you do an A/B test
                        sequentially (test A then a week later test B), it may have an impact.
                        However I'm not totally clear if you mean network issues versus long
                        page load times because of file size.

                        Regardless, there are hardly any good reasons to do a sequential A/B
                        test, especially with Google Web Optimizer around to do A/B split
                        tests for free.

                        Also the qualitative question is hard to figure out using A/B testing.
                        You can make theories with fairly good success, but getting a
                        definitive Why? is hard. One of the examples I like to use is fear
                        based messaging versus feature messaging. "Protect your family"
                        versus "Comes in 4 colors". If you test those 2, then you can
                        hypothesize the winner resonates with the audience because they are
                        buying the product for that reason.

                        However, this is not as easy to do with A/B testing. A/B testing is
                        great for large changes to the website, such as a full creative
                        redesign or a total change in functionality or usability of the
                        website. For more granular changes and asking those "why?" questions,
                        multivariate testing is much more effective.

                        If you want to learn more about testing and optimization, I write
                        about them exclusively at my blog.

                        Some of the more relevant articles:
                        -Template testing versus inplace testing:
                        http://testingblog.widemile.com/2007/11/28/two-types-of-tests-and-im-not-talking-mv-or-split/
                        -5 tips for a/b split testing:
                        http://testingblog.widemile.com/2007/12/05/5-quick-tips-to-effective-ab-and-split-testing/
                        -A primer on multivariate testing:
                        http://testingblog.widemile.com/2007/12/26/multivariate-testing-a-quick-primer/

                        -----
                        Billy Shih
                        Optimization Analyst, Widemile
                        http://testingblog.widemile.com

                        --- In webanalytics@yahoogroups.com, Adam Berlinger <aberlinger1@...>
                        wrote:
                        >
                        > Actually, let me re-phrase. Once you have performaned an A/B or
                        multivarite test...how do you determine that your results have not
                        been influenced by a performance issue. For example, perhaps an object
                        did not load adequately in Site "B" vs. "A" or the average response
                        time of your application differed across your geographic points of
                        interest.
                        >
                        > So how deep do most organizations dig into the data they're
                        presented after such initiatives. Conversions (quantity) might be
                        higher in "A" or "B" but why (quality)? That's the substance to my
                        question and where I am trying to educate myself.
                        >
                        > Thanks,
                        > Adam
                        > http://analyticsbyadam.blogspot.com
                        >
                        >
                        >
                        > ----- Original Message ----
                        > From: mbchoe <mbchoe@...>
                        > To: webanalytics@yahoogroups.com
                        > Sent: Sunday, April 13, 2008 2:54:04 AM
                        > Subject: [webanalytics] Re: A/B Test Question
                        >
                        >
                        > adam,
                        >
                        > can you be more specific or elaborate with an example? i'm not clear
                        > on what you're asking...
                        >
                        > michael
                        >
                        >
                        > --- In webanalytics@ yahoogroups. com, Adam Berlinger <aberlinger1@ ...>
                        > wrote:
                        > >
                        > > Hello everyone. I'm curious to see how you gauge the success of an
                        > A/B or multivariate test? What are the metrics that are measured? What
                        > are the things that you look for in these types of initiatives? I
                        > appreciate your insight & feedback.
                        > >
                        > > Thanks,
                        > > Adam
                        > > http://analyticsbya dam.blogspot. com
                        > >
                        > > ____________ _________ _________ _________ _________ __
                        > > Do You Yahoo!?
                        > > Tired of spam? Yahoo! Mail has the best spam protection around
                        > > http://mail. yahoo.com
                        > >
                        > > [Non-text portions of this message have been removed]
                        > >
                        >
                        >
                        >
                        >
                        >
                        >
                        ____________________________________________________________________________________
                        > Be a better friend, newshound, and
                        > know-it-all with Yahoo! Mobile. Try it now.
                        http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
                        >
                        > [Non-text portions of this message have been removed]
                        >
                      • mbchoe
                        tim, the last 2 links you posted don t work. i can get to avinash s site but not the other 2 tinyurl s. -mc
                        Message 11 of 17 , Apr 14, 2008
                          tim,
                          the last 2 links you posted don't work. i can get to avinash's site
                          but not the other 2 tinyurl's.

                          -mc


                          --- In webanalytics@yahoogroups.com, "Tim Wilson" <twilson@...> wrote:
                          >
                          >
                          > Avinash tackled this subject a few months ago, so it's worth taking a
                          > look at his thoughts: http://tinyurl.com/2tvkoe.
                          >
                          > His post actually inspired our senior analyst, JT Buser, to develop a
                          > couple of Excel-based tools for assessing the results of an A/B test.
                          > JT's working on getting some supporting documentation and material
                          > together and posted on our web site, but it seemed sufficiently salient
                          > to this discussion that I've gone ahead and uploaded both files to the
                          > "Files" section of this group.
                          >
                          > The simple calculator uses a 90% confidence interval as the default.
                          > It's posted here: http://tinyurl.com/5v2e84
                          >
                          > The advanced calculator allows you to specify a confidence interval.
                          > And, it includes a second worksheet for helping you estimate the sample
                          > size you need to use for additional tests. It's posted here:
                          > http://tinyurl.com/68zuek
                          >
                          > Both calculators provide information beyond simply "the answer" -- how
                          > to actually interpret the results. I can't speak to the formulas behind
                          > the calculator...but that's one of the things we hope to get up on our
                          > web site fairly soon (the link in the spreadsheet will, ultimately, get
                          > redirected to that information).
                          >
                          >
                          > Tim Wilson
                          > http://www.gilliganondata.com
                          > http://www.bulldogsolutions.com
                          >
                          >
                          > ________________________________
                          >
                          > From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
                          > On Behalf Of Adam Berlinger
                          > Sent: Saturday, April 12, 2008 7:46 PM
                          > To: webanalytics@yahoogroups.com
                          > Subject: [webanalytics] A/B Test Question
                          >
                          >
                          >
                          > Hello everyone. I'm curious to see how you gauge the success of an A/B
                          > or multivariate test? What are the metrics that are measured? What are
                          > the things that you look for in these types of initiatives? I appreciate
                          > your insight & feedback.
                          >
                          > Thanks,
                          > Adam
                          > http://analyticsbyadam.blogspot.com
                          > <http://analyticsbyadam.blogspot.com>
                          >
                          > __________________________________________________
                          > Do You Yahoo!?
                          > Tired of spam? Yahoo! Mail has the best spam protection around
                          > http://mail.yahoo.com <http://mail.yahoo.com>
                          >
                          > [Non-text portions of this message have been removed]
                          >
                        • Edward Vielmetti
                          Make sure you don t spend so much time setting up, refining, testing, and analyzing the A/B results that the cost of doing the test far outweighs the
                          Message 12 of 17 , Apr 15, 2008
                            Make sure you don't spend so much time
                            setting up, refining, testing, and analyzing
                            the A/B results that the cost of doing the
                            test far outweighs the incremental benefit
                            from the effort.

                            Multivariate tests, especially in situations where
                            you don't have a lot of traffic but you do have
                            a lot of variations, can be very expensive to
                            get to the point where you "prove" results.

                            Ed

                            On Sat, Apr 12, 2008 at 7:46 PM, Adam Berlinger <aberlinger1@...>
                            wrote:

                            > Hello everyone. I'm curious to see how you gauge the success of an A/B
                            > or multivariate test? What are the metrics that are measured? What are the
                            > things that you look for in these types of initiatives? I appreciate your
                            > insight & feedback.
                            >
                            > Thanks,
                            > Adam
                            > http://analyticsbyadam.blogspot.com
                            >
                            > __________________________________________________
                            > Do You Yahoo!?
                            > Tired of spam? Yahoo! Mail has the best spam protection around
                            > http://mail.yahoo.com
                            >
                            > [Non-text portions of this message have been removed]
                            >
                            >
                            >



                            --
                            Edward Vielmetti +1 734 330 2465


                            [Non-text portions of this message have been removed]
                          • Craig Sullivan
                            Ed, I agree with you on two counts. Our experience is that is better to chuck a load of stuff in the multivariate bucket, stop bickering and just put it to the
                            Message 13 of 17 , Apr 21, 2008
                              Ed,



                              I agree with you on two counts.



                              Our experience is that is better to chuck a load of stuff in the
                              multivariate bucket, stop bickering and just put it to the test. We
                              used to spend time achieving an optimal design but used a raft of
                              techniques to come up with very limited candidate designs ("Yeah, lets
                              go with B, I like the background" etc. etc.)



                              These designs had full input from customer research, usability testing,
                              interaction designers and loads of people (laughs).



                              What we missed though was that we were frequently deluded in our
                              opinions that we knew what would work for conversion. Seemingly odd,
                              illogical and downright trivial changes make huge changes that we could
                              never have guessed. What happens now is that we test loads of stuff,
                              including some counter intuitive elements. We've found out that we
                              don't know really what will work unless we run experimental testing and
                              that with our traffic, small gains make honking big paybacks.



                              In terms of using multivariate testing for low traffic sites, you are
                              going to hit a couple of variables - How much traffic have you really
                              got and How many variables do you want to test?



                              One of the fundamental problems we have for low traffic funnels is that
                              we either need to send more traffic to them or reduce the variants we
                              test. Depending on your business model, it may be best to run a long
                              term and continuous MV test but I find that shorter tests with more
                              iterations are best. So how do we solve this problem?



                              In terms of traffic, you can always run a promotion during the MV test
                              to give you more data and bigger samples to work with. The other way is
                              to continually whittle elements out of the test. Every element you can
                              discard has just halved the number of versions you need to test (and
                              brings down the test time). If you can show elements that contribute
                              poorly towards conversion, you can remove these earlier than the end of
                              the test.



                              I'd love to be a data junkie but I don't have time - fortunately we use
                              Optimost who do all the whittling for us. I'm not suggesting we can get
                              round the issue of traffic but depending on who you use, and what
                              methodology they have for testing, you may be able to bend the rules a
                              bit.



                              Regards,







                              <http://www.lovefilm.com/>

                              Craig Sullivan

                              Product Manager - Digital and Usability
                              LOVEFiLM.com <http://www.lovefilm.com/>

                              No.9 | 6 Portal Way | London | W3 6RU
                              T: (020) 8896 8050 | M: (0)7711 657315 | F: 0208 896 8110
                              craig.sullivan@...





                              ________________________________

                              From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]
                              On Behalf Of Edward Vielmetti
                              Sent: 15 April 2008 14:44
                              To: webanalytics@yahoogroups.com
                              Subject: Re: [webanalytics] A/B Test Question



                              Make sure you don't spend so much time
                              setting up, refining, testing, and analyzing
                              the A/B results that the cost of doing the
                              test far outweighs the incremental benefit
                              from the effort.

                              Multivariate tests, especially in situations where
                              you don't have a lot of traffic but you do have
                              a lot of variations, can be very expensive to
                              get to the point where you "prove" results.

                              Ed

                              On Sat, Apr 12, 2008 at 7:46 PM, Adam Berlinger <aberlinger1@...
                              <mailto:aberlinger1%40yahoo.com> >
                              wrote:

                              > Hello everyone. I'm curious to see how you gauge the success of an A/B
                              > or multivariate test? What are the metrics that are measured? What are
                              the
                              > things that you look for in these types of initiatives? I appreciate
                              your
                              > insight & feedback.
                              >
                              > Thanks,
                              > Adam
                              > http://analyticsbyadam.blogspot.com
                              <http://analyticsbyadam.blogspot.com>
                              >
                              > __________________________________________________
                              > Do You Yahoo!?
                              > Tired of spam? Yahoo! Mail has the best spam protection around
                              > http://mail.yahoo.com <http://mail.yahoo.com>
                              >
                              > [Non-text portions of this message have been removed]
                              >
                              >
                              >

                              --
                              Edward Vielmetti +1 734 330 2465

                              [Non-text portions of this message have been removed]



                              Click here
                              <https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==
                              mHMqCnE!1qsfLNid0urNh!O5HL0OMoqXoKigD!zsoiWCl0voCwpf!F+jTScgm4ywuyzlMmMl
                              K8+VpZLWIkgo6j5Kl1ZFB1f4FNK3oPJE6cCDLy1QCYIuXC35BW5ARy2qh56UsMf8c!IM0qxc
                              1+0pOQfrWXO5JzWsES74xrx58k01SYFyEpEHQoQr> to report this email as spam.



                              LOVEFiLM International Limited is a company registered in England and Wales. Registered Number: 04392195. Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

                              This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, please delete it from your system and notify the sender immediately.

                              This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com


                              [Non-text portions of this message have been removed]
                            Your message has been successfully submitted and would be delivered to recipients shortly.