Loading ...
Sorry, an error occurred while loading the content.

Re: [webanalytics] What makes "iterative testing" iterative?

Expand Messages
  • Craig Sullivan
    Hi, I agree with Matthew - you need your head examined if you want to do multiple A/B tests at different time periods. Let s take an example - you have an
    Message 1 of 6 , Jun 1, 2011
    • 0 Attachment
      Hi,

      I agree with Matthew - you need your head examined if you want to do
      multiple A/B tests at different time periods.

      Let's take an example - you have an 'Orange' homepage and a 'Blue' homepage.
      In July, Orange wins by 10% and in August, Blue wins by 5%. What should
      you do?

      The issue here is that traffic patterns are ALWAYS changing on your site -
      by spreading A/B splits over time, you're going to get (potentially) rubbish
      data. Rubbish? Why? - unless you are segmenting your data capture (to
      compare conversion by segment) and also monitoring your incoming traffic mix
      (paid, organic, direct etc.) then you have no way of knowing if the
      difference is down to your creative or an external event.

      The answer is to run the A/B/C/D versions simultaneously or use a
      multi-variate test. I've discovered this issue to my cost and now advise
      people to do multivariate for this reason - there are too many companies
      doing time dislocated A/B tests and scratching their heads over the
      'strange' results they get.

      What I call iterative testing is running different A/B/C/D or multivariate
      tests in wave after wave. Each wave learns from the last (this worked, that
      failed, this needs tweaked) with each wave aiming to improve the result,
      based on analysis and new ideas fed into each test. The important thing -
      always test with the same traffic.

      Regards,


      C.

      On Wed, Jun 1, 2011 at 12:32 AM, Matthew Sundquist <matt.sundquist@...
      > wrote:

      >
      >
      > Hi Dave,
      >
      > Thanks for sending these out. I enjoy your posts, and find them thoughtful
      > and very well-written.
      >
      > If I read this one right (and I may not have), it seems you're advocating
      > testing each version once, for a week, over a five-week period. I drew this
      > idea from this section:
      >
      > "Suppose I've tested the five following page headlines, and achieved the
      > following points scores (per day), running each one for a week, so that the
      > total test lasted five weeks."
      >
      > I wonder if it might be more productive to run them all at once for five
      > weeks or run a multivariate test. This gives you live results, and means
      > you can eliminate the versions with lower gains as you go forward. Then you
      > can gradually pit the best two or three against one another to maximize
      > conversions during testing. This might help with calculating statistical
      > significance, running corrections on your data, gathering guiding
      > principles, and avoiding the sampling bias associated with a weekend,
      > holiday, seasonal dip, sporting event (I imagine the Champions League or
      > NBA
      > Finals might spike traffic on certain sites and products), news event, etc.
      >
      > Perhaps this is what you meant, and if so, please forgive me. It's clear
      > you know a good deal more about these matters than I do, and I am eager to
      > hear your view so I can understand this more.
      >
      > Thanks for sending these out, and I'll look forward to your next post.
      >
      > All the best,
      > Matt
      >
      >
      > On Tue, May 31, 2011 at 12:02 PM, Dave <tregowandave@...> wrote:
      >
      > >
      > >
      > > Hello again group,
      > >
      > > Thanks to all for your comments on my previous posts - I'm back again
      > > looking at something that came up during the recent Omniture EMEA Summit,
      > > namely iterative testing. What makes it iterative, what's the point, and
      > > what's different from normal testing?
      > >
      > > http://bit.ly/j2yZof
      > >
      > > As ever, comments sought and welcomed.
      > >
      > > Thanks
      > >
      > > David
      > >
      > >
      > >
      >
      > [Non-text portions of this message have been removed]
      >
      >
      >



      --
      Craig Sullivan
      http://www.linkedin.com/in/craigsullivan
      +44-(0)7711-657315
      +44-(0)208-318-9290

      Not sent from my blackberry <grin>


      [Non-text portions of this message have been removed]
    • Dave
      Hi Matt, Thanks very much for your kind comments - I m pleased you re enjoying the posts and finding them useful. In my post, I was aiming to show that even
      Message 2 of 6 , Jun 1, 2011
      • 0 Attachment
        Hi Matt,

        Thanks very much for your kind comments - I'm pleased you're enjoying the posts and finding them useful.

        In my post, I was aiming to show that even without A/B testing or MVT, it's still possible to 'do' testing at a basic level. This kind of sequential testing isn't as reliable or definitive as simultaneous testing (A/B or MVT) but can still be used to provide useful insights. I've already planned to follow up this post with an intro to A/B and then to MVT, so you've already spotted where I'm going with this mini-series!

        And yes, in the sequential testing I've been involved with, we have seen significant variation depending on external factors - for example, the weather (fewer people go online when the weather's good), sports events, current affairs and so on.

        Thanks again

        David


        --- In webanalytics@yahoogroups.com, Matthew Sundquist <matt.sundquist@...> wrote:
        >
        > Hi Dave,
        >
        > Thanks for sending these out. I enjoy your posts, and find them thoughtful
        > and very well-written.
        >
        > If I read this one right (and I may not have), it seems you're advocating
        > testing each version once, for a week, over a five-week period. I drew this
        > idea from this section:
        >
        > "Suppose I've tested the five following page headlines, and achieved the
        > following points scores (per day), running each one for a week, so that the
        > total test lasted five weeks."
        >
        > I wonder if it might be more productive to run them all at once for five
        > weeks or run a multivariate test. This gives you live results, and means
        > you can eliminate the versions with lower gains as you go forward. Then you
        > can gradually pit the best two or three against one another to maximize
        > conversions during testing. This might help with calculating statistical
        > significance, running corrections on your data, gathering guiding
        > principles, and avoiding the sampling bias associated with a weekend,
        > holiday, seasonal dip, sporting event (I imagine the Champions League or NBA
        > Finals might spike traffic on certain sites and products), news event, etc.
        >
        > Perhaps this is what you meant, and if so, please forgive me. It's clear
        > you know a good deal more about these matters than I do, and I am eager to
        > hear your view so I can understand this more.
        >
        > Thanks for sending these out, and I'll look forward to your next post.
        >
        > All the best,
        > Matt
        >
        > On Tue, May 31, 2011 at 12:02 PM, Dave <tregowandave@...> wrote:
        >
        > >
        > >
        > > Hello again group,
        > >
        > > Thanks to all for your comments on my previous posts - I'm back again
        > > looking at something that came up during the recent Omniture EMEA Summit,
        > > namely iterative testing. What makes it iterative, what's the point, and
        > > what's different from normal testing?
        > >
        > > http://bit.ly/j2yZof
        > >
        > > As ever, comments sought and welcomed.
        > >
        > > Thanks
        > >
        > > David
        > >
        > >
        > >
        >
        >
        > [Non-text portions of this message have been removed]
        >
      • Aaron Alberg
        One potential reason to run iterative tests over larger simultaneous tests is traffic volumes. Spreading low traffic over a large number of variations
        Message 3 of 6 , Jun 1, 2011
        • 0 Attachment
          One potential reason to run iterative tests over larger simultaneous tests is traffic volumes. Spreading low traffic over a large number of variations increases the amount of time to achieve statistical significance, thus preventing you from scientifically removing the poorly performing versions. Further, multiple elements of the site can be changing over time so that having one test run for a long time potentially ignores the interactions between the test content and other changes on the site.

          Regards,

          AARON ALBERG

          ASCENTIUM
          425-679-5879 [direct]
          425-519-7795 [main]

          ascentium.com

          Posted by: "Matthew Sundquist" matt.sundquist@... <mailto:matt.sundquist@...%20>
          Tue May 31, 2011 11:04 pm (PDT)
          Hi Dave,

          Thanks for sending these out. I enjoy your posts, and find them thoughtful
          and very well-written.

          If I read this one right (and I may not have), it seems you're advocating
          testing each version once, for a week, over a five-week period. I drew this
          idea from this section:

          "Suppose I've tested the five following page headlines, and achieved the
          following points scores (per day), running each one for a week, so that the
          total test lasted five weeks."

          I wonder if it might be more productive to run them all at once for five
          weeks or run a multivariate test. This gives you live results, and means
          you can eliminate the versions with lower gains as you go forward. Then you
          can gradually pit the best two or three against one another to maximize
          conversions during testing. This might help with calculating statistical
          significance, running corrections on your data, gathering guiding
          principles, and avoiding the sampling bias associated with a weekend,
          holiday, seasonal dip, sporting event (I imagine the Champions League or NBA
          Finals might spike traffic on certain sites and products), news event, etc.

          Perhaps this is what you meant, and if so, please forgive me. It's clear
          you know a good deal more about these matters than I do, and I am eager to
          hear your view so I can understand this more.

          Thanks for sending these out, and I'll look forward to your next post.

          All the best,
          Matt


          [Non-text portions of this message have been removed]
        • Craig Sullivan
          Dave, Agree with you completely - using clicktale to iron out bugs, issues and other problems (that are a no brainer) is all part of our optimisation toolkit.
          Message 4 of 6 , Jun 3, 2011
          • 0 Attachment
            Dave,

            Agree with you completely - using clicktale to iron out bugs, issues and
            other problems (that are a no brainer) is all part of our optimisation
            toolkit.

            For example, do you really need to run an A/B test to tell you that your
            form validation sucks? Nah - just fix the stuff you can or a/b test a batch
            of fixes in one go.

            Agreed also on sequential testing. There is a third way (instead of all
            simultaneously or all separate) - and that is to run testing in waves,
            discarding poor performers. This way you test stuff against each other (so
            noting external factors) but also get the benefit of shorter test iterations
            and less variants to wade through.

            Autonomy work closely with us on doing this and it works to get results
            extremely fast, without boiling the ocean in testing terms.

            Regards,

            C.

            On Wed, Jun 1, 2011 at 8:10 PM, Dave <tregowandave@...> wrote:

            >
            >
            > Hi Matt,
            >
            > Thanks very much for your kind comments - I'm pleased you're enjoying the
            > posts and finding them useful.
            >
            > In my post, I was aiming to show that even without A/B testing or MVT, it's
            > still possible to 'do' testing at a basic level. This kind of sequential
            > testing isn't as reliable or definitive as simultaneous testing (A/B or MVT)
            > but can still be used to provide useful insights. I've already planned to
            > follow up this post with an intro to A/B and then to MVT, so you've already
            > spotted where I'm going with this mini-series!
            >
            > And yes, in the sequential testing I've been involved with, we have seen
            > significant variation depending on external factors - for example, the
            > weather (fewer people go online when the weather's good), sports events,
            > current affairs and so on.
            >
            > Thanks again
            >
            > David
            >
            >
            > --- In webanalytics@yahoogroups.com, Matthew Sundquist <matt.sundquist@...>
            > wrote:
            > >
            > > Hi Dave,
            > >
            > > Thanks for sending these out. I enjoy your posts, and find them
            > thoughtful
            > > and very well-written.
            > >
            > > If I read this one right (and I may not have), it seems you're advocating
            > > testing each version once, for a week, over a five-week period. I drew
            > this
            > > idea from this section:
            > >
            > > "Suppose I've tested the five following page headlines, and achieved the
            > > following points scores (per day), running each one for a week, so that
            > the
            > > total test lasted five weeks."
            > >
            > > I wonder if it might be more productive to run them all at once for five
            > > weeks or run a multivariate test. This gives you live results, and means
            > > you can eliminate the versions with lower gains as you go forward. Then
            > you
            > > can gradually pit the best two or three against one another to maximize
            > > conversions during testing. This might help with calculating statistical
            > > significance, running corrections on your data, gathering guiding
            > > principles, and avoiding the sampling bias associated with a weekend,
            > > holiday, seasonal dip, sporting event (I imagine the Champions League or
            > NBA
            > > Finals might spike traffic on certain sites and products), news event,
            > etc.
            > >
            > > Perhaps this is what you meant, and if so, please forgive me. It's clear
            > > you know a good deal more about these matters than I do, and I am eager
            > to
            > > hear your view so I can understand this more.
            > >
            > > Thanks for sending these out, and I'll look forward to your next post.
            > >
            > > All the best,
            > > Matt
            > >
            > > On Tue, May 31, 2011 at 12:02 PM, Dave <tregowandave@...> wrote:
            > >
            > > >
            > > >
            > > > Hello again group,
            > > >
            > > > Thanks to all for your comments on my previous posts - I'm back again
            > > > looking at something that came up during the recent Omniture EMEA
            > Summit,
            > > > namely iterative testing. What makes it iterative, what's the point,
            > and
            > > > what's different from normal testing?
            > > >
            > > > http://bit.ly/j2yZof
            > > >
            > > > As ever, comments sought and welcomed.
            > > >
            > > > Thanks
            > > >
            > > > David
            >
            > > >
            > > >
            > > >
            > >
            > >
            > > [Non-text portions of this message have been removed]
            > >
            >
            >
            >



            --
            Craig Sullivan
            http://www.linkedin.com/in/craigsullivan
            +44-(0)7711-657315
            +44-(0)208-318-9290

            Not sent from my blackberry <grin>


            [Non-text portions of this message have been removed]
          Your message has been successfully submitted and would be delivered to recipients shortly.