Loading ...
Sorry, an error occurred while loading the content.

28031Re: What makes "iterative testing" iterative?

Expand Messages
  • Dave
    Jun 1, 2011
      Hi Matt,

      Thanks very much for your kind comments - I'm pleased you're enjoying the posts and finding them useful.

      In my post, I was aiming to show that even without A/B testing or MVT, it's still possible to 'do' testing at a basic level. This kind of sequential testing isn't as reliable or definitive as simultaneous testing (A/B or MVT) but can still be used to provide useful insights. I've already planned to follow up this post with an intro to A/B and then to MVT, so you've already spotted where I'm going with this mini-series!

      And yes, in the sequential testing I've been involved with, we have seen significant variation depending on external factors - for example, the weather (fewer people go online when the weather's good), sports events, current affairs and so on.

      Thanks again

      David


      --- In webanalytics@yahoogroups.com, Matthew Sundquist <matt.sundquist@...> wrote:
      >
      > Hi Dave,
      >
      > Thanks for sending these out. I enjoy your posts, and find them thoughtful
      > and very well-written.
      >
      > If I read this one right (and I may not have), it seems you're advocating
      > testing each version once, for a week, over a five-week period. I drew this
      > idea from this section:
      >
      > "Suppose I've tested the five following page headlines, and achieved the
      > following points scores (per day), running each one for a week, so that the
      > total test lasted five weeks."
      >
      > I wonder if it might be more productive to run them all at once for five
      > weeks or run a multivariate test. This gives you live results, and means
      > you can eliminate the versions with lower gains as you go forward. Then you
      > can gradually pit the best two or three against one another to maximize
      > conversions during testing. This might help with calculating statistical
      > significance, running corrections on your data, gathering guiding
      > principles, and avoiding the sampling bias associated with a weekend,
      > holiday, seasonal dip, sporting event (I imagine the Champions League or NBA
      > Finals might spike traffic on certain sites and products), news event, etc.
      >
      > Perhaps this is what you meant, and if so, please forgive me. It's clear
      > you know a good deal more about these matters than I do, and I am eager to
      > hear your view so I can understand this more.
      >
      > Thanks for sending these out, and I'll look forward to your next post.
      >
      > All the best,
      > Matt
      >
      > On Tue, May 31, 2011 at 12:02 PM, Dave <tregowandave@...> wrote:
      >
      > >
      > >
      > > Hello again group,
      > >
      > > Thanks to all for your comments on my previous posts - I'm back again
      > > looking at something that came up during the recent Omniture EMEA Summit,
      > > namely iterative testing. What makes it iterative, what's the point, and
      > > what's different from normal testing?
      > >
      > > http://bit.ly/j2yZof
      > >
      > > As ever, comments sought and welcomed.
      > >
      > > Thanks
      > >
      > > David
      > >
      > >
      > >
      >
      >
      > [Non-text portions of this message have been removed]
      >
    • Show all 6 messages in this topic