- I just read how important multivariate testing was b/c in some case it
would have taken 96 times as long (as in impossible) if they had just
done A/B testing. I cant say for sure whether this was just an
exception used to make multivariate testing look extremely cool, but
its not the first time that I hear about its application in web
Is there any particular statistical knowledge necessary for doing
multivariate testing? If so what kind of knowledge? Multiple
regression analysis/testing? or something completely different?
My guess would be it is about multiple regression analysis but in a
more complicated way than I'm used to (as I'm only used to testing one
variable at a time in multiple regression models..)
- Hello there...
Its been our experience that both Multivariate tests and A/B tests
have their virtues in the appropriate context. We use A/B tests for
layout comparisons and have used both linear methods and technical
substitution (similar to traffic splitting) to calculate results.
Depending on your goal and the point on the path to the goal that the
page/element being tested, time frames will vary. Depending on traffic
you may be able to perform a successful A/B in 7-10 days.
When you are testing sections, or layout swapping and shifting,
particular elements and other features which may positively influence
your conversion rate, you may want to take the time to examine
multivariate options. Its advisable, at this point, to be diligent in
your efforts to understand the process, the available tools, and
Multivariate testing can be a very complicated process if performing
it in-house. On the other hand, it can be very simple under the right
circumstances. There are a few programs which are available to help
you split traffic to variations of pages or individual elements. My
recommendation is that you explore the usefulness of the FREE Google
Website Optimizer to help get a grip on the rudimentary ideas which
govern MVT success. If you have the resources and the support, it
might be better to invest in a third party like Offermatica, SiteSpect
or Vertster. Each of these will use its own way to replace control
content with variations of pages, in flow, with the specified
alternates and produce results for you in an interface which is pretty
Last point, as not to run on too much, either A/B or multivariate
testing can run long if you choose a single goal. Multivariate tests
can be particularly lengthy with the compounding combinations. If you
essentially run 256 different page combinations at the primary stages
of your conversion funnel, in a full-factorial methodology based on a
single goal to indicate conversion, it could literally take months to
get solid results. For that purpose, we have, in circumstances, used
Taguchi statistical methods to shorten tests and provide equally
beneficial data with regard to certain methods. Usually, we run a
follow-up on these to be sure.
If you have any questions on technical issues associated with testing,
please feel free to give me a yell or post questions or comments on my
blog. I try to only write there when I feel I have something to offer
so, its not a day-to-day account of my stuff. However, I think it
might be useful to some people trying to understand how and what they
should be testing.
Daniel W. Shields
PS: We'll actually be migrating the blog to its new home on the
CableOrganizer.com domain shortly. I'll post a note when that happens.
--- In firstname.lastname@example.org, "nevertrustab" <patriccc@...>
> I just read how important multivariate testing was b/c in some case it
> would have taken 96 times as long (as in impossible) if they had just
> done A/B testing. I cant say for sure whether this was just an
> exception used to make multivariate testing look extremely cool, but
> its not the first time that I hear about its application in web
> Is there any particular statistical knowledge necessary for doing
> multivariate testing? If so what kind of knowledge? Multiple
> regression analysis/testing? or something completely different?
> My guess would be it is about multiple regression analysis but in a
> more complicated way than I'm used to (as I'm only used to testing one
> variable at a time in multiple regression models..)