Loading ...
Sorry, an error occurred while loading the content.

multivariate testing

Expand Messages
  • nevertrustab
    I just read how important multivariate testing was b/c in some case it would have taken 96 times as long (as in impossible) if they had just done A/B testing.
    Message 1 of 2 , Jul 5 5:07 PM
    • 0 Attachment
      I just read how important multivariate testing was b/c in some case it
      would have taken 96 times as long (as in impossible) if they had just
      done A/B testing. I cant say for sure whether this was just an
      exception used to make multivariate testing look extremely cool, but
      its not the first time that I hear about its application in web
      analytics.

      Is there any particular statistical knowledge necessary for doing
      multivariate testing? If so what kind of knowledge? Multiple
      regression analysis/testing? or something completely different?

      My guess would be it is about multiple regression analysis but in a
      more complicated way than I'm used to (as I'm only used to testing one
      variable at a time in multiple regression models..)

      thx!
    • romanojon
      Hello there... Its been our experience that both Multivariate tests and A/B tests have their virtues in the appropriate context. We use A/B tests for layout
      Message 2 of 2 , Jul 7 4:53 AM
      • 0 Attachment
        Hello there...

        Its been our experience that both Multivariate tests and A/B tests
        have their virtues in the appropriate context. We use A/B tests for
        layout comparisons and have used both linear methods and technical
        substitution (similar to traffic splitting) to calculate results.
        Depending on your goal and the point on the path to the goal that the
        page/element being tested, time frames will vary. Depending on traffic
        you may be able to perform a successful A/B in 7-10 days.

        When you are testing sections, or layout swapping and shifting,
        particular elements and other features which may positively influence
        your conversion rate, you may want to take the time to examine
        multivariate options. Its advisable, at this point, to be diligent in
        your efforts to understand the process, the available tools, and
        methodologies.

        Multivariate testing can be a very complicated process if performing
        it in-house. On the other hand, it can be very simple under the right
        circumstances. There are a few programs which are available to help
        you split traffic to variations of pages or individual elements. My
        recommendation is that you explore the usefulness of the FREE Google
        Website Optimizer to help get a grip on the rudimentary ideas which
        govern MVT success. If you have the resources and the support, it
        might be better to invest in a third party like Offermatica, SiteSpect
        or Vertster. Each of these will use its own way to replace control
        content with variations of pages, in flow, with the specified
        alternates and produce results for you in an interface which is pretty
        easily understood.

        Last point, as not to run on too much, either A/B or multivariate
        testing can run long if you choose a single goal. Multivariate tests
        can be particularly lengthy with the compounding combinations. If you
        essentially run 256 different page combinations at the primary stages
        of your conversion funnel, in a full-factorial methodology based on a
        single goal to indicate conversion, it could literally take months to
        get solid results. For that purpose, we have, in circumstances, used
        Taguchi statistical methods to shorten tests and provide equally
        beneficial data with regard to certain methods. Usually, we run a
        follow-up on these to be sure.

        If you have any questions on technical issues associated with testing,
        please feel free to give me a yell or post questions or comments on my
        blog. I try to only write there when I feel I have something to offer
        so, its not a day-to-day account of my stuff. However, I think it
        might be useful to some people trying to understand how and what they
        should be testing.

        Sincerely,

        Daniel W. Shields
        http://danalytics.blogspot.com

        PS: We'll actually be migrating the blog to its new home on the
        CableOrganizer.com domain shortly. I'll post a note when that happens.
        --- In webanalytics@yahoogroups.com, "nevertrustab" <patriccc@...>

        wrote:
        >
        > I just read how important multivariate testing was b/c in some case it
        > would have taken 96 times as long (as in impossible) if they had just
        > done A/B testing. I cant say for sure whether this was just an
        > exception used to make multivariate testing look extremely cool, but
        > its not the first time that I hear about its application in web
        > analytics.
        >
        > Is there any particular statistical knowledge necessary for doing
        > multivariate testing? If so what kind of knowledge? Multiple
        > regression analysis/testing? or something completely different?
        >
        > My guess would be it is about multiple regression analysis but in a
        > more complicated way than I'm used to (as I'm only used to testing one
        > variable at a time in multiple regression models..)
        >
        > thx!
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.