Loading ...
Sorry, an error occurred while loading the content.

Multivariate Tests and Multi-step process Reporting Best Practices

Expand Messages
  • rlc_60504
    Let s say that you have a multi-step process that you want to track and report on in terms of a conversion funnel. Maybe it is a 5 step process. You use a
    Message 1 of 2 , Mar 8, 2013
    • 0 Attachment
      Let's say that you have a multi-step process that you want to track and report on in terms of a conversion funnel. Maybe it is a 5 step process.

      You use a vendor to perform multivariate tests on that same process. They take the 5 different steps and shuffle them around.

      This makes your step tags fire in different orders which impacts the integrity of your conversion funnel reports in Net Insight, Webtrends, Google, etc... (Example...now step 3 tags fire first, step 5 tags fire 2nd, step 1 tags fire 3rd, etc...etc...) Your conversion funnel comes out looking like a conversion anvil.

      Are there any best practices for using your web analytics tool in an enviornment where multivariate tests are going on?

      Beyond getting the tags to be more fluid and dynamic, there is also the concept of the reports, which it would seem need to be fluid and dynamic. In the course of a 3 step flow with multi variate testing going on, it seems like you would need 9 reports to report on the 9 different incarnations that may be possible with a 3 step flow. You would want to see those broken down by each flow possibility to understand their performance.

      So, are their best practices around implementing reporting in your webanalytics tool OR is it just best practice to leave the reporting up to the multivariate testing tool vendor during multivariate tests and not rely on your web analytics tool during that time?

      Thanks!!!

      RLC
    • Craig Sullivan
      No, You should always have analytics tracking to compare with the testing tool - this is vital to check any instrumentation issues. The problem here is that
      Message 2 of 2 , Mar 9, 2013
      • 0 Attachment
        No,



        You should always have analytics tracking to compare with the testing tool -
        this is vital to check any instrumentation issues.



        The problem here is that they are varying the steps, without you knowing
        what order is being used.



        Consider setting a custom variable or some other integration with analytics
        so that you record two variables on each page:



        Step=1

        Order=51243



        Now you can then segment the analytics data by the specific order seen (to
        see conversion)



        The thing you've learned is that you sometimes need to re-instrument or find
        a way to make the testing tool and analytics co-exist. I've had so much
        value out of having both aligned - that it's worth sitting down with the
        developers, analytics guys and testing vendor. And say - "Make this work"



        C.



        Craig Sullivan
        Conversion Optimisation Director

        +44-(0)7711-657315
        +44-(0)208-852-8114

        http://www.linkedin.com/in/craigsullivan

        http://www.twitter.com/OptimiseOrDie



        From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com] On
        Behalf Of rlc_60504
        Sent: 08 March 2013 18:48
        To: webanalytics@yahoogroups.com
        Subject: [webanalytics] Multivariate Tests and Multi-step process Reporting
        Best Practices





        Let's say that you have a multi-step process that you want to track and
        report on in terms of a conversion funnel. Maybe it is a 5 step process.

        You use a vendor to perform multivariate tests on that same process. They
        take the 5 different steps and shuffle them around.

        This makes your step tags fire in different orders which impacts the
        integrity of your conversion funnel reports in Net Insight, Webtrends,
        Google, etc... (Example...now step 3 tags fire first, step 5 tags fire 2nd,
        step 1 tags fire 3rd, etc...etc...) Your conversion funnel comes out looking
        like a conversion anvil.

        Are there any best practices for using your web analytics tool in an
        enviornment where multivariate tests are going on?

        Beyond getting the tags to be more fluid and dynamic, there is also the
        concept of the reports, which it would seem need to be fluid and dynamic. In
        the course of a 3 step flow with multi variate testing going on, it seems
        like you would need 9 reports to report on the 9 different incarnations that
        may be possible with a 3 step flow. You would want to see those broken down
        by each flow possibility to understand their performance.

        So, are their best practices around implementing reporting in your
        webanalytics tool OR is it just best practice to leave the reporting up to
        the multivariate testing tool vendor during multivariate tests and not rely
        on your web analytics tool during that time?

        Thanks!!!

        RLC





        [Non-text portions of this message have been removed]
      Your message has been successfully submitted and would be delivered to recipients shortly.