Loading ...
Sorry, an error occurred while loading the content.

RE: [webanalytics] A/B testing "by hand"

Expand Messages
  • Melissa Cornwell
    If you can create a report that looks at cookie values then you can do it. Just make all the different variations a different cookie value and create reports
    Message 1 of 8 , Dec 1, 2006
    • 0 Attachment
      If you can create a report that looks at cookie values then you can do it.
      Just make all the different variations a different cookie value and create
      reports based on those values to get at the KPIs you care about.



      I did this a few times and it takes a bit to set up the reports and do the
      testing, but otherwise it works out great and doesn't cost you anything.



      Melissa



      _____

      From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com] On
      Behalf Of Benjamin Rosenthal
      Sent: Thursday, November 30, 2006 9:43 AM
      To: webanalytics@yahoogroups.com
      Subject: [webanalytics] A/B testing "by hand"



      Hi folks,

      I'd like to do some A/B testing of different elements on the home
      page of our website. Essentially, I'd like to figure out if changing
      different elements affects our bounce rate and/or conversion rate.

      I'm wondering...

      Without using any testing software specifically designed for this
      sort of thing is there one agreed upon "gold standard" for doing A/B
      testing manually? In other words, what steps would you use to test
      this? I'm sure there are lots of different ways that people on the
      board are doing this right now. I know I can easily change elements
      and test during different time periods, but ideally, I'd like to test
      2 or 3 elements dynamically during the same time period.

      The only tools I have at my disposal are Google Analytics and
      Coldfusion. I can modify all the code on the site, but don't have
      easy access to the log files or the server application settings.

      (I'm also not opposed to hearing about any "free" packages out there
      for A/B testing? I know that Google has Google Optimizer in Beta
      testing right now.)

      Any suggestions, recipes, or links would be appreciated.

      thanks!

      -Benjamin





      [Non-text portions of this message have been removed]
    • Brad Warthan
      Ben, did you ever get a response back from this? Brad Brad Warthan (812) 457-4479 ... From: webanalytics@yahoogroups.com
      Message 2 of 8 , Dec 4, 2006
      • 0 Attachment
        Ben, did you ever get a response back from this?

        Brad

        Brad Warthan
        (812) 457-4479

        -----Original Message-----
        From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]On
        Behalf Of Benjamin Rosenthal
        Sent: Thursday, November 30, 2006 12:43 PM
        To: webanalytics@yahoogroups.com
        Subject: [webanalytics] A/B testing "by hand"


        Hi folks,

        I'd like to do some A/B testing of different elements on the home
        page of our website. Essentially, I'd like to figure out if changing
        different elements affects our bounce rate and/or conversion rate.

        I'm wondering...

        Without using any testing software specifically designed for this
        sort of thing is there one agreed upon "gold standard" for doing A/B
        testing manually? In other words, what steps would you use to test
        this? I'm sure there are lots of different ways that people on the
        board are doing this right now. I know I can easily change elements
        and test during different time periods, but ideally, I'd like to test
        2 or 3 elements dynamically during the same time period.

        The only tools I have at my disposal are Google Analytics and
        Coldfusion. I can modify all the code on the site, but don't have
        easy access to the log files or the server application settings.

        (I'm also not opposed to hearing about any "free" packages out there
        for A/B testing? I know that Google has Google Optimizer in Beta
        testing right now.)

        Any suggestions, recipes, or links would be appreciated.

        thanks!

        -Benjamin






        [Non-text portions of this message have been removed]
      • HAKIMALY@NETSCAPE.NET
        Are you looking for a tool to serve the ads or a tool to analyze the results in a statistically valid way? Hakim ... From: bradwarthan@gmail.com To:
        Message 3 of 8 , Dec 5, 2006
        • 0 Attachment
          Are you looking for a tool to serve the ads or a tool to analyze the results in a statistically valid way? Hakim


          -----Original Message-----
          From: bradwarthan@...
          To: webanalytics@yahoogroups.com
          Sent: Mon, 4 Dec 2006 11:46 AM
          Subject: RE: [webanalytics] A/B testing "by hand"


          Ben, did you ever get a response back from this?

          Brad

          Brad Warthan
          (812) 457-4479

          -----Original Message-----
          From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]On
          Behalf Of Benjamin Rosenthal
          Sent: Thursday, November 30, 2006 12:43 PM
          To: webanalytics@yahoogroups.com
          Subject: [webanalytics] A/B testing "by hand"

          Hi folks,

          I'd like to do some A/B testing of different elements on the home
          page of our website. Essentially, I'd like to figure out if changing
          different elements affects our bounce rate and/or conversion rate.

          I'm wondering...

          Without using any testing software specifically designed for this
          sort of thing is there one agreed upon "gold standard" for doing A/B
          testing manually? In other words, what steps would you use to test
          this? I'm sure there are lots of different ways that people on the
          board are doing this right now. I know I can easily change elements
          and test during different time periods, but ideally, I'd like to test
          2 or 3 elements dynamically during the same time period.

          The only tools I have at my disposal are Google Analytics and
          Coldfusion. I can modify all the code on the site, but don't have
          easy access to the log files or the server application settings.

          (I'm also not opposed to hearing about any "free" packages out there
          for A/B testing? I know that Google has Google Optimizer in Beta
          testing right now.)

          Any suggestions, recipes, or links would be appreciated.

          thanks!

          -Benjamin

          [Non-text portions of this message have been removed]



          ________________________________________________________________________
          Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-leading spam and email virus protection.


          [Non-text portions of this message have been removed]
        • jsbellsey
          Hi Benjamin, This should be pretty easy to set up. I ve written a custom suite in ASP, so doing it in CF would be similar: * create a random cookie generator,
          Message 4 of 8 , Dec 5, 2006
          • 0 Attachment
            Hi Benjamin,

            This should be pretty easy to set up. I've written a custom suite in
            ASP, so doing it in CF would be similar:

            * create a random cookie generator, up to N values for each test
            * track page views by calling a javascript function which loads a
            server-side page. The server page writes the page view into a
            database, with any associated information you need (or just the page
            version).
            * track conversions by calling a different javascript function,
            loading another server-side page. This just closes the loop, and might
            also write out custom info into the database.


            Analyzing the database is a creative process. I have a nice chart
            program installed that makes it easy for my marketing folks to
            interpret results. Here's what I present as a starting point:

            * Page views
            * Conversions (and conversion rate)
            * Avg order value (an extra param is passed on the conversion tracker)
            * Total $ if all visitors had been shown version N


            Writing custom A/B software is a total pleasure, but it's not a small
            job to do it well. Getting the info into the database is the easy
            part; getting it out is where things get interesting...

            Regards,
            Jeff Bellsey


            --- In webanalytics@yahoogroups.com, "Melissa Cornwell"
            <melissa.cornwell@...> wrote:
            >
            > If you can create a report that looks at cookie values then you can
            do it.
            > Just make all the different variations a different cookie value and
            create
            > reports based on those values to get at the KPIs you care about.
            >
            >
            > Melissa
            >
            >
            >
            > _____
            >
            > From: webanalytics@yahoogroups.com
            [mailto:webanalytics@yahoogroups.com] On
            > Behalf Of Benjamin Rosenthal
            > Sent: Thursday, November 30, 2006 9:43 AM
            > To: webanalytics@yahoogroups.com
            > Subject: [webanalytics] A/B testing "by hand"
            >
            >
            >
            > Hi folks,
            >
            > I'd like to do some A/B testing of different elements on the home
            > page of our website. Essentially, I'd like to figure out if changing
            > different elements affects our bounce rate and/or conversion rate.
            >
            > I'm wondering...
            >
            > Without using any testing software specifically designed for this
            > sort of thing is there one agreed upon "gold standard" for doing A/B
            > testing manually? In other words, what steps would you use to test
            > this? I'm sure there are lots of different ways that people on the
            > board are doing this right now. I know I can easily change elements
            > and test during different time periods, but ideally, I'd like to test
            > 2 or 3 elements dynamically during the same time period.
            >
            > The only tools I have at my disposal are Google Analytics and
            > Coldfusion. I can modify all the code on the site, but don't have
            > easy access to the log files or the server application settings.
            >
            > -Benjamin
            >
            >
            >
          • Sarah Brown
            Ben, I recommend the Fundamentals of Online Testing course offered via Marketing Experiments (http://www.marketingexperiments.com) ... which happens to
            Message 5 of 8 , Dec 6, 2006
            • 0 Attachment
              Ben,

              I recommend the "Fundamentals of Online Testing" course
              offered via Marketing Experiments
              (http://www.marketingexperiments.com) ... which happens
              to include a free widget for conducting A/B tests.

              Best,
              Sarah

              ________________________________

              From: webanalytics@yahoogroups.com
              [mailto:webanalytics@yahoogroups.com] On Behalf Of
              HAKIMALY@...
              Sent: Tuesday, December 05, 2006 12:27 PM
              To: webanalytics@yahoogroups.com
              Subject: Re: [webanalytics] A/B testing "by hand"



              Are you looking for a tool to serve the ads or a tool to
              analyze the results in a statistically valid way? Hakim


              -----Original Message-----
              From: bradwarthan@...
              <mailto:bradwarthan%40gmail.com>
              To: webanalytics@yahoogroups.com
              <mailto:webanalytics%40yahoogroups.com>
              Sent: Mon, 4 Dec 2006 11:46 AM
              Subject: RE: [webanalytics] A/B testing "by hand"

              Ben, did you ever get a response back from this?

              Brad

              Brad Warthan
              (812) 457-4479

              -----Original Message-----
              From: webanalytics@yahoogroups.com
              <mailto:webanalytics%40yahoogroups.com>
              [mailto:webanalytics@yahoogroups.com
              <mailto:webanalytics%40yahoogroups.com> ]On
              Behalf Of Benjamin Rosenthal
              Sent: Thursday, November 30, 2006 12:43 PM
              To: webanalytics@yahoogroups.com
              <mailto:webanalytics%40yahoogroups.com>
              Subject: [webanalytics] A/B testing "by hand"

              Hi folks,

              I'd like to do some A/B testing of different elements on
              the home
              page of our website. Essentially, I'd like to figure out
              if changing
              different elements affects our bounce rate and/or
              conversion rate.

              I'm wondering...

              Without using any testing software specifically designed
              for this
              sort of thing is there one agreed upon "gold standard"
              for doing A/B
              testing manually? In other words, what steps would you
              use to test
              this? I'm sure there are lots of different ways that
              people on the
              board are doing this right now. I know I can easily
              change elements
              and test during different time periods, but ideally, I'd
              like to test
              2 or 3 elements dynamically during the same time period.

              The only tools I have at my disposal are Google Analytics
              and
              Coldfusion. I can modify all the code on the site, but
              don't have
              easy access to the log files or the server application
              settings.

              (I'm also not opposed to hearing about any "free"
              packages out there
              for A/B testing? I know that Google has Google Optimizer
              in Beta
              testing right now.)

              Any suggestions, recipes, or links would be appreciated.

              thanks!

              -Benjamin

              [Non-text portions of this message have been removed]

              _________________________________________________________
              _
              Check Out the new free AIM(R) Mail -- 2 GB of storage and
              industry-leading spam and email virus protection.

              [Non-text portions of this message have been removed]






              [Non-text portions of this message have been removed]
            • Benjamin Rosenthal
              Hi Hakim, I m interested in both. I re-read my question and to summarize it in a different way: I m interested in what people on this forum tend to do when
              Message 6 of 8 , Dec 6, 2006
              • 0 Attachment
                Hi Hakim,

                I'm interested in both.

                I re-read my question and to summarize it in a different way: I'm
                interested in what people on this forum tend to do when they don't
                have a proper budget for a/b testing sofware but need or want to do
                it anyway.

                thanks,

                -Benjamin


                --- In webanalytics@yahoogroups.com, HAKIMALY@... wrote:
                >
                > Are you looking for a tool to serve the ads or a tool to analyze
                the results in a statistically valid way? Hakim
                >
                >
                > -----Original Message-----
                > From: bradwarthan@...
                > To: webanalytics@yahoogroups.com
                > Sent: Mon, 4 Dec 2006 11:46 AM
                > Subject: RE: [webanalytics] A/B testing "by hand"
                >
                >
                > Ben, did you ever get a response back from this?
                >
                > Brad
                >
                > Brad Warthan
                > (812) 457-4479
                >
                > -----Original Message-----
                > From: webanalytics@yahoogroups.com
                [mailto:webanalytics@yahoogroups.com]On
                > Behalf Of Benjamin Rosenthal
                > Sent: Thursday, November 30, 2006 12:43 PM
                > To: webanalytics@yahoogroups.com
                > Subject: [webanalytics] A/B testing "by hand"
                >
                > Hi folks,
                >
                > I'd like to do some A/B testing of different elements on the home
                > page of our website. Essentially, I'd like to figure out if changing
                > different elements affects our bounce rate and/or conversion rate.
                >
                > I'm wondering...
                >
                > Without using any testing software specifically designed for this
                > sort of thing is there one agreed upon "gold standard" for doing A/B
                > testing manually? In other words, what steps would you use to test
                > this? I'm sure there are lots of different ways that people on the
                > board are doing this right now. I know I can easily change elements
                > and test during different time periods, but ideally, I'd like to
                test
                > 2 or 3 elements dynamically during the same time period.
                >
                > The only tools I have at my disposal are Google Analytics and
                > Coldfusion. I can modify all the code on the site, but don't have
                > easy access to the log files or the server application settings.
                >
                > (I'm also not opposed to hearing about any "free" packages out there
                > for A/B testing? I know that Google has Google Optimizer in Beta
                > testing right now.)
                >
                > Any suggestions, recipes, or links would be appreciated.
                >
                > thanks!
                >
                > -Benjamin
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >
                >
                ______________________________________________________________________
                __
                > Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-
                leading spam and email virus protection.
                >
                >
                > [Non-text portions of this message have been removed]
                >
              • HAKIMALY@NETSCAPE.NET
                Benjamin, I d take a look at Split Test Accelerator which does both. I have not used it my self, but from what I have read, it seems to be an easy-to-use tool.
                Message 7 of 8 , Dec 7, 2006
                • 0 Attachment
                  Benjamin, I'd take a look at Split Test Accelerator which does both. I have not used it my self, but from what I have read, it seems to be an easy-to-use tool. Contact Jim (whom I am copying). Hakim


                  -----Original Message-----
                  From: benjamin@...
                  To: webanalytics@yahoogroups.com
                  Sent: Wed, 6 Dec 2006 9:49 AM
                  Subject: [webanalytics] Re: A/B testing "by hand"


                  Hi Hakim,

                  I'm interested in both.

                  I re-read my question and to summarize it in a different way: I'm
                  interested in what people on this forum tend to do when they don't
                  have a proper budget for a/b testing sofware but need or want to do
                  it anyway.

                  thanks,

                  -Benjamin


                  --- In webanalytics@yahoogroups.com, HAKIMALY@... wrote:
                  >
                  > Are you looking for a tool to serve the ads or a tool to analyze
                  the results in a statistically valid way? Hakim
                  >
                  >
                  > -----Original Message-----
                  > From: bradwarthan@...
                  > To: webanalytics@yahoogroups.com
                  > Sent: Mon, 4 Dec 2006 11:46 AM
                  > Subject: RE: [webanalytics] A/B testing "by hand"
                  >
                  >
                  > Ben, did you ever get a response back from this?
                  >
                  > Brad
                  >
                  > Brad Warthan
                  > (812) 457-4479
                  >
                  > -----Original Message-----
                  > From: webanalytics@yahoogroups.com
                  [mailto:webanalytics@yahoogroups.com]On
                  > Behalf Of Benjamin Rosenthal
                  > Sent: Thursday, November 30, 2006 12:43 PM
                  > To: webanalytics@yahoogroups.com
                  > Subject: [webanalytics] A/B testing "by hand"
                  >
                  > Hi folks,
                  >
                  > I'd like to do some A/B testing of different elements on the home
                  > page of our website. Essentially, I'd like to figure out if changing
                  > different elements affects our bounce rate and/or conversion rate.
                  >
                  > I'm wondering...
                  >
                  > Without using any testing software specifically designed for this
                  > sort of thing is there one agreed upon "gold standard" for doing A/B
                  > testing manually? In other words, what steps would you use to test
                  > this? I'm sure there are lots of different ways that people on the
                  > board are doing this right now. I know I can easily change elements
                  > and test during different time periods, but ideally, I'd like to
                  test
                  > 2 or 3 elements dynamically during the same time period.
                  >
                  > The only tools I have at my disposal are Google Analytics and
                  > Coldfusion. I can modify all the code on the site, but don't have
                  > easy access to the log files or the server application settings.
                  >
                  > (I'm also not opposed to hearing about any "free" packages out there
                  > for A/B testing? I know that Google has Google Optimizer in Beta
                  > testing right now.)
                  >
                  > Any suggestions, recipes, or links would be appreciated.
                  >
                  > thanks!
                  >
                  > -Benjamin
                  >
                  > [Non-text portions of this message have been removed]
                  >
                  >
                  >
                  >
                  __________________________________________________________
                  __
                  > Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-
                  leading spam and email virus protection.
                  >
                  >
                  > [Non-text portions of this message have been removed]
                  >



                  ________________________________________________________________________
                  Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-leading spam and email virus protection.


                  [Non-text portions of this message have been removed]
                Your message has been successfully submitted and would be delivered to recipients shortly.