Loading ...
Sorry, an error occurred while loading the content.

A/B testing "by hand"

Expand Messages
  • Benjamin Rosenthal
    Hi folks, I d like to do some A/B testing of different elements on the home page of our website. Essentially, I d like to figure out if changing different
    Message 1 of 8 , Nov 30, 2006
    • 0 Attachment
      Hi folks,

      I'd like to do some A/B testing of different elements on the home
      page of our website. Essentially, I'd like to figure out if changing
      different elements affects our bounce rate and/or conversion rate.

      I'm wondering...

      Without using any testing software specifically designed for this
      sort of thing is there one agreed upon "gold standard" for doing A/B
      testing manually? In other words, what steps would you use to test
      this? I'm sure there are lots of different ways that people on the
      board are doing this right now. I know I can easily change elements
      and test during different time periods, but ideally, I'd like to test
      2 or 3 elements dynamically during the same time period.

      The only tools I have at my disposal are Google Analytics and
      Coldfusion. I can modify all the code on the site, but don't have
      easy access to the log files or the server application settings.

      (I'm also not opposed to hearing about any "free" packages out there
      for A/B testing? I know that Google has Google Optimizer in Beta
      testing right now.)

      Any suggestions, recipes, or links would be appreciated.

      thanks!

      -Benjamin
    • Melissa Cornwell
      If you can create a report that looks at cookie values then you can do it. Just make all the different variations a different cookie value and create reports
      Message 2 of 8 , Dec 1, 2006
      • 0 Attachment
        If you can create a report that looks at cookie values then you can do it.
        Just make all the different variations a different cookie value and create
        reports based on those values to get at the KPIs you care about.



        I did this a few times and it takes a bit to set up the reports and do the
        testing, but otherwise it works out great and doesn't cost you anything.



        Melissa



        _____

        From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com] On
        Behalf Of Benjamin Rosenthal
        Sent: Thursday, November 30, 2006 9:43 AM
        To: webanalytics@yahoogroups.com
        Subject: [webanalytics] A/B testing "by hand"



        Hi folks,

        I'd like to do some A/B testing of different elements on the home
        page of our website. Essentially, I'd like to figure out if changing
        different elements affects our bounce rate and/or conversion rate.

        I'm wondering...

        Without using any testing software specifically designed for this
        sort of thing is there one agreed upon "gold standard" for doing A/B
        testing manually? In other words, what steps would you use to test
        this? I'm sure there are lots of different ways that people on the
        board are doing this right now. I know I can easily change elements
        and test during different time periods, but ideally, I'd like to test
        2 or 3 elements dynamically during the same time period.

        The only tools I have at my disposal are Google Analytics and
        Coldfusion. I can modify all the code on the site, but don't have
        easy access to the log files or the server application settings.

        (I'm also not opposed to hearing about any "free" packages out there
        for A/B testing? I know that Google has Google Optimizer in Beta
        testing right now.)

        Any suggestions, recipes, or links would be appreciated.

        thanks!

        -Benjamin





        [Non-text portions of this message have been removed]
      • Brad Warthan
        Ben, did you ever get a response back from this? Brad Brad Warthan (812) 457-4479 ... From: webanalytics@yahoogroups.com
        Message 3 of 8 , Dec 4, 2006
        • 0 Attachment
          Ben, did you ever get a response back from this?

          Brad

          Brad Warthan
          (812) 457-4479

          -----Original Message-----
          From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]On
          Behalf Of Benjamin Rosenthal
          Sent: Thursday, November 30, 2006 12:43 PM
          To: webanalytics@yahoogroups.com
          Subject: [webanalytics] A/B testing "by hand"


          Hi folks,

          I'd like to do some A/B testing of different elements on the home
          page of our website. Essentially, I'd like to figure out if changing
          different elements affects our bounce rate and/or conversion rate.

          I'm wondering...

          Without using any testing software specifically designed for this
          sort of thing is there one agreed upon "gold standard" for doing A/B
          testing manually? In other words, what steps would you use to test
          this? I'm sure there are lots of different ways that people on the
          board are doing this right now. I know I can easily change elements
          and test during different time periods, but ideally, I'd like to test
          2 or 3 elements dynamically during the same time period.

          The only tools I have at my disposal are Google Analytics and
          Coldfusion. I can modify all the code on the site, but don't have
          easy access to the log files or the server application settings.

          (I'm also not opposed to hearing about any "free" packages out there
          for A/B testing? I know that Google has Google Optimizer in Beta
          testing right now.)

          Any suggestions, recipes, or links would be appreciated.

          thanks!

          -Benjamin






          [Non-text portions of this message have been removed]
        • HAKIMALY@NETSCAPE.NET
          Are you looking for a tool to serve the ads or a tool to analyze the results in a statistically valid way? Hakim ... From: bradwarthan@gmail.com To:
          Message 4 of 8 , Dec 5, 2006
          • 0 Attachment
            Are you looking for a tool to serve the ads or a tool to analyze the results in a statistically valid way? Hakim


            -----Original Message-----
            From: bradwarthan@...
            To: webanalytics@yahoogroups.com
            Sent: Mon, 4 Dec 2006 11:46 AM
            Subject: RE: [webanalytics] A/B testing "by hand"


            Ben, did you ever get a response back from this?

            Brad

            Brad Warthan
            (812) 457-4479

            -----Original Message-----
            From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com]On
            Behalf Of Benjamin Rosenthal
            Sent: Thursday, November 30, 2006 12:43 PM
            To: webanalytics@yahoogroups.com
            Subject: [webanalytics] A/B testing "by hand"

            Hi folks,

            I'd like to do some A/B testing of different elements on the home
            page of our website. Essentially, I'd like to figure out if changing
            different elements affects our bounce rate and/or conversion rate.

            I'm wondering...

            Without using any testing software specifically designed for this
            sort of thing is there one agreed upon "gold standard" for doing A/B
            testing manually? In other words, what steps would you use to test
            this? I'm sure there are lots of different ways that people on the
            board are doing this right now. I know I can easily change elements
            and test during different time periods, but ideally, I'd like to test
            2 or 3 elements dynamically during the same time period.

            The only tools I have at my disposal are Google Analytics and
            Coldfusion. I can modify all the code on the site, but don't have
            easy access to the log files or the server application settings.

            (I'm also not opposed to hearing about any "free" packages out there
            for A/B testing? I know that Google has Google Optimizer in Beta
            testing right now.)

            Any suggestions, recipes, or links would be appreciated.

            thanks!

            -Benjamin

            [Non-text portions of this message have been removed]



            ________________________________________________________________________
            Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-leading spam and email virus protection.


            [Non-text portions of this message have been removed]
          • jsbellsey
            Hi Benjamin, This should be pretty easy to set up. I ve written a custom suite in ASP, so doing it in CF would be similar: * create a random cookie generator,
            Message 5 of 8 , Dec 5, 2006
            • 0 Attachment
              Hi Benjamin,

              This should be pretty easy to set up. I've written a custom suite in
              ASP, so doing it in CF would be similar:

              * create a random cookie generator, up to N values for each test
              * track page views by calling a javascript function which loads a
              server-side page. The server page writes the page view into a
              database, with any associated information you need (or just the page
              version).
              * track conversions by calling a different javascript function,
              loading another server-side page. This just closes the loop, and might
              also write out custom info into the database.


              Analyzing the database is a creative process. I have a nice chart
              program installed that makes it easy for my marketing folks to
              interpret results. Here's what I present as a starting point:

              * Page views
              * Conversions (and conversion rate)
              * Avg order value (an extra param is passed on the conversion tracker)
              * Total $ if all visitors had been shown version N


              Writing custom A/B software is a total pleasure, but it's not a small
              job to do it well. Getting the info into the database is the easy
              part; getting it out is where things get interesting...

              Regards,
              Jeff Bellsey


              --- In webanalytics@yahoogroups.com, "Melissa Cornwell"
              <melissa.cornwell@...> wrote:
              >
              > If you can create a report that looks at cookie values then you can
              do it.
              > Just make all the different variations a different cookie value and
              create
              > reports based on those values to get at the KPIs you care about.
              >
              >
              > Melissa
              >
              >
              >
              > _____
              >
              > From: webanalytics@yahoogroups.com
              [mailto:webanalytics@yahoogroups.com] On
              > Behalf Of Benjamin Rosenthal
              > Sent: Thursday, November 30, 2006 9:43 AM
              > To: webanalytics@yahoogroups.com
              > Subject: [webanalytics] A/B testing "by hand"
              >
              >
              >
              > Hi folks,
              >
              > I'd like to do some A/B testing of different elements on the home
              > page of our website. Essentially, I'd like to figure out if changing
              > different elements affects our bounce rate and/or conversion rate.
              >
              > I'm wondering...
              >
              > Without using any testing software specifically designed for this
              > sort of thing is there one agreed upon "gold standard" for doing A/B
              > testing manually? In other words, what steps would you use to test
              > this? I'm sure there are lots of different ways that people on the
              > board are doing this right now. I know I can easily change elements
              > and test during different time periods, but ideally, I'd like to test
              > 2 or 3 elements dynamically during the same time period.
              >
              > The only tools I have at my disposal are Google Analytics and
              > Coldfusion. I can modify all the code on the site, but don't have
              > easy access to the log files or the server application settings.
              >
              > -Benjamin
              >
              >
              >
            • Sarah Brown
              Ben, I recommend the Fundamentals of Online Testing course offered via Marketing Experiments (http://www.marketingexperiments.com) ... which happens to
              Message 6 of 8 , Dec 6, 2006
              • 0 Attachment
                Ben,

                I recommend the "Fundamentals of Online Testing" course
                offered via Marketing Experiments
                (http://www.marketingexperiments.com) ... which happens
                to include a free widget for conducting A/B tests.

                Best,
                Sarah

                ________________________________

                From: webanalytics@yahoogroups.com
                [mailto:webanalytics@yahoogroups.com] On Behalf Of
                HAKIMALY@...
                Sent: Tuesday, December 05, 2006 12:27 PM
                To: webanalytics@yahoogroups.com
                Subject: Re: [webanalytics] A/B testing "by hand"



                Are you looking for a tool to serve the ads or a tool to
                analyze the results in a statistically valid way? Hakim


                -----Original Message-----
                From: bradwarthan@...
                <mailto:bradwarthan%40gmail.com>
                To: webanalytics@yahoogroups.com
                <mailto:webanalytics%40yahoogroups.com>
                Sent: Mon, 4 Dec 2006 11:46 AM
                Subject: RE: [webanalytics] A/B testing "by hand"

                Ben, did you ever get a response back from this?

                Brad

                Brad Warthan
                (812) 457-4479

                -----Original Message-----
                From: webanalytics@yahoogroups.com
                <mailto:webanalytics%40yahoogroups.com>
                [mailto:webanalytics@yahoogroups.com
                <mailto:webanalytics%40yahoogroups.com> ]On
                Behalf Of Benjamin Rosenthal
                Sent: Thursday, November 30, 2006 12:43 PM
                To: webanalytics@yahoogroups.com
                <mailto:webanalytics%40yahoogroups.com>
                Subject: [webanalytics] A/B testing "by hand"

                Hi folks,

                I'd like to do some A/B testing of different elements on
                the home
                page of our website. Essentially, I'd like to figure out
                if changing
                different elements affects our bounce rate and/or
                conversion rate.

                I'm wondering...

                Without using any testing software specifically designed
                for this
                sort of thing is there one agreed upon "gold standard"
                for doing A/B
                testing manually? In other words, what steps would you
                use to test
                this? I'm sure there are lots of different ways that
                people on the
                board are doing this right now. I know I can easily
                change elements
                and test during different time periods, but ideally, I'd
                like to test
                2 or 3 elements dynamically during the same time period.

                The only tools I have at my disposal are Google Analytics
                and
                Coldfusion. I can modify all the code on the site, but
                don't have
                easy access to the log files or the server application
                settings.

                (I'm also not opposed to hearing about any "free"
                packages out there
                for A/B testing? I know that Google has Google Optimizer
                in Beta
                testing right now.)

                Any suggestions, recipes, or links would be appreciated.

                thanks!

                -Benjamin

                [Non-text portions of this message have been removed]

                _________________________________________________________
                _
                Check Out the new free AIM(R) Mail -- 2 GB of storage and
                industry-leading spam and email virus protection.

                [Non-text portions of this message have been removed]






                [Non-text portions of this message have been removed]
              • Benjamin Rosenthal
                Hi Hakim, I m interested in both. I re-read my question and to summarize it in a different way: I m interested in what people on this forum tend to do when
                Message 7 of 8 , Dec 6, 2006
                • 0 Attachment
                  Hi Hakim,

                  I'm interested in both.

                  I re-read my question and to summarize it in a different way: I'm
                  interested in what people on this forum tend to do when they don't
                  have a proper budget for a/b testing sofware but need or want to do
                  it anyway.

                  thanks,

                  -Benjamin


                  --- In webanalytics@yahoogroups.com, HAKIMALY@... wrote:
                  >
                  > Are you looking for a tool to serve the ads or a tool to analyze
                  the results in a statistically valid way? Hakim
                  >
                  >
                  > -----Original Message-----
                  > From: bradwarthan@...
                  > To: webanalytics@yahoogroups.com
                  > Sent: Mon, 4 Dec 2006 11:46 AM
                  > Subject: RE: [webanalytics] A/B testing "by hand"
                  >
                  >
                  > Ben, did you ever get a response back from this?
                  >
                  > Brad
                  >
                  > Brad Warthan
                  > (812) 457-4479
                  >
                  > -----Original Message-----
                  > From: webanalytics@yahoogroups.com
                  [mailto:webanalytics@yahoogroups.com]On
                  > Behalf Of Benjamin Rosenthal
                  > Sent: Thursday, November 30, 2006 12:43 PM
                  > To: webanalytics@yahoogroups.com
                  > Subject: [webanalytics] A/B testing "by hand"
                  >
                  > Hi folks,
                  >
                  > I'd like to do some A/B testing of different elements on the home
                  > page of our website. Essentially, I'd like to figure out if changing
                  > different elements affects our bounce rate and/or conversion rate.
                  >
                  > I'm wondering...
                  >
                  > Without using any testing software specifically designed for this
                  > sort of thing is there one agreed upon "gold standard" for doing A/B
                  > testing manually? In other words, what steps would you use to test
                  > this? I'm sure there are lots of different ways that people on the
                  > board are doing this right now. I know I can easily change elements
                  > and test during different time periods, but ideally, I'd like to
                  test
                  > 2 or 3 elements dynamically during the same time period.
                  >
                  > The only tools I have at my disposal are Google Analytics and
                  > Coldfusion. I can modify all the code on the site, but don't have
                  > easy access to the log files or the server application settings.
                  >
                  > (I'm also not opposed to hearing about any "free" packages out there
                  > for A/B testing? I know that Google has Google Optimizer in Beta
                  > testing right now.)
                  >
                  > Any suggestions, recipes, or links would be appreciated.
                  >
                  > thanks!
                  >
                  > -Benjamin
                  >
                  > [Non-text portions of this message have been removed]
                  >
                  >
                  >
                  >
                  ______________________________________________________________________
                  __
                  > Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-
                  leading spam and email virus protection.
                  >
                  >
                  > [Non-text portions of this message have been removed]
                  >
                • HAKIMALY@NETSCAPE.NET
                  Benjamin, I d take a look at Split Test Accelerator which does both. I have not used it my self, but from what I have read, it seems to be an easy-to-use tool.
                  Message 8 of 8 , Dec 7, 2006
                  • 0 Attachment
                    Benjamin, I'd take a look at Split Test Accelerator which does both. I have not used it my self, but from what I have read, it seems to be an easy-to-use tool. Contact Jim (whom I am copying). Hakim


                    -----Original Message-----
                    From: benjamin@...
                    To: webanalytics@yahoogroups.com
                    Sent: Wed, 6 Dec 2006 9:49 AM
                    Subject: [webanalytics] Re: A/B testing "by hand"


                    Hi Hakim,

                    I'm interested in both.

                    I re-read my question and to summarize it in a different way: I'm
                    interested in what people on this forum tend to do when they don't
                    have a proper budget for a/b testing sofware but need or want to do
                    it anyway.

                    thanks,

                    -Benjamin


                    --- In webanalytics@yahoogroups.com, HAKIMALY@... wrote:
                    >
                    > Are you looking for a tool to serve the ads or a tool to analyze
                    the results in a statistically valid way? Hakim
                    >
                    >
                    > -----Original Message-----
                    > From: bradwarthan@...
                    > To: webanalytics@yahoogroups.com
                    > Sent: Mon, 4 Dec 2006 11:46 AM
                    > Subject: RE: [webanalytics] A/B testing "by hand"
                    >
                    >
                    > Ben, did you ever get a response back from this?
                    >
                    > Brad
                    >
                    > Brad Warthan
                    > (812) 457-4479
                    >
                    > -----Original Message-----
                    > From: webanalytics@yahoogroups.com
                    [mailto:webanalytics@yahoogroups.com]On
                    > Behalf Of Benjamin Rosenthal
                    > Sent: Thursday, November 30, 2006 12:43 PM
                    > To: webanalytics@yahoogroups.com
                    > Subject: [webanalytics] A/B testing "by hand"
                    >
                    > Hi folks,
                    >
                    > I'd like to do some A/B testing of different elements on the home
                    > page of our website. Essentially, I'd like to figure out if changing
                    > different elements affects our bounce rate and/or conversion rate.
                    >
                    > I'm wondering...
                    >
                    > Without using any testing software specifically designed for this
                    > sort of thing is there one agreed upon "gold standard" for doing A/B
                    > testing manually? In other words, what steps would you use to test
                    > this? I'm sure there are lots of different ways that people on the
                    > board are doing this right now. I know I can easily change elements
                    > and test during different time periods, but ideally, I'd like to
                    test
                    > 2 or 3 elements dynamically during the same time period.
                    >
                    > The only tools I have at my disposal are Google Analytics and
                    > Coldfusion. I can modify all the code on the site, but don't have
                    > easy access to the log files or the server application settings.
                    >
                    > (I'm also not opposed to hearing about any "free" packages out there
                    > for A/B testing? I know that Google has Google Optimizer in Beta
                    > testing right now.)
                    >
                    > Any suggestions, recipes, or links would be appreciated.
                    >
                    > thanks!
                    >
                    > -Benjamin
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >
                    >
                    __________________________________________________________
                    __
                    > Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-
                    leading spam and email virus protection.
                    >
                    >
                    > [Non-text portions of this message have been removed]
                    >



                    ________________________________________________________________________
                    Check Out the new free AIM(R) Mail -- 2 GB of storage and industry-leading spam and email virus protection.


                    [Non-text portions of this message have been removed]
                  Your message has been successfully submitted and would be delivered to recipients shortly.