Loading ...
Sorry, an error occurred while loading the content.

Cost/Benefit model for best bets?

Expand Messages
  • Lee Romero
    Hi all - I m wondering if anyone might have considered how to establish a model that provides some insight on the cost and benefits of managing best bets
    Message 1 of 10 , Jan 25, 2010
    • 0 Attachment
      Hi all - I'm wondering if anyone might have considered how to
      establish a model that provides some insight on the cost and benefits
      of managing best bets manually in a search solution?

      Intuitively, you have to hope that the more search views in which a
      user finds a best bet the better overall success users will have
      (assuming you have good best bet targets, anyway !). On the other
      hand, the cost of identifying the terms, targets, etc., for best bets
      goes up the more you add.

      Added on top of that is that if you want to increase the percentage of
      searches showing a best bet, you will have to deal with the Zipf curve
      of searches - to move from 10% of searches showing a best bet to 20%
      will require much more than a doubling of best bets you configure.

      Any pointers to any articles or blog posts on this would be greatly appreciated.

      Regards
      Lee Romero
    • Louis Rosenfeld
      Hi Lee, great question; thanks for bringing this up! I ve not seen anything on developing a best bets cost benefit model. I know that some folks would argue
      Message 2 of 10 , Jan 26, 2010
      • 0 Attachment
        Hi Lee, great question; thanks for bringing this up!

        I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

        My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

        And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords.msu.edu/a-z/directory.asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

        The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

        I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizational implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders--ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

        Hope this helps!

        cheers



        Louis Rosenfeld :: http://louisrosenfeld.com :: @louisrosenfeld
        Rosenfeld Media :: http://rosenfeldmedia.com :: @rosenfeldmedia


        On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@...> wrote:
         

        Hi all - I'm wondering if anyone might have considered how to
        establish a model that provides some insight on the cost and benefits
        of managing best bets manually in a search solution?

        Intuitively, you have to hope that the more search views in which a
        user finds a best bet the better overall success users will have
        (assuming you have good best bet targets, anyway !). On the other
        hand, the cost of identifying the terms, targets, etc., for best bets
        goes up the more you add.

        Added on top of that is that if you want to increase the percentage of
        searches showing a best bet, you will have to deal with the Zipf curve
        of searches - to move from 10% of searches showing a best bet to 20%
        will require much more than a doubling of best bets you configure.

        Any pointers to any articles or blog posts on this would be greatly appreciated.

        Regards
        Lee Romero


      • Walter Underwood
        Good point, best bets are manual, and therefore expensive. I use best bets to solve a specific problem in search behavior and linguistics, that people use a
        Message 3 of 10 , Jan 26, 2010
        • 0 Attachment
          Good point, best bets are manual, and therefore expensive.

          I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.

          Examples:

          * "batman" when searching for "The Dark Knight"

          * "avp" when searching for "Alien vs. Predator"

          * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")

          It is really easy to mess up your results with excessive best bets. 

          You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.

          wunder

          On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:

           

          Hi Lee, great question; thanks for bringing this up!

          I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

          My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

          And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords. msu.edu/a- z/directory. asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

          The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

          I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizati onal implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders- -ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

          Hope this helps!

          cheers



          Louis Rosenfeld :: http://louisrosenfe ld.com :: @louisrosenfeld
          Rosenfeld Media :: http://rosenfeldmed ia.com :: @rosenfeldmedia


          On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@gmail. com> wrote:
           

          Hi all - I'm wondering if anyone might have considered how to
          establish a model that provides some insight on the cost and benefits
          of managing best bets manually in a search solution?

          Intuitively, you have to hope that the more search views in which a
          user finds a best bet the better overall success users will have
          (assuming you have good best bet targets, anyway !). On the other
          hand, the cost of identifying the terms, targets, etc., for best bets
          goes up the more you add.

          Added on top of that is that if you want to increase the percentage of
          searches showing a best bet, you will have to deal with the Zipf curve
          of searches - to move from 10% of searches showing a best bet to 20%
          will require much more than a doubling of best bets you configure.

          Any pointers to any articles or blog posts on this would be greatly appreciated.

          Regards
          Lee Romero




        • Richard Wiggins
          Walter, who is wickedly smart and whose words I normally trust with impunity, is wrong. Best Bets are NOT expensive. The Zipf curve proves that a very small
          Message 4 of 10 , Jan 26, 2010
          • 0 Attachment
            Walter, who is wickedly smart and whose words I normally trust with impunity, is wrong.
             
            Best Bets are NOT expensive.  The Zipf curve proves that a very small number of Best Bets covers a huge fraction of searches performed. 
             
            A Best Bets index of 500 or 1000 search terms may cover one half of searches performed.  If you build this corpus over say a year, the cost is extremely low.
             
            Now, if you are Netflix, with constantly varying content, perhaps the cost is a tad higher.  But even in that case I claim Wunder is wrong.  The cost of maintaining, say, the top 100 movies of current interest is minuscule.  Surely it is no large investment to have one Flo at the front door helping the masses find what they seek each day.
             
            About 1/2 of all site visitors to a typical site will use the Search box.  Therefore, Search is 1/2 of your IA.  Therefore, Search is worth a bit of manual labor.  And a small amount of manual labor reaps huge benefits. 
             
            How much time do you spend building your browsing nav?    If one-half of site visitors use the search box, spend 1/2 of your energy making search perform well. 
             
            Search is too important to trust to robots.
             
            This is not even a close case.  If you don't understand the value of Best Bets, you don't understand the ZIpf curve.
             
            /rich

            On Tue, Jan 26, 2010 at 11:41 AM, Walter Underwood <wunder@...> wrote:
             

            Good point, best bets are manual, and therefore expensive.


            I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.

            Examples:

            * "batman" when searching for "The Dark Knight"

            * "avp" when searching for "Alien vs. Predator"

            * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")

            It is really easy to mess up your results with excessive best bets. 

            You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.

            wunder

            On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:

             

            Hi Lee, great question; thanks for bringing this up!

            I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

            My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

            And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords.msu.edu/a-z/directory.asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

            The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

            I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizational implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders--ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

            Hope this helps!

            cheers



            Louis Rosenfeld :: http://louisrosenfeld.com :: @louisrosenfeld
            Rosenfeld Media :: http://rosenfeldmedia.com :: @rosenfeldmedia


            On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@...> wrote:
             

            Hi all - I'm wondering if anyone might have considered how to
            establish a model that provides some insight on the cost and benefits
            of managing best bets manually in a search solution?

            Intuitively, you have to hope that the more search views in which a
            user finds a best bet the better overall success users will have
            (assuming you have good best bet targets, anyway !). On the other
            hand, the cost of identifying the terms, targets, etc., for best bets
            goes up the more you add.

            Added on top of that is that if you want to increase the percentage of
            searches showing a best bet, you will have to deal with the Zipf curve
            of searches - to move from 10% of searches showing a best bet to 20%
            will require much more than a doubling of best bets you configure.

            Any pointers to any articles or blog posts on this would be greatly appreciated.

            Regards
            Lee Romero





          • Jordan Cassel
            Richard, Are you suggesting that Best Bets (with the notation that they are best bet results) should be created for all of the most popular X number of
            Message 5 of 10 , Jan 26, 2010
            • 0 Attachment
              Richard,

              Are you suggesting that Best Bets (with the notation that they are best bet results) should be created for all of the most popular X number of searches even if the organic search results are delivering desired matches in most cases?  Or, just the subset of those most popular searches where desired matches are not showing in the top 1 or 2 or 3 positions?

              Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc.

              Thanks,

              Jordan



              From: Richard Wiggins <richard.wiggins@...>
              To: SearchCoP@yahoogroups.com
              Sent: Tue, January 26, 2010 2:40:24 PM
              Subject: Re: [SearchCoP] Cost/Benefit model for best bets?

               

              Walter, who is wickedly smart and whose words I normally trust with impunity, is wrong.
               
              Best Bets are NOT expensive.  The Zipf curve proves that a very small number of Best Bets covers a huge fraction of searches performed. 
               
              A Best Bets index of 500 or 1000 search terms may cover one half of searches performed.  If you build this corpus over say a year, the cost is extremely low.
               
              Now, if you are Netflix, with constantly varying content, perhaps the cost is a tad higher.  But even in that case I claim Wunder is wrong.  The cost of maintaining, say, the top 100 movies of current interest is minuscule.  Surely it is no large investment to have one Flo at the front door helping the masses find what they seek each day.
               
              About 1/2 of all site visitors to a typical site will use the Search box.  Therefore, Search is 1/2 of your IA.  Therefore, Search is worth a bit of manual labor.  And a small amount of manual labor reaps huge benefits. 
               
              How much time do you spend building your browsing nav?    If one-half of site visitors use the search box, spend 1/2 of your energy making search perform well. 
               
              Search is too important to trust to robots.
               
              This is not even a close case.  If you don't understand the value of Best Bets, you don't understand the ZIpf curve.
               
              /rich

              On Tue, Jan 26, 2010 at 11:41 AM, Walter Underwood <wunder@wunderwood. org> wrote:
               

              Good point, best bets are manual, and therefore expensive.


              I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.

              Examples:

              * "batman" when searching for "The Dark Knight"

              * "avp" when searching for "Alien vs. Predator"

              * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")

              It is really easy to mess up your results with excessive best bets. 

              You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.

              wunder

              On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:

               

              Hi Lee, great question; thanks for bringing this up!

              I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

              My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

              And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords. msu.edu/a- z/directory. asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

              The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

              I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizati onal implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders- -ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

              Hope this helps!

              cheers



              Louis Rosenfeld :: http://louisrosenfeld.com/ :: @louisrosenfeld
              Rosenfeld Media :: http://rosenfeldmedia.com/ :: @rosenfeldmedia


              On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@gmail. com> wrote:
               

              Hi all - I'm wondering if anyone might have considered how to
              establish a model that provides some insight on the cost and benefits
              of managing best bets manually in a search solution?

              Intuitively, you have to hope that the more search views in which a
              user finds a best bet the better overall success users will have
              (assuming you have good best bet targets, anyway !). On the other
              hand, the cost of identifying the terms, targets, etc., for best bets
              goes up the more you add.

              Added on top of that is that if you want to increase the percentage of
              searches showing a best bet, you will have to deal with the Zipf curve
              of searches - to move from 10% of searches showing a best bet to 20%
              will require much more than a doubling of best bets you configure.

              Any pointers to any articles or blog posts on this would be greatly appreciated.

              Regards
              Lee Romero






            • Matt Moore
              Jordan - Enjoying this discussion though I recently was unsuccessful in getting best bets added to our site search experience with the argument that
              Message 6 of 10 , Jan 26, 2010
              • 0 Attachment
                Jordan - "Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc."

                This strikes me as very short-sighted approach (so you have my sympathies) - and one that demonstrates contempt for the user.

                One important issue is the stability of the "best bet" Zipf curve. Within organizations, certain areas of the intranet are more heavily trafficked than others (e.g. HR forms, expense claims, etc). This content is pretty stable. For these, it makes absolute sense to make these as finadable as quickly as possible. If the search engine cannot do this automatically then a best bet fix makes a heap of sense.

                On the other hand, if the content of your site is highly dynamic (e.g. a public news site), then key searched for items may be extremely volatile. And "best bets" do not make sense here.

                I suspect no one has done a full cost/benefit model (although I am willing to be surprised). Instead a point on the Zipf curve is arbitarily taken (top 10/50/100/500)for management purposes and I suspect that this is dependent on the headcount resource within an organization devoted to search maintenance/optimization - i.e. what can be shoe-horned into someone's role.



                From: Jordan Cassel <jordan_cassel@...>
                To: SearchCoP@yahoogroups.com
                Sent: Wed, January 27, 2010 11:34:10 AM
                Subject: Re: [SearchCoP] Cost/Benefit model for best bets?

                 

                Richard,

                Are you suggesting that Best Bets (with the notation that they are best bet results) should be created for all of the most popular X number of searches even if the organic search results are delivering desired matches in most cases?  Or, just the subset of those most popular searches where desired matches are not showing in the top 1 or 2 or 3 positions?

                Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything

                which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc.

                Thanks,

                Jordan



                From: Richard Wiggins <richard.wiggins@ gmail.com>
                To: SearchCoP@yahoogrou ps.com
                Sent: Tue, January 26, 2010 2:40:24 PM
                Subject: Re: [SearchCoP] Cost/Benefit model for best bets?

                 

                Walter, who is wickedly smart and whose words I normally trust with impunity, is wrong.
                 
                Best Bets are NOT expensive.  The Zipf curve proves that a very small number of Best Bets covers a huge fraction of searches performed. 
                 
                A Best Bets index of 500 or 1000 search terms may cover one half of searches performed.  If you build this corpus over say a year, the cost is extremely low.
                 
                Now, if you are Netflix, with constantly varying content, perhaps the cost is a tad higher.  But even in that case I claim Wunder is wrong.  The cost of maintaining, say, the top 100 movies of current interest is minuscule.  Surely it is no large investment to have one Flo at the front door helping the masses find what they seek each day.
                 
                About 1/2 of all site visitors to a typical site will use the Search box.  Therefore, Search is 1/2 of your IA.  Therefore, Search is worth a bit of manual labor.  And a small amount of manual labor reaps huge benefits. 
                 
                How much time do you spend building your browsing nav?    If one-half of site visitors use the search box, spend 1/2 of your energy making search perform well. 
                 
                Search is too important to trust to robots.
                 
                This is not even a close case.  If you don't understand the value of Best Bets, you don't understand the ZIpf curve.
                 
                /rich

                On Tue, Jan 26, 2010 at 11:41 AM, Walter Underwood <wunder@wunderwood. org> wrote:
                 

                Good point, best bets are manual, and therefore expensive.


                I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.

                Examples:

                * "batman" when searching for "The Dark Knight"

                * "avp" when searching for "Alien vs. Predator"

                * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")

                It is really easy to mess up your results with excessive best bets. 

                You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.

                wunder

                On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:

                 

                Hi Lee, great question; thanks for bringing this up!

                I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

                My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

                And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords. msu.edu/a- z/directory. asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

                The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

                I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizati onal implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders- -ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

                Hope this helps!

                cheers



                Louis Rosenfeld :: http://louisrosenfe ld.com/ :: @louisrosenfeld
                Rosenfeld Media :: http://rosenfeldmed ia.com/ :: @rosenfeldmedia


                On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@gmail. com> wrote:
                 

                Hi all - I'm wondering if anyone might have considered how to
                establish a model that provides some insight on the cost and benefits
                of managing best bets manually in a search solution?

                Intuitively, you have to hope that the more search views in which a
                user finds a best bet the better overall success users will have
                (assuming you have good best bet targets, anyway !). On the other
                hand, the cost of identifying the terms, targets, etc., for best bets
                goes up the more you add.

                Added on top of that is that if you want to increase the percentage of
                searches showing a best bet, you will have to deal with the Zipf curve
                of searches - to move from 10% of searches showing a best bet to 20%
                will require much more than a doubling of best bets you configure.

                Any pointers to any articles or blog posts on this would be greatly appreciated.

                Regards
                Lee Romero







              • Walter Underwood
                No tool is perfect. Best Bets have an obvious strength, direct control, but weaknesses, too. Here are some weaknesses: 1. They fix one query at a time. You ll
                Message 7 of 10 , Jan 26, 2010
                • 0 Attachment
                  No tool is perfect. Best Bets have an obvious strength, direct control, but weaknesses, too. Here are some weaknesses:

                  1. They fix one query at a time. You'll never fix 10,000 queries with even a few Best Bets, but you can do that with an algorithm change, like properly tuned fuzzy search. You'd go insane trying to fix misspellings with Best Bets.

                  2. They ignore the long tail. Best Bets (properly) focus on popular queries. Fuzzy search will help misspellings of "Coraline" (popular) but also "Le Samourai" (way, way down the tail).

                  3. They get stale. Constant vigilance is required to weed out dead links and find new, better Best Bets. This bit us at least once at Netflix, and we had a fairly small number of Best Bets.

                  4. They are often added to satisfy political requests. Best Bets are absolute power over ranking, and you know what they say about absolute power. On the other hand, sometimes you need to make your manager happy, even if it doesn't help users.

                  I tried searching for "parking" at Michigan State. Check out these results:


                  The first hit from Google (Bing and Yahoo, too) is a very informative page about visitor parking that also has a link to info on football parking. That page is not in the MSU Best Bets (probably staleness). Also, the first MSU Best Bet is "MSU Bike Project", which doesn't even talk about bike parking (either a political entry or the page changed since it was added, another kind of staleness).

                  In general, if 50% of search requests are being satisfied with any single tool or algorithm, that tool is probably way overused and being used where other tools are more appropriate.

                  wunder

                  On Jan 26, 2010, at 5:13 PM, Matt Moore wrote:

                   

                  Jordan - "Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc."

                  This strikes me as very short-sighted approach (so you have my sympathies) - and one that demonstrates contempt for the user.

                  One important issue is the stability of the "best bet" Zipf curve. Within organizations, certain areas of the intranet are more heavily trafficked than others (e.g. HR forms, expense claims, etc). This content is pretty stable. For these, it makes absolute sense to make these as finadable as quickly as possible. If the search engine cannot do this automatically then a best bet fix makes a heap of sense.

                  On the other hand, if the content of your site is highly dynamic (e.g. a public news site), then key searched for items may be extremely volatile. And "best bets" do not make sense here.

                  I suspect no one has done a full cost/benefit model (although I am willing to be surprised). Instead a point on the Zipf curve is arbitarily taken (top 10/50/100/500) for management purposes and I suspect that this is dependent on the headcount resource within an organization devoted to search maintenance/ optimization - i.e. what can be shoe-horned into someone's role.



                  From: Jordan Cassel <jordan_cassel@yahoo.com>
                  To: SearchCoP@yahoogroups.com
                  Sent: Wed, January 27, 2010 11:34:10 AM
                  Subject: Re: [SearchCoP] Cost/Benefit model for best bets?

                   

                  Richard,

                  Are you suggesting that Best Bets (with the notation that they are best bet results) should be created for all of the most popular X number of searches even if the organic search results are delivering desired matches in most cases?  Or, just the subset of those most popular searches where desired matches are not showing in the top 1 or 2 or 3 positions?

                  Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything

                  which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc.

                  Thanks,

                  Jordan



                  From: Richard Wiggins <richard.wiggins@ gmail.com>
                  To: SearchCoP@yahoogrou ps.com
                  Sent: Tue, January 26, 2010 2:40:24 PM
                  Subject: Re: [SearchCoP] Cost/Benefit model for best bets?

                   

                  Walter, who is wickedly smart and whose words I normally trust with impunity, is wrong.
                   
                  Best Bets are NOT expensive.  The Zipf curve proves that a very small number of Best Bets covers a huge fraction of searches performed. 
                   
                  A Best Bets index of 500 or 1000 search terms may cover one half of searches performed.  If you build this corpus over say a year, the cost is extremely low.
                   
                  Now, if you are Netflix, with constantly varying content, perhaps the cost is a tad higher.  But even in that case I claim Wunder is wrong.  The cost of maintaining, say, the top 100 movies of current interest is minuscule.  Surely it is no large investment to have one Flo at the front door helping the masses find what they seek each day.
                   
                  About 1/2 of all site visitors to a typical site will use the Search box.  Therefore, Search is 1/2 of your IA.  Therefore, Search is worth a bit of manual labor.  And a small amount of manual labor reaps huge benefits. 
                   
                  How much time do you spend building your browsing nav?    If one-half of site visitors use the search box, spend 1/2 of your energy making search perform well. 
                   
                  Search is too important to trust to robots.
                   
                  This is not even a close case.  If you don't understand the value of Best Bets, you don't understand the ZIpf curve.
                   
                  /rich

                  On Tue, Jan 26, 2010 at 11:41 AM, Walter Underwood <wunder@wunderwood. org> wrote:
                   

                  Good point, best bets are manual, and therefore expensive.


                  I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.

                  Examples:

                  * "batman" when searching for "The Dark Knight"

                  * "avp" when searching for "Alien vs. Predator"

                  * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")

                  It is really easy to mess up your results with excessive best bets. 

                  You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.

                  wunder

                  On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:

                   

                  Hi Lee, great question; thanks for bringing this up!

                  I've not seen anything on developing a best bets cost benefit model.  I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).

                  My feeling:  you'd never want to increase the percentage of searches showing a best bet as a goal on its own.  Your goal is improving the search experience.  So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job.  Top ten, twenty, or so; beyond that, there will be diminishing returns.

                  And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value.  For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords. msu.edu/a- z/directory. asp ).  Check it out--each index entry is a best bet search result, simply listed alphabetically. 

                  The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).

                  I'm sure there are some other great ways to leverage an investment in best bets.  Another that comes to mind is the cultural/organizati onal implications of creating new high-value real estate (this time, within search results).  Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support).  Whose goes first?  Does a political shitstorm break out?  If so, good.  One of the best ways to engage content owners and other stakeholders- -ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this.  Better to engage with them, even if negatively, then have a stasis of ignorance and abstention.  Short term pain leads to long term gain. 

                  Hope this helps!

                  cheers



                  Louis Rosenfeld :: http://louisrosenfe ld.com/ :: @louisrosenfeld
                  Rosenfeld Media :: http://rosenfeldmed ia.com/ :: @rosenfeldmedia


                  On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@gmail. com> wrote:
                   

                  Hi all - I'm wondering if anyone might have considered how to
                  establish a model that provides some insight on the cost and benefits
                  of managing best bets manually in a search solution?

                  Intuitively, you have to hope that the more search views in which a
                  user finds a best bet the better overall success users will have
                  (assuming you have good best bet targets, anyway !). On the other
                  hand, the cost of identifying the terms, targets, etc., for best bets
                  goes up the more you add.

                  Added on top of that is that if you want to increase the percentage of
                  searches showing a best bet, you will have to deal with the Zipf curve
                  of searches - to move from 10% of searches showing a best bet to 20%
                  will require much more than a doubling of best bets you configure.

                  Any pointers to any articles or blog posts on this would be greatly appreciated.

                  Regards
                  Lee Romero










                • Louis Rosenfeld
                  ... Argh. You have my deepest sympathies. I imagine that, like me, you re looking forward to a New Day, when those who make these decisions understand that
                  Message 8 of 10 , Jan 26, 2010
                  • 0 Attachment
                    On Tue, Jan 26, 2010 at 7:34 PM, Jordan Cassel <jordan_cassel@...> wrote:
                    Enjoying this discussion though I recently was unsuccessful in getting 'best bets' added to our site search experience with the argument that anything which requires this manual step of maintaining a database of best bet matches is too laborious, inefficient, etc.

                    Argh.  You have my deepest sympathies.

                    I imagine that, like me, you're looking forward to a New Day, when those who make these decisions understand that the best solutions are usually well-conceived hybrids of various manual and automated solutions, not one or the other.


                    Louis Rosenfeld :: http://louisrosenfeld.com :: @louisrosenfeld
                    Rosenfeld Media :: http://rosenfeldmedia.com :: @rosenfeldmedia


                  • tbwendt
                    Best Bets are essential for enterprise search. The movies title scenario is a different animal and not analogous to the corporate intranet. When you are
                    Message 9 of 10 , Jan 26, 2010
                    • 0 Attachment
                      Best Bets are essential for enterprise search. The movies title scenario is a different animal and not analogous to the corporate intranet. When you are talking about searching research papers, policies, news, product information, press releases, procedures, employee resources, web content, 3rd party content, file shares, and SharePoint then Best Bets are crucial to good search and happy employees.

                      The long tail graphs show us that the vast amount of searches are for a small amount of content. These are the navigators of the company. Employees want to find commonly used resources quickly.

                      Best Bets do not cost, they save in search productivity. Studies show that knowledge workers spend 15-40% of their week searching. Best Bets service a huge part of that percentage.

                      We let our users enter the Best Bets. We have a web application where users can enter the meta data about the Best Bet. There is some governance around this database. Best Bets have an expiration date. The owner is emailed before their link is expired.

                      As a search analyst, I study the metrics on our top searches at the beginning of each month. If something shows up that is not a Best Bets I can find who should be the content owner and walk them through the process of creating a Best Bet.

                      You could try to communicate to management that most high value, corporate content has insufficient content for a search engine accurately rank. High-value content (targets of Best Bets) often have lousy meta data. For example, a login screen to an important web application usually has little or no meta data. How can a search engine rank what it doesn't have? Especially in a corpus of millions of documents?

                      It is cheaper and more realistic to create a Best Bet than pay developers or content owners to create meta data on existing web pages and within existing content.

                      Anecdote, in December we had a lot of bad weather here in the Midwest. Looking at the search metrics, we had a couple thousand searches on things like "plant closing" and "snow policy." The results were terrible. There were no links to an up-to-date snow policy. There were probably hundreds of employees that cursed our search engine that day. Looking at the logs, you could see employees struggling to enter a search that would return the policy they sought. Turns out that the actual policy had no mention of snow!

                      The point is that no search engine is going to fix that "snow policy" search without human intervention. Best Bets allow you to fix those issues at a very low cost. It cost me about a 5 minute phone call to the guy in charge of the "shift interruption" policy. That 5 minute Best Bet is going to save thousands of $ in lost productivity during the next blizzard. Think of the amount of time wasted by several hundred employees fumbling with the search engine.

                      Tim

                      --- In SearchCoP@yahoogroups.com, Walter Underwood <wunder@...> wrote:
                      >
                      > Good point, best bets are manual, and therefore expensive.
                      >
                      > I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.
                      >
                      > Examples:
                      >
                      > * "batman" when searching for "The Dark Knight"
                      >
                      > * "avp" when searching for "Alien vs. Predator"
                      >
                      > * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")
                      >
                      > It is really easy to mess up your results with excessive best bets.
                      >
                      > You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.
                      >
                      > wunder
                      >
                      > On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:
                      >
                      > > Hi Lee, great question; thanks for bringing this up!
                      > >
                      > > I've not seen anything on developing a best bets cost benefit model. I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).
                      > >
                      > > My feeling: you'd never want to increase the percentage of searches showing a best bet as a goal on its own. Your goal is improving the search experience. So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job. Top ten, twenty, or so; beyond that, there will be diminishing returns.
                      > >
                      > > And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value. For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords.msu.edu/a-z/directory.asp ). Check it out--each index entry is a best bet search result, simply listed alphabetically.
                      > >
                      > > The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).
                      > >
                      > > I'm sure there are some other great ways to leverage an investment in best bets. Another that comes to mind is the cultural/organizational implications of creating new high-value real estate (this time, within search results). Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support). Whose goes first? Does a political shitstorm break out? If so, good. One of the best ways to engage content owners and other stakeholders--ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this. Better to engage with them, even if negatively, then have a stasis of ignorance and abstention. Short term pain leads to long term gain.
                      > >
                      > > Hope this helps!
                      > >
                      > > cheers
                      > >
                      > >
                      > >
                      > > Louis Rosenfeld :: http://louisrosenfeld.com :: @louisrosenfeld
                      > > Rosenfeld Media :: http://rosenfeldmedia.com :: @rosenfeldmedia
                      > >
                      > >
                      > >
                      > > On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@...> wrote:
                      > >
                      > > Hi all - I'm wondering if anyone might have considered how to
                      > > establish a model that provides some insight on the cost and benefits
                      > > of managing best bets manually in a search solution?
                      > >
                      > > Intuitively, you have to hope that the more search views in which a
                      > > user finds a best bet the better overall success users will have
                      > > (assuming you have good best bet targets, anyway !). On the other
                      > > hand, the cost of identifying the terms, targets, etc., for best bets
                      > > goes up the more you add.
                      > >
                      > > Added on top of that is that if you want to increase the percentage of
                      > > searches showing a best bet, you will have to deal with the Zipf curve
                      > > of searches - to move from 10% of searches showing a best bet to 20%
                      > > will require much more than a doubling of best bets you configure.
                      > >
                      > > Any pointers to any articles or blog posts on this would be greatly appreciated.
                      > >
                      > > Regards
                      > > Lee Romero
                      > >
                      > >
                      > >
                      >
                    • Walter Underwood
                      Just beware of always jumping to Best Bets. That only solves it for one page. That s probably the right answer for snow policy , but not for everything. Your
                      Message 10 of 10 , Jan 27, 2010
                      • 0 Attachment
                        Just beware of always jumping to Best Bets. That only solves it for one page. That's probably the right answer for "snow policy", but not for everything.

                        Your top intranet searches probably include lots of variants for vacation. What about synonyms?

                        FTO,PTO,vacation,holiday,time off,sick day,timecard,time card

                        That fixes every HR page or e-mail that talks about the subject. Test it first, of course.

                        wunder

                        On Jan 26, 2010, at 8:48 PM, tbwendt wrote:

                         

                        Best Bets are essential for enterprise search. The movies title scenario is a different animal and not analogous to the corporate intranet. When you are talking about searching research papers, policies, news, product information, press releases, procedures, employee resources, web content, 3rd party content, file shares, and SharePoint then Best Bets are crucial to good search and happy employees.

                        The long tail graphs show us that the vast amount of searches are for a small amount of content. These are the navigators of the company. Employees want to find commonly used resources quickly.

                        Best Bets do not cost, they save in search productivity. Studies show that knowledge workers spend 15-40% of their week searching. Best Bets service a huge part of that percentage.

                        We let our users enter the Best Bets. We have a web application where users can enter the meta data about the Best Bet. There is some governance around this database. Best Bets have an expiration date. The owner is emailed before their link is expired.

                        As a search analyst, I study the metrics on our top searches at the beginning of each month. If something shows up that is not a Best Bets I can find who should be the content owner and walk them through the process of creating a Best Bet.

                        You could try to communicate to management that most high value, corporate content has insufficient content for a search engine accurately rank. High-value content (targets of Best Bets) often have lousy meta data. For example, a login screen to an important web application usually has little or no meta data. How can a search engine rank what it doesn't have? Especially in a corpus of millions of documents?

                        It is cheaper and more realistic to create a Best Bet than pay developers or content owners to create meta data on existing web pages and within existing content.

                        Anecdote, in December we had a lot of bad weather here in the Midwest. Looking at the search metrics, we had a couple thousand searches on things like "plant closing" and "snow policy." The results were terrible. There were no links to an up-to-date snow policy. There were probably hundreds of employees that cursed our search engine that day. Looking at the logs, you could see employees struggling to enter a search that would return the policy they sought. Turns out that the actual policy had no mention of snow!

                        The point is that no search engine is going to fix that "snow policy" search without human intervention. Best Bets allow you to fix those issues at a very low cost. It cost me about a 5 minute phone call to the guy in charge of the "shift interruption" policy. That 5 minute Best Bet is going to save thousands of $ in lost productivity during the next blizzard. Think of the amount of time wasted by several hundred employees fumbling with the search engine.

                        Tim

                        --- In SearchCoP@yahoogrou ps.com, Walter Underwood <wunder@...> wrote:
                        >
                        > Good point, best bets are manual, and therefore expensive.
                        >
                        > I use best bets to solve a specific problem in search behavior and linguistics, that people use a search query that is not prominent in the best match document, maybe not present at all.
                        >
                        > Examples:
                        >
                        > * "batman" when searching for "The Dark Knight"
                        >
                        > * "avp" when searching for "Alien vs. Predator"
                        >
                        > * "14 up" when searching for "7 Plus Seven" (movie between "7 Up" and "21 Up")
                        >
                        > It is really easy to mess up your results with excessive best bets.
                        >
                        > You want to estimate benefit for any change. Estimate the number of queries per day affected by an improvement, and you will have made that many visitors happier. Take a look at the time and effort to implement, then choose.
                        >
                        > wunder
                        >
                        > On Jan 26, 2010, at 7:55 AM, Louis Rosenfeld wrote:
                        >
                        > > Hi Lee, great question; thanks for bringing this up!
                        > >
                        > > I've not seen anything on developing a best bets cost benefit model. I know that some folks would argue that best bets are a bad idea in general, as your search engine should be smart enough to address the same types of searcher needs (and I would disagree).
                        > >
                        > > My feeling: you'd never want to increase the percentage of searches showing a best bet as a goal on its own. Your goal is improving the search experience. So you'd want to develop best bets FOR the queries at the very top of the Zipf Curve WHEN your search engine was doing a crappy job. Top ten, twenty, or so; beyond that, there will be diminishing returns.
                        > >
                        > > And if you're going to invest in best bets, you should leverage that investment in as many ways as you can that provide value. For example, Rich Wiggins and friends at Michigan State University repurposed their many best bet results (which they already manage using a database) as an A-Z index (see http://keywords. msu.edu/a- z/directory. asp ). Check it out--each index entry is a best bet search result, simply listed alphabetically.
                        > >
                        > > The benefits here are that 1) the editorial effort is reused (as already stated); 2) you can actually provide an A-Z index that has a reasonably good scope (much easier than doing so from scratch).
                        > >
                        > > I'm sure there are some other great ways to leverage an investment in best bets. Another that comes to mind is the cultural/organizati onal implications of creating new high-value real estate (this time, within search results). Let's say a popular query could support multiple best bets (one from marketing, another from sales, another from support). Whose goes first? Does a political shitstorm break out? If so, good. One of the best ways to engage content owners and other stakeholders- -ones that might otherwise be tuned out--is just through unpleasant political imbroglios like this. Better to engage with them, even if negatively, then have a stasis of ignorance and abstention. Short term pain leads to long term gain.
                        > >
                        > > Hope this helps!
                        > >
                        > > cheers
                        > >
                        > >
                        > >
                        > > Louis Rosenfeld :: http://louisrosenfe ld.com :: @louisrosenfeld
                        > > Rosenfeld Media :: http://rosenfeldmed ia.com :: @rosenfeldmedia
                        > >
                        > >
                        > >
                        > > On Mon, Jan 25, 2010 at 9:09 AM, Lee Romero <pekadad@... > wrote:
                        > >
                        > > Hi all - I'm wondering if anyone might have considered how to
                        > > establish a model that provides some insight on the cost and benefits
                        > > of managing best bets manually in a search solution?
                        > >
                        > > Intuitively, you have to hope that the more search views in which a
                        > > user finds a best bet the better overall success users will have
                        > > (assuming you have good best bet targets, anyway !). On the other
                        > > hand, the cost of identifying the terms, targets, etc., for best bets
                        > > goes up the more you add.
                        > >
                        > > Added on top of that is that if you want to increase the percentage of
                        > > searches showing a best bet, you will have to deal with the Zipf curve
                        > > of searches - to move from 10% of searches showing a best bet to 20%
                        > > will require much more than a doubling of best bets you configure.
                        > >
                        > > Any pointers to any articles or blog posts on this would be greatly appreciated.
                        > >
                        > > Regards
                        > > Lee Romero
                        > >
                        > >
                        > >
                        >


                      Your message has been successfully submitted and would be delivered to recipients shortly.