Loading ...
Sorry, an error occurred while loading the content.

Re[2]: Search Bot

Expand Messages
  • c.hauser@itassistance.ch
    ... Set up Apache to respond to a If-Modified-Since somedate request with 304 response containing the whole page. To be honest I havent t set it up, as I
    Message 1 of 1 , Jan 28, 2004
      Shawn wrote:
      > Hi, I have been trying to figure out a way to limit the massive amount
      > of bandwidth that search bots (Googlebot/2.1) consume daily from my
      > website. My problem is that I am running Apache::ASP and about 90% of
      > the site is dynamic content, links such as product.htm?id=100. The
      > dynamic content gets changed quite a bit so I don’t want to use any
      > caching for regular users, but it would be fine for the bots to use a
      > cached copy for a month or so. The solution I came up with is manually
      > modifying the headers to keeping sending back 304 HTTP_NOT_MODIFIED for
      > a month before allowing new content to be served up to only search bots
      > and not to regular web browsers. Can anyone tell me if there are some
      > problems you for see with doing something like this? I have only tested
      > this on a dev server and was just wondering if anyone else had this
      > problem or any suggestions they might have.

      Set up Apache to respond to a 'If-Modified-Since somedate' request with 304
      response containing the whole page.

      To be honest I havent't set it up, as I don't know the way to do that in
      httpd.conf over all host as a default way to respind. The only way I did it is
      via the request header negotiation in mod_perl.

      Best Regards, Christian

      Reporting bugs: http://perl.apache.org/bugs/
      Mail list info: http://perl.apache.org/maillist/modperl.html
      List etiquette: http://perl.apache.org/maillist/email-etiquette.html
    Your message has been successfully submitted and would be delivered to recipients shortly.