Loading ...
Sorry, an error occurred while loading the content.

8748Re: [NTO] robots.txt question

Expand Messages
  • Mike Breiding - Morgantown WV
    Mar 22, 2010
    • 0 Attachment
      Thanks, Steve.
      -Mike
      ==========================
      Stephen Riddle wrote:

      > <snips>
      > > > Mike wrote:
      > > > >Can robots.txt be used for multiple functions?
      > > > >
      > > > >Such this:
      > > > >
      > > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
      > > > <http://gsuttonbreid ing.net/sitemap. xml
      > <http://gsuttonbreiding.net/sitemap.xml>>
      > > > >User-Agent: *
      > > > >Disallow: /dir/
      > > >
      > > > Not sure what you mean with multiple functions. Do you mean lock out
      > > > different bots from different directories?
      > >
      > > No, I mean block spidering *and* point to the site map.
      >
      > <snips>
      >
      > A good place to find info on robots.txt is
      > http://www.robotstx t.org/robotstxt. html
      > <http://www.robotstxt.org/robotstxt.html>
      >
      > Basically, robots.txt is just for robots. Anyone can see it. Robots
      > don't have to honor your wishes, and do so only because they are "nice."
      > To make sure a spider knows where your site map is, make sure you have a
      > link back to it from each page on your site.
      >
      > Good luck, Stephen
    • Show all 8 messages in this topic