Loading ...
Sorry, an error occurred while loading the content.

8746Re: [NTO] robots.txt question

Expand Messages
  • loro
    Mar 21, 2010
      Mike wrote:
      >loro wrote:
      > > Mike wrote:
      > > >Can robots.txt be used for multiple functions?
      > > >
      > > >Such this:
      > > >
      > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
      > > <http://gsuttonbreiding.net/sitemap.xml>
      > > >User-Agent: *
      > > >Disallow: /dir/
      > >
      > > Not sure what you mean with multiple functions. Do you mean lock out
      > > different bots from different directories?
      >
      >No, I mean block spidering *and* point to the site map.
      >All the examples I have seen on the web are for spidering only.
      >Thanks,
      >-Mike

      Oh,sorry! Yeah, sure.

      Lotta
    • Show all 8 messages in this topic