8745Re: [NTO] robots.txt question
- Mar 21, 2010loro wrote:
>No, I mean block spidering *and* point to the site map.
> Mike wrote:
> >Can robots.txt be used for multiple functions?
> >Such this:
> >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
> >User-Agent: *
> >Disallow: /dir/
> Not sure what you mean with multiple functions. Do you mean lock out
> different bots from different directories?
All the examples I have seen on the web are for spidering only.
>Like disallow bot1, bot2
> and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.
- << Previous post in topic Next post in topic >>