8746Re: [NTO] robots.txt question
- Mar 21, 2010Mike wrote:
>loro wrote:Oh,sorry! Yeah, sure.
> > Mike wrote:
> > >Can robots.txt be used for multiple functions?
> > >
> > >Such this:
> > >
> > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
> > <http://gsuttonbreiding.net/sitemap.xml>
> > >User-Agent: *
> > >Disallow: /dir/
> > Not sure what you mean with multiple functions. Do you mean lock out
> > different bots from different directories?
>No, I mean block spidering *and* point to the site map.
>All the examples I have seen on the web are for spidering only.
- << Previous post in topic Next post in topic >>