8748Re: [NTO] robots.txt question
- Mar 22, 2010Thanks, Steve.
Stephen Riddle wrote:
> > > Mike wrote:
> > > >Can robots.txt be used for multiple functions?
> > > >
> > > >Such this:
> > > >
> > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
> > > <http://gsuttonbreid ing.net/sitemap. xml
> > > >User-Agent: *
> > > >Disallow: /dir/
> > >
> > > Not sure what you mean with multiple functions. Do you mean lock out
> > > different bots from different directories?
> > No, I mean block spidering *and* point to the site map.
> A good place to find info on robots.txt is
> http://www.robotstx t.org/robotstxt. html
> Basically, robots.txt is just for robots. Anyone can see it. Robots
> don't have to honor your wishes, and do so only because they are "nice."
> To make sure a spider knows where your site map is, make sure you have a
> link back to it from each page on your site.
> Good luck, Stephen
- << Previous post in topic Next post in topic >>