Loading ...
Sorry, an error occurred while loading the content.

Re: [NTO] robots.txt question

Expand Messages
  • Mike Breiding - Morgantown WV
    ... No, I mean block spidering *and* point to the site map. All the examples I have seen on the web are for spidering only. Thanks, -Mike
    Message 1 of 8 , Mar 21 12:17 PM
    • 0 Attachment
      loro wrote:
      >
      >
      > Mike wrote:
      > >Can robots.txt be used for multiple functions?
      > >
      > >Such this:
      > >
      > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
      > <http://gsuttonbreiding.net/sitemap.xml>
      > >User-Agent: *
      > >Disallow: /dir/
      >
      > Not sure what you mean with multiple functions. Do you mean lock out
      > different bots from different directories?

      No, I mean block spidering *and* point to the site map.
      All the examples I have seen on the web are for spidering only.
      Thanks,
      -Mike

      >Like disallow bot1, bot2
      > and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.
      >
      > Lotta
      >
    • loro
      ... Oh,sorry! Yeah, sure. Lotta
      Message 2 of 8 , Mar 21 9:18 PM
      • 0 Attachment
        Mike wrote:
        >loro wrote:
        > > Mike wrote:
        > > >Can robots.txt be used for multiple functions?
        > > >
        > > >Such this:
        > > >
        > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
        > > <http://gsuttonbreiding.net/sitemap.xml>
        > > >User-Agent: *
        > > >Disallow: /dir/
        > >
        > > Not sure what you mean with multiple functions. Do you mean lock out
        > > different bots from different directories?
        >
        >No, I mean block spidering *and* point to the site map.
        >All the examples I have seen on the web are for spidering only.
        >Thanks,
        >-Mike

        Oh,sorry! Yeah, sure.

        Lotta
      • Stephen Riddle
        ... A good place to find info on robots.txt is http://www.robotstxt.org/robotstxt.html Basically, robots.txt is just for robots. Anyone can
        Message 3 of 8 , Mar 21 9:23 PM
        • 0 Attachment
          <snips>
          > > Mike wrote:
          > > >Can robots.txt be used for multiple functions?
          > > >
          > > >Such this:
          > > >
          > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
          > > <http://gsuttonbreiding.net/sitemap.xml>
          > > >User-Agent: *
          > > >Disallow: /dir/
          > >
          > > Not sure what you mean with multiple functions. Do you mean lock out
          > > different bots from different directories?
          >
          > No, I mean block spidering *and* point to the site map.

          <snips>

          A good place to find info on robots.txt is
          http://www.robotstxt.org/robotstxt.html

          Basically, robots.txt is just for robots. Anyone can see it. Robots don't have to honor your wishes, and do so only because they are "nice."
          To make sure a spider knows where your site map is, make sure you have a link back to it from each page on your site.

          Good luck,

          Stephen
        • Mike Breiding - Morgantown WV
          Thanks, Steve. -Mike ==========================
          Message 4 of 8 , Mar 22 9:05 AM
          • 0 Attachment
            Thanks, Steve.
            -Mike
            ==========================
            Stephen Riddle wrote:

            > <snips>
            > > > Mike wrote:
            > > > >Can robots.txt be used for multiple functions?
            > > > >
            > > > >Such this:
            > > > >
            > > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
            > > > <http://gsuttonbreid ing.net/sitemap. xml
            > <http://gsuttonbreiding.net/sitemap.xml>>
            > > > >User-Agent: *
            > > > >Disallow: /dir/
            > > >
            > > > Not sure what you mean with multiple functions. Do you mean lock out
            > > > different bots from different directories?
            > >
            > > No, I mean block spidering *and* point to the site map.
            >
            > <snips>
            >
            > A good place to find info on robots.txt is
            > http://www.robotstx t.org/robotstxt. html
            > <http://www.robotstxt.org/robotstxt.html>
            >
            > Basically, robots.txt is just for robots. Anyone can see it. Robots
            > don't have to honor your wishes, and do so only because they are "nice."
            > To make sure a spider knows where your site map is, make sure you have a
            > link back to it from each page on your site.
            >
            > Good luck, Stephen
          • Axel Berger
            ... I don t know the answer to your question, but I would assume, that every robot that knows about sitemaps will also know about the Meta-tag. So just write:
            Message 5 of 8 , Mar 22 9:21 AM
            • 0 Attachment
              Mike Breiding - Morgantown WV wrote:
              > Can robots.txt be used for multiple functions?
              > Sitemap: http://gsuttonbreiding.net/sitemap.xml

              I don't know the answer to your question, but I would assume, that every
              robot that knows about sitemaps will also know about the Meta-tag. So
              just write:

              <link rel="sitemap" href="sitemap.xml" type="application/xml" />

              Axel
            • Mike Breiding - Morgantown WV
              ... I think my sitemap is OK. I was just wondering what all was allowed in a Robot.txt file. Thanks, -Mike
              Message 6 of 8 , Mar 22 9:30 AM
              • 0 Attachment
                Axel Berger wrote:
                >
                >
                > Mike Breiding - Morgantown WV wrote:
                > > Can robots.txt be used for multiple functions?
                > > Sitemap: http://gsuttonbreid ing.net/sitemap. xml
                > <http://gsuttonbreiding.net/sitemap.xml>
                >
                > I don't know the answer to your question, but I would assume, that every
                > robot that knows about sitemaps will also know about the Meta-tag. So
                > just write:
                >
                > <link rel="sitemap" href="sitemap. xml" type="application/ xml" />
                >
                > Axel

                I think my sitemap is OK.
                I was just wondering what all was allowed in a Robot.txt file.

                Thanks,
                -Mike
              Your message has been successfully submitted and would be delivered to recipients shortly.