Loading ...
Sorry, an error occurred while loading the content.

Re: [NTO] robots.txt question

Expand Messages
  • loro
    ... Not sure what you mean with multiple functions. Do you mean lock out different bots from different directories? Like disallow bot1, bot2 and bot3 from
    Message 1 of 8 , Mar 21, 2010
    • 0 Attachment
      Mike wrote:
      >Can robots.txt be used for multiple functions?
      >
      >Such this:
      >
      >Sitemap: http://gsuttonbreiding.net/sitemap.xml
      >User-Agent: *
      >Disallow: /dir/

      Not sure what you mean with multiple functions. Do you mean lock out
      different bots from different directories? Like disallow bot1, bot2
      and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.

      Lotta
    • Mike Breiding - Morgantown WV
      ... No, I mean block spidering *and* point to the site map. All the examples I have seen on the web are for spidering only. Thanks, -Mike
      Message 2 of 8 , Mar 21, 2010
      • 0 Attachment
        loro wrote:
        >
        >
        > Mike wrote:
        > >Can robots.txt be used for multiple functions?
        > >
        > >Such this:
        > >
        > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
        > <http://gsuttonbreiding.net/sitemap.xml>
        > >User-Agent: *
        > >Disallow: /dir/
        >
        > Not sure what you mean with multiple functions. Do you mean lock out
        > different bots from different directories?

        No, I mean block spidering *and* point to the site map.
        All the examples I have seen on the web are for spidering only.
        Thanks,
        -Mike

        >Like disallow bot1, bot2
        > and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.
        >
        > Lotta
        >
      • loro
        ... Oh,sorry! Yeah, sure. Lotta
        Message 3 of 8 , Mar 21, 2010
        • 0 Attachment
          Mike wrote:
          >loro wrote:
          > > Mike wrote:
          > > >Can robots.txt be used for multiple functions?
          > > >
          > > >Such this:
          > > >
          > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
          > > <http://gsuttonbreiding.net/sitemap.xml>
          > > >User-Agent: *
          > > >Disallow: /dir/
          > >
          > > Not sure what you mean with multiple functions. Do you mean lock out
          > > different bots from different directories?
          >
          >No, I mean block spidering *and* point to the site map.
          >All the examples I have seen on the web are for spidering only.
          >Thanks,
          >-Mike

          Oh,sorry! Yeah, sure.

          Lotta
        • Stephen Riddle
          ... A good place to find info on robots.txt is http://www.robotstxt.org/robotstxt.html Basically, robots.txt is just for robots. Anyone can
          Message 4 of 8 , Mar 21, 2010
          • 0 Attachment
            <snips>
            > > Mike wrote:
            > > >Can robots.txt be used for multiple functions?
            > > >
            > > >Such this:
            > > >
            > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
            > > <http://gsuttonbreiding.net/sitemap.xml>
            > > >User-Agent: *
            > > >Disallow: /dir/
            > >
            > > Not sure what you mean with multiple functions. Do you mean lock out
            > > different bots from different directories?
            >
            > No, I mean block spidering *and* point to the site map.

            <snips>

            A good place to find info on robots.txt is
            http://www.robotstxt.org/robotstxt.html

            Basically, robots.txt is just for robots. Anyone can see it. Robots don't have to honor your wishes, and do so only because they are "nice."
            To make sure a spider knows where your site map is, make sure you have a link back to it from each page on your site.

            Good luck,

            Stephen
          • Mike Breiding - Morgantown WV
            Thanks, Steve. -Mike ==========================
            Message 5 of 8 , Mar 22, 2010
            • 0 Attachment
              Thanks, Steve.
              -Mike
              ==========================
              Stephen Riddle wrote:

              > <snips>
              > > > Mike wrote:
              > > > >Can robots.txt be used for multiple functions?
              > > > >
              > > > >Such this:
              > > > >
              > > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
              > > > <http://gsuttonbreid ing.net/sitemap. xml
              > <http://gsuttonbreiding.net/sitemap.xml>>
              > > > >User-Agent: *
              > > > >Disallow: /dir/
              > > >
              > > > Not sure what you mean with multiple functions. Do you mean lock out
              > > > different bots from different directories?
              > >
              > > No, I mean block spidering *and* point to the site map.
              >
              > <snips>
              >
              > A good place to find info on robots.txt is
              > http://www.robotstx t.org/robotstxt. html
              > <http://www.robotstxt.org/robotstxt.html>
              >
              > Basically, robots.txt is just for robots. Anyone can see it. Robots
              > don't have to honor your wishes, and do so only because they are "nice."
              > To make sure a spider knows where your site map is, make sure you have a
              > link back to it from each page on your site.
              >
              > Good luck, Stephen
            • Axel Berger
              ... I don t know the answer to your question, but I would assume, that every robot that knows about sitemaps will also know about the Meta-tag. So just write:
              Message 6 of 8 , Mar 22, 2010
              • 0 Attachment
                Mike Breiding - Morgantown WV wrote:
                > Can robots.txt be used for multiple functions?
                > Sitemap: http://gsuttonbreiding.net/sitemap.xml

                I don't know the answer to your question, but I would assume, that every
                robot that knows about sitemaps will also know about the Meta-tag. So
                just write:

                <link rel="sitemap" href="sitemap.xml" type="application/xml" />

                Axel
              • Mike Breiding - Morgantown WV
                ... I think my sitemap is OK. I was just wondering what all was allowed in a Robot.txt file. Thanks, -Mike
                Message 7 of 8 , Mar 22, 2010
                • 0 Attachment
                  Axel Berger wrote:
                  >
                  >
                  > Mike Breiding - Morgantown WV wrote:
                  > > Can robots.txt be used for multiple functions?
                  > > Sitemap: http://gsuttonbreid ing.net/sitemap. xml
                  > <http://gsuttonbreiding.net/sitemap.xml>
                  >
                  > I don't know the answer to your question, but I would assume, that every
                  > robot that knows about sitemaps will also know about the Meta-tag. So
                  > just write:
                  >
                  > <link rel="sitemap" href="sitemap. xml" type="application/ xml" />
                  >
                  > Axel

                  I think my sitemap is OK.
                  I was just wondering what all was allowed in a Robot.txt file.

                  Thanks,
                  -Mike
                Your message has been successfully submitted and would be delivered to recipients shortly.