Loading ...
Sorry, an error occurred while loading the content.
 

robots.txt question

Expand Messages
  • Mike Breiding - Morgantown WV
    Can robots.txt be used for multiple functions? Such this: Sitemap: http://gsuttonbreiding.net/sitemap.xml User-Agent: * Disallow: /dir/ I assume it can, but I
    Message 1 of 8 , Mar 21, 2010
      Can robots.txt be used for multiple functions?

      Such this:

      Sitemap: http://gsuttonbreiding.net/sitemap.xml
      User-Agent: *
      Disallow: /dir/

      I assume it can, but I could not find any example on the web.

      TMike

      --


      Morgantown WV

      www.EpicRoadTrips.us
    • loro
      ... Not sure what you mean with multiple functions. Do you mean lock out different bots from different directories? Like disallow bot1, bot2 and bot3 from
      Message 2 of 8 , Mar 21, 2010
        Mike wrote:
        >Can robots.txt be used for multiple functions?
        >
        >Such this:
        >
        >Sitemap: http://gsuttonbreiding.net/sitemap.xml
        >User-Agent: *
        >Disallow: /dir/

        Not sure what you mean with multiple functions. Do you mean lock out
        different bots from different directories? Like disallow bot1, bot2
        and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.

        Lotta
      • Mike Breiding - Morgantown WV
        ... No, I mean block spidering *and* point to the site map. All the examples I have seen on the web are for spidering only. Thanks, -Mike
        Message 3 of 8 , Mar 21, 2010
          loro wrote:
          >
          >
          > Mike wrote:
          > >Can robots.txt be used for multiple functions?
          > >
          > >Such this:
          > >
          > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
          > <http://gsuttonbreiding.net/sitemap.xml>
          > >User-Agent: *
          > >Disallow: /dir/
          >
          > Not sure what you mean with multiple functions. Do you mean lock out
          > different bots from different directories?

          No, I mean block spidering *and* point to the site map.
          All the examples I have seen on the web are for spidering only.
          Thanks,
          -Mike

          >Like disallow bot1, bot2
          > and bot3 from /dir/dir/ but only bot1 from /dir/. No, I don't think so.
          >
          > Lotta
          >
        • loro
          ... Oh,sorry! Yeah, sure. Lotta
          Message 4 of 8 , Mar 21, 2010
            Mike wrote:
            >loro wrote:
            > > Mike wrote:
            > > >Can robots.txt be used for multiple functions?
            > > >
            > > >Such this:
            > > >
            > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
            > > <http://gsuttonbreiding.net/sitemap.xml>
            > > >User-Agent: *
            > > >Disallow: /dir/
            > >
            > > Not sure what you mean with multiple functions. Do you mean lock out
            > > different bots from different directories?
            >
            >No, I mean block spidering *and* point to the site map.
            >All the examples I have seen on the web are for spidering only.
            >Thanks,
            >-Mike

            Oh,sorry! Yeah, sure.

            Lotta
          • Stephen Riddle
            ... A good place to find info on robots.txt is http://www.robotstxt.org/robotstxt.html Basically, robots.txt is just for robots. Anyone can
            Message 5 of 8 , Mar 21, 2010
              <snips>
              > > Mike wrote:
              > > >Can robots.txt be used for multiple functions?
              > > >
              > > >Such this:
              > > >
              > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
              > > <http://gsuttonbreiding.net/sitemap.xml>
              > > >User-Agent: *
              > > >Disallow: /dir/
              > >
              > > Not sure what you mean with multiple functions. Do you mean lock out
              > > different bots from different directories?
              >
              > No, I mean block spidering *and* point to the site map.

              <snips>

              A good place to find info on robots.txt is
              http://www.robotstxt.org/robotstxt.html

              Basically, robots.txt is just for robots. Anyone can see it. Robots don't have to honor your wishes, and do so only because they are "nice."
              To make sure a spider knows where your site map is, make sure you have a link back to it from each page on your site.

              Good luck,

              Stephen
            • Mike Breiding - Morgantown WV
              Thanks, Steve. -Mike ==========================
              Message 6 of 8 , Mar 22, 2010
                Thanks, Steve.
                -Mike
                ==========================
                Stephen Riddle wrote:

                > <snips>
                > > > Mike wrote:
                > > > >Can robots.txt be used for multiple functions?
                > > > >
                > > > >Such this:
                > > > >
                > > > >Sitemap: http://gsuttonbreid ing.net/sitemap. xml
                > > > <http://gsuttonbreid ing.net/sitemap. xml
                > <http://gsuttonbreiding.net/sitemap.xml>>
                > > > >User-Agent: *
                > > > >Disallow: /dir/
                > > >
                > > > Not sure what you mean with multiple functions. Do you mean lock out
                > > > different bots from different directories?
                > >
                > > No, I mean block spidering *and* point to the site map.
                >
                > <snips>
                >
                > A good place to find info on robots.txt is
                > http://www.robotstx t.org/robotstxt. html
                > <http://www.robotstxt.org/robotstxt.html>
                >
                > Basically, robots.txt is just for robots. Anyone can see it. Robots
                > don't have to honor your wishes, and do so only because they are "nice."
                > To make sure a spider knows where your site map is, make sure you have a
                > link back to it from each page on your site.
                >
                > Good luck, Stephen
              • Axel Berger
                ... I don t know the answer to your question, but I would assume, that every robot that knows about sitemaps will also know about the Meta-tag. So just write:
                Message 7 of 8 , Mar 22, 2010
                  Mike Breiding - Morgantown WV wrote:
                  > Can robots.txt be used for multiple functions?
                  > Sitemap: http://gsuttonbreiding.net/sitemap.xml

                  I don't know the answer to your question, but I would assume, that every
                  robot that knows about sitemaps will also know about the Meta-tag. So
                  just write:

                  <link rel="sitemap" href="sitemap.xml" type="application/xml" />

                  Axel
                • Mike Breiding - Morgantown WV
                  ... I think my sitemap is OK. I was just wondering what all was allowed in a Robot.txt file. Thanks, -Mike
                  Message 8 of 8 , Mar 22, 2010
                    Axel Berger wrote:
                    >
                    >
                    > Mike Breiding - Morgantown WV wrote:
                    > > Can robots.txt be used for multiple functions?
                    > > Sitemap: http://gsuttonbreid ing.net/sitemap. xml
                    > <http://gsuttonbreiding.net/sitemap.xml>
                    >
                    > I don't know the answer to your question, but I would assume, that every
                    > robot that knows about sitemaps will also know about the Meta-tag. So
                    > just write:
                    >
                    > <link rel="sitemap" href="sitemap. xml" type="application/ xml" />
                    >
                    > Axel

                    I think my sitemap is OK.
                    I was just wondering what all was allowed in a Robot.txt file.

                    Thanks,
                    -Mike
                  Your message has been successfully submitted and would be delivered to recipients shortly.