Loading ...
Sorry, an error occurred while loading the content.

[xenu-usergroup] Future feature - Honoring robot META tags ?

Expand Messages
  • Richard A. DeVenezia
    Here is an idea for future versions: Have a checkbox to tell Xenu to honor directives of the robots name in a META tag For example, some of my pages are just
    Message 1 of 1 , Mar 16, 2005
    View Source
    • 0 Attachment
      Here is an idea for future versions:

      Have a checkbox to tell Xenu to honor directives of the robots name in a
      META tag

      For example, some of my pages are just 'link aggregators' (which come in
      handy when viewing the site in HotMetalPro)
      In this case, if the "Honor META robots directive" was checked, Xenu would
      _not_ try to follow all the HREFS on the page, because the META content has
      NOFOLLOW.

      <HTML>
      <HEAD>
      <TITLE>Requires of Mary Lupardi</TITLE>
      <META NAME="robots" CONTENT="noindex,nofollow">
      <LINK HREF=".htaccess">
      <LINK HREF="../index.html">
      <LINK HREF="../memoirs.php">
      <LINK HREF="../images/navdot.png">
      <LINK HREF="php/load_nav_array.inc">
      <LINK HREF="php/nav_array.inc">
      <LINK HREF="php/nav.php">
      <LINK HREF="php/section.php">
      <LINK HREF="php/annotate.php">
      <LINK HREF="php/counter.inc">
      <LINK HREF="html/guestbook.htm">
      <LINK HREF="html/mem00.htm"> <LINK HREF="html/mem01.htm">
      <LINK HREF="html/mem02.htm"> <LINK HREF="html/mem03.htm">
      <LINK HREF="html/mem04.htm"> <LINK HREF="html/mem05.htm">
      </HEAD>
      <BODY> </BODY>
      </HTML>


      Likewise, honoring the directives of a "robots.txt" that is found in a
      folder would sometimes be handy.

      Richard A. DeVenezia
    Your message has been successfully submitted and would be delivered to recipients shortly.