Loading ...
Sorry, an error occurred while loading the content.

wholesale Xenu abuse and blocking

Expand Messages
  • andi
    Hi everyone, I have just scanned all these user-group emails going back to Feb 04 and do not see this problem addressed. The Xenu link Sleuth user agent is
    Message 1 of 6 , Mar 24, 2006
    • 0 Attachment
      Hi everyone,
       
      I have just scanned all these user-group emails going back to Feb '04 and do not see this problem addressed.
       
      The "Xenu link Sleuth" user agent is being blocked by an increasing number of sites and hosts and this is becoming a serious problem for me to the point where I may have to begin a search for a new link checker.  I do not abuse the capability of Xenu to ping many levels of pages, I check a local page of links compiled from my database and my use could hardly be considered a nuisance.
       
      But I have been blocking Xenu from my own site for months now because of the increasing abuse by others and now have learned that my web host, FutureQuest is blocking Xenu from all their sites.  This is bad for the internet but it's worse for Xenu.  Perhaps a version of Xenu that is less easily abused as a bandwidth hogging rogue bot (with a new user agent name) could stem this problem.
       
      reference this thread at my host's community forum:
       
       
      Andi
    • frank visser
      hi tilmann, could you comment on andy s message? what are the options? frank ... Feb 04 and do not see this problem addressed. ... number of sites and hosts
      Message 2 of 6 , Mar 28, 2006
      • 0 Attachment
        hi tilmann,

        could you comment on andy's message?

        what are the options?

        frank

        --- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
        >
        > Hi everyone,
        >
        > I have just scanned all these user-group emails going back to
        Feb '04 and do not see this problem addressed.
        >
        > The "Xenu link Sleuth" user agent is being blocked by an increasing
        number of sites and hosts and this is becoming a serious problem for
        me to the point where I may have to begin a search for a new link
        checker. I do not abuse the capability of Xenu to ping many levels
        of pages, I check a local page of links compiled from my database and
        my use could hardly be considered a nuisance.
        >
        > But I have been blocking Xenu from my own site for months now
        because of the increasing abuse by others and now have learned that
        my web host, FutureQuest is blocking Xenu from all their sites. This
        is bad for the internet but it's worse for Xenu. Perhaps a version
        of Xenu that is less easily abused as a bandwidth hogging rogue bot
        (with a new user agent name) could stem this problem.
        >
        > reference this thread at my host's community forum:
        >
        > http://www.aota.net/forums/showthread.php?t=21146
        >
        > Andi
        >
      • andi
        From the beginning, which for me was late 2002, there have been a few sites blocking Xenu. They were usually the very geekiest sites, the ones who are
        Message 3 of 6 , Mar 28, 2006
        • 0 Attachment
          From the beginning, which for me was late 2002, there have been a few sites blocking Xenu.  They were usually the very geekiest sites, the ones who are constantly scanning their logs for rogue bots.  I admit, I do this from time to time.  I have thought that my site attracted a lot of Xenu abuse because of my huge number of outbound links and the fact that I have a page *about* Xenu which attracts link checking traffic from Google as well as from the link on the snafu.de site itself. 
           
          But many of the "bad bot lists" that have been circulating lately list Xenu.  My web host, FutureQuest, is usually on the leading edge with security and bandwidth issues so it doesn't surprise me that they are among the first to block Xenu wholesale.  I expect they won't be the last.  The number is still small as a percentage but it is definitely trending upward. 
          ----- Original Message -----
          Sent: Tuesday, March 28, 2006 4:20 AM
          Subject: [xenu-usergroup] Re: wholesale Xenu abuse and blocking

          hi tilmann,

          could you comment on andy's message?

          what are the options?

          frank

          --- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
          >
          > Hi everyone,
          >
          > I have just scanned all these user-group emails going back to
          Feb '04 and do not see this problem addressed.
          >
          > The "Xenu link Sleuth" user agent is being blocked by an increasing
          number of sites and hosts and this is becoming a serious problem for
          me to the point where I may have to begin a search for a new link
          checker.  I do not abuse the capability of Xenu to ping many levels
          of pages, I check a local page of links compiled from my database and
          my use could hardly be considered a nuisance.
          >
          > But I have been blocking Xenu from my own site for months now
          because of the increasing abuse by others and now have learned that
          my web host, FutureQuest is blocking Xenu from all their sites.  This
          is bad for the internet but it's worse for Xenu.  Perhaps a version
          of Xenu that is less easily abused as a bandwidth hogging rogue bot
          (with a new user agent name) could stem this problem.
          >
          > reference this thread at my host's community forum:
          >
          > http://www.aota.net/forums/showthread.php?t=21146
          >
          > Andi
          >




        • Tilman Hausherr
          ... There are none. I can t really make a solution that would use less bandwith. Xenu is already optimized to use HEAD instead of GET in many cases - but a
          Message 4 of 6 , Mar 28, 2006
          • 0 Attachment
            On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

            >hi tilmann,
            >
            >could you comment on andy's message?
            >
            >what are the options?^

            There are none. I can't really make a solution that would use less
            bandwith. Xenu is already optimized to use HEAD instead of GET in many
            cases - but a link checker must go through a whole website. I suspect
            that other link checkers are not disabled simply because they are less
            popular, therefore below the radar of paranoid sysadmins.

            Tilman

            >
            >frank
            >
            >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
            >>
            >> Hi everyone,
            >>
            >> I have just scanned all these user-group emails going back to
            >Feb '04 and do not see this problem addressed.
            >>
            >> The "Xenu link Sleuth" user agent is being blocked by an increasing
            >number of sites and hosts and this is becoming a serious problem for
            >me to the point where I may have to begin a search for a new link
            >checker. I do not abuse the capability of Xenu to ping many levels
            >of pages, I check a local page of links compiled from my database and
            >my use could hardly be considered a nuisance.
            >>
            >> But I have been blocking Xenu from my own site for months now
            >because of the increasing abuse by others and now have learned that
            >my web host, FutureQuest is blocking Xenu from all their sites. This
            >is bad for the internet but it's worse for Xenu. Perhaps a version
            >of Xenu that is less easily abused as a bandwidth hogging rogue bot
            >(with a new user agent name) could stem this problem.
            >>
            >> reference this thread at my host's community forum:
            >>
            >> http://www.aota.net/forums/showthread.php?t=21146
            >>
            >> Andi
            >>
            >
            >
            >
            >
            >
            >
            >
            >Yahoo! Groups Links
            >
            >
            >
            >
            >
            >
          • Tom Aman
            This is mostly a comment for Tilman. I know you have, so far, been resistant to having Xenu use a different user agent name. As I indicated when I applied to
            Message 5 of 6 , Mar 28, 2006
            • 0 Attachment
              This is mostly a comment for Tilman. 
               
              I know you have, so far, been resistant to having Xenu use a different user agent name.  As I indicated when I applied to join the group, I also market a link check (CyberSpyder Link Test) and I would suggest that the usefulness of Xenu would be improved if it included an option to "cloak" itself (via user-agent) as a browser such as IE, Netscape, etc. 
               
              I have looked into this issue to some extent and, aside from the blocking issue, some sites will not respond properly (will give a 403 or 500 error) if the user-agent is anything other than a browser.   I identified this problem by using my WebBug program - using the normal WebBug user agent on some sites, I would get an error.  Changing this to a browser user agent would result in a normal page being returned.  For this reason, the new version of my program will include options for this kind of cloaking.  I suggest Xenu would benefit and be more useful if it also included such an option.
               
              Tom Aman

              Tilman Hausherr <tilman@...> wrote:
              On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

              >hi tilmann,
              >
              >could you comment on andy's message?
              >
              >what are the options?^

              There are none. I can't really make a solution that would use less
              bandwith. Xenu is already optimized to use HEAD instead of GET in many
              cases - but a link checker must go through a whole website. I suspect
              that other link checkers are not disabled simply because they are less
              popular, therefore below the radar of paranoid sysadmins.

              Tilman

              >
              >frank
              >
              >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
              >>
              >> Hi everyone,
              >>
              >> I have just scanned all these user-group emails going back to
              >Feb '04 and do not see this problem addressed.
              >>
              >> The "Xenu link Sleuth" user agent is being blocked by an increasing
              >number of sites and hosts and this is becoming a serious problem for
              >me to the point where I may have to begin a search for a new link
              >checker.  I do not abuse the capability of Xenu to ping many levels
              >of pages, I check a local page of links compiled from my database and
              >my use could hardly be considered a nuisance.
              >>
              >> But I have been blocking Xenu from my own site for months now
              >because of the increasing abuse by others and now have learned that
              >my web host, FutureQuest is blocking Xenu from all their sites.  This
              >is bad for the internet but it's worse for Xenu.  Perhaps a version
              >of Xenu that is less easily abused as a bandwidth hogging rogue bot
              >(with a new user agent name) could stem this problem.
              >>
              >> reference this thread at my host's community forum:
              >>
              >> http://www.aota.net/forums/showthread.php?t=21146
              >>
              >> Andi
              >>
              >
              >
              >
              >
              >
              >
              >
              >Yahoo! Groups Links
              >
              >
              >
              >
              >
              >




              Yahoo! Messenger with Voice. PC-to-Phone calls for ridiculously low rates.

            • andi
              The trend I see is more agressive bots and more routine blocking. I guess market share doesn t really matter with a free application but if Xenu s popularity
              Message 6 of 6 , Mar 29, 2006
              • 0 Attachment
                The trend I see is more agressive bots and more routine blocking.  I guess market share doesn't really matter with a free application but if Xenu's popularity causes it to be less useful the options aren't pleasant.
                 
                I don't know how large the demand and therefore whether it is practical but a checker with a different user agent name that only checked links on specific single local files would be more useful to me than the current one which requires a manual recheck on those forbidden.  I check links in my database by making local pages specifically for checking partly because it treads more lightly on the network and partly because it is easier to match them up when exported to the database.
                 
                It seems those capabilities could be toggled in a future version.  If the user agent remains the same on the lighter version there's no difference.
                 
                But I may be unique in that regard...
                 
                Andi
                ----- Original Message -----
                Sent: Tuesday, March 28, 2006 11:10 AM
                Subject: Re: [xenu-usergroup] Re: wholesale Xenu abuse and blocking

                On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

                >hi tilmann,
                >
                >could you comment on andy's message?
                >
                >what are the options?^

                There are none. I can't really make a solution that would use less
                bandwith. Xenu is already optimized to use HEAD instead of GET in many
                cases - but a link checker must go through a whole website. I suspect
                that other link checkers are not disabled simply because they are less
                popular, therefore below the radar of paranoid sysadmins.

                Tilman

                >
                >frank
                >
                >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
                >>
                >> Hi everyone,
                >>
                >> I have just scanned all these user-group emails going back to
                >Feb '04 and do not see this problem addressed.
                >>
                >> The "Xenu link Sleuth" user agent is being blocked by an increasing
                >number of sites and hosts and this is becoming a serious problem for
                >me to the point where I may have to begin a search for a new link
                >checker.  I do not abuse the capability of Xenu to ping many levels
                >of pages, I check a local page of links compiled from my database and
                >my use could hardly be considered a nuisance.
                >>
                >> But I have been blocking Xenu from my own site for months now
                >because of the increasing abuse by others and now have learned that
                >my web host, FutureQuest is blocking Xenu from all their sites.  This
                >is bad for the internet but it's worse for Xenu.  Perhaps a version
                >of Xenu that is less easily abused as a bandwidth hogging rogue bot
                >(with a new user agent name) could stem this problem.
                >>
                >> reference this thread at my host's community forum:
                >>
                >> http://www.aota.net/forums/showthread.php?t=21146
                >>
                >> Andi
                >>
                >
                >
                >
                >
                >
                >
                >
                >Yahoo! Groups Links
                >
                >
                >
                >
                >
                >


              Your message has been successfully submitted and would be delivered to recipients shortly.