Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] Re: wholesale Xenu abuse and blocking

Expand Messages
  • Tom Aman
    This is mostly a comment for Tilman. I know you have, so far, been resistant to having Xenu use a different user agent name. As I indicated when I applied to
    Message 1 of 6 , Mar 28, 2006
    • 0 Attachment
      This is mostly a comment for Tilman. 
       
      I know you have, so far, been resistant to having Xenu use a different user agent name.  As I indicated when I applied to join the group, I also market a link check (CyberSpyder Link Test) and I would suggest that the usefulness of Xenu would be improved if it included an option to "cloak" itself (via user-agent) as a browser such as IE, Netscape, etc. 
       
      I have looked into this issue to some extent and, aside from the blocking issue, some sites will not respond properly (will give a 403 or 500 error) if the user-agent is anything other than a browser.   I identified this problem by using my WebBug program - using the normal WebBug user agent on some sites, I would get an error.  Changing this to a browser user agent would result in a normal page being returned.  For this reason, the new version of my program will include options for this kind of cloaking.  I suggest Xenu would benefit and be more useful if it also included such an option.
       
      Tom Aman

      Tilman Hausherr <tilman@...> wrote:
      On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

      >hi tilmann,
      >
      >could you comment on andy's message?
      >
      >what are the options?^

      There are none. I can't really make a solution that would use less
      bandwith. Xenu is already optimized to use HEAD instead of GET in many
      cases - but a link checker must go through a whole website. I suspect
      that other link checkers are not disabled simply because they are less
      popular, therefore below the radar of paranoid sysadmins.

      Tilman

      >
      >frank
      >
      >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
      >>
      >> Hi everyone,
      >>
      >> I have just scanned all these user-group emails going back to
      >Feb '04 and do not see this problem addressed.
      >>
      >> The "Xenu link Sleuth" user agent is being blocked by an increasing
      >number of sites and hosts and this is becoming a serious problem for
      >me to the point where I may have to begin a search for a new link
      >checker.  I do not abuse the capability of Xenu to ping many levels
      >of pages, I check a local page of links compiled from my database and
      >my use could hardly be considered a nuisance.
      >>
      >> But I have been blocking Xenu from my own site for months now
      >because of the increasing abuse by others and now have learned that
      >my web host, FutureQuest is blocking Xenu from all their sites.  This
      >is bad for the internet but it's worse for Xenu.  Perhaps a version
      >of Xenu that is less easily abused as a bandwidth hogging rogue bot
      >(with a new user agent name) could stem this problem.
      >>
      >> reference this thread at my host's community forum:
      >>
      >> http://www.aota.net/forums/showthread.php?t=21146
      >>
      >> Andi
      >>
      >
      >
      >
      >
      >
      >
      >
      >Yahoo! Groups Links
      >
      >
      >
      >
      >
      >




      Yahoo! Messenger with Voice. PC-to-Phone calls for ridiculously low rates.

    • andi
      The trend I see is more agressive bots and more routine blocking. I guess market share doesn t really matter with a free application but if Xenu s popularity
      Message 2 of 6 , Mar 29, 2006
      • 0 Attachment
        The trend I see is more agressive bots and more routine blocking.  I guess market share doesn't really matter with a free application but if Xenu's popularity causes it to be less useful the options aren't pleasant.
         
        I don't know how large the demand and therefore whether it is practical but a checker with a different user agent name that only checked links on specific single local files would be more useful to me than the current one which requires a manual recheck on those forbidden.  I check links in my database by making local pages specifically for checking partly because it treads more lightly on the network and partly because it is easier to match them up when exported to the database.
         
        It seems those capabilities could be toggled in a future version.  If the user agent remains the same on the lighter version there's no difference.
         
        But I may be unique in that regard...
         
        Andi
        ----- Original Message -----
        Sent: Tuesday, March 28, 2006 11:10 AM
        Subject: Re: [xenu-usergroup] Re: wholesale Xenu abuse and blocking

        On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

        >hi tilmann,
        >
        >could you comment on andy's message?
        >
        >what are the options?^

        There are none. I can't really make a solution that would use less
        bandwith. Xenu is already optimized to use HEAD instead of GET in many
        cases - but a link checker must go through a whole website. I suspect
        that other link checkers are not disabled simply because they are less
        popular, therefore below the radar of paranoid sysadmins.

        Tilman

        >
        >frank
        >
        >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
        >>
        >> Hi everyone,
        >>
        >> I have just scanned all these user-group emails going back to
        >Feb '04 and do not see this problem addressed.
        >>
        >> The "Xenu link Sleuth" user agent is being blocked by an increasing
        >number of sites and hosts and this is becoming a serious problem for
        >me to the point where I may have to begin a search for a new link
        >checker.  I do not abuse the capability of Xenu to ping many levels
        >of pages, I check a local page of links compiled from my database and
        >my use could hardly be considered a nuisance.
        >>
        >> But I have been blocking Xenu from my own site for months now
        >because of the increasing abuse by others and now have learned that
        >my web host, FutureQuest is blocking Xenu from all their sites.  This
        >is bad for the internet but it's worse for Xenu.  Perhaps a version
        >of Xenu that is less easily abused as a bandwidth hogging rogue bot
        >(with a new user agent name) could stem this problem.
        >>
        >> reference this thread at my host's community forum:
        >>
        >> http://www.aota.net/forums/showthread.php?t=21146
        >>
        >> Andi
        >>
        >
        >
        >
        >
        >
        >
        >
        >Yahoo! Groups Links
        >
        >
        >
        >
        >
        >


      Your message has been successfully submitted and would be delivered to recipients shortly.