Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] Re: wholesale Xenu abuse and blocking

Expand Messages
  • Tom Aman
    This is mostly a comment for Tilman. I know you have, so far, been resistant to having Xenu use a different user agent name. As I indicated when I applied to
    Message 1 of 6 , Mar 28 2:24 PM
    • 0 Attachment
      This is mostly a comment for Tilman. 
       
      I know you have, so far, been resistant to having Xenu use a different user agent name.  As I indicated when I applied to join the group, I also market a link check (CyberSpyder Link Test) and I would suggest that the usefulness of Xenu would be improved if it included an option to "cloak" itself (via user-agent) as a browser such as IE, Netscape, etc. 
       
      I have looked into this issue to some extent and, aside from the blocking issue, some sites will not respond properly (will give a 403 or 500 error) if the user-agent is anything other than a browser.   I identified this problem by using my WebBug program - using the normal WebBug user agent on some sites, I would get an error.  Changing this to a browser user agent would result in a normal page being returned.  For this reason, the new version of my program will include options for this kind of cloaking.  I suggest Xenu would benefit and be more useful if it also included such an option.
       
      Tom Aman

      Tilman Hausherr <tilman@...> wrote:
      On Tue, 28 Mar 2006 11:20:50 -0000, frank visser wrote:

      >hi tilmann,
      >
      >could you comment on andy's message?
      >
      >what are the options?^

      There are none. I can't really make a solution that would use less
      bandwith. Xenu is already optimized to use HEAD instead of GET in many
      cases - but a link checker must go through a whole website. I suspect
      that other link checkers are not disabled simply because they are less
      popular, therefore below the radar of paranoid sysadmins.

      Tilman

      >
      >frank
      >
      >--- In xenu-usergroup@yahoogroups.com, "andi" <andi@...> wrote:
      >>
      >> Hi everyone,
      >>
      >> I have just scanned all these user-group emails going back to
      >Feb '04 and do not see this problem addressed.
      >>
      >> The "Xenu link Sleuth" user agent is being blocked by an increasing
      >number of sites and hosts and this is becoming a serious problem for
      >me to the point where I may have to begin a search for a new link
      >checker.  I do not abuse the capability of Xenu to ping many levels
      >of pages, I check a local page of links compiled from my database and
      >my use could hardly be considered a nuisance.
      >>
      >> But I have been blocking Xenu from my own site for months now
      >because of the increasing abuse by others and now have learned that
      >my web host, FutureQuest is blocking Xenu from all their sites.  This
      >is bad for the internet but it's worse for Xenu.  Perhaps a version
      >of Xenu that is less easily abused as a bandwidth hogging rogue bot
      >(with a new user agent name) could stem this problem.
      >>
      >> reference this thread at my host's community forum:
      >>
      >> http://www.aota.net/forums/showthread.php?t=21146
      >>
      >> Andi
      >>
      >
      >
      >
      >
      >
      >
      >
      >Yahoo! Groups Links
      >
      >
      >
      >
      >
      >




      Yahoo! Messenger with Voice. PC-to-Phone calls for ridiculously low rates.

    Your message has been successfully submitted and would be delivered to recipients shortly.