Loading ...
Sorry, an error occurred while loading the content.

380Re: [xenu-usergroup] Re: Wikipedia links not working for Xenu?

Expand Messages
  • andi
    Jul 26, 2005
    • 0 Attachment
      Thanks for this Tom. When I wrote that (It has been over a year) and put Prudential on the exeception list it was returning dozens of redirects to the Xenu result page. They must have fixed the bug in their rewrite rule (or whatever, perhaps there's a change in the Xenu version) in the meantime. I'll revise the text on my Link checking page. Thanks again.

      ----- Original Message -----
      From: adiscretemale
      To: xenu-usergroup@yahoogroups.com
      Sent: Monday, July 25, 2005 3:57 AM
      Subject: [xenu-usergroup] Re: Wikipedia links not working for Xenu?

      Andi, I checked your page and noted that you say "There is one major
      company URL (http://www.prudential.com/) that will recycle through
      dozens of redirects for Xenu yet behaves normally for a browser so I
      simply omit it as well." Out of curiousity, I tried starting with
      that URL with Xenu and it only went through 3 redirects before
      actually going into the site. The redirects appear to be the same as
      those a browser gets since if you enter http://www.prudential.com/ in
      browser, when the page comes up, the URL shows as

      Investigating further, I determined that
      first redirects to
      that, in turn, redirects to
      and that, again, redirects to
      and that returns a normal page.

      --- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
      > I've long had this same experience with Wikipedia using other bots
      as well as Xenu. (returns "Wikimedia error") I have always assumed
      that Wikipedia blocks anything that even slightly smells like a bot
      to keep their content from being spidered. Irresponsible bots are
      becoming so numerous that I'd like to do this on my site as well but
      such blocking is more difficult with static pages. I do find that my
      list of sites seeming to block Xenu continually grows (partial list
      here http://www.andilinks.com/linkckg.htm#55000).
      > Andi


      a.. Visit your group "xenu-usergroup" on the web.

      b.. To unsubscribe from this group, send an email to:

      c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


      [Non-text portions of this message have been removed]
    • Show all 6 messages in this topic