Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] Re: Wikipedia links not working for Xenu?

Expand Messages
  • andi
    Thanks for this Tom. When I wrote that (It has been over a year) and put Prudential on the exeception list it was returning dozens of redirects to the Xenu
    Message 1 of 6 , Jul 26, 2005
    • 0 Attachment
      Thanks for this Tom. When I wrote that (It has been over a year) and put Prudential on the exeception list it was returning dozens of redirects to the Xenu result page. They must have fixed the bug in their rewrite rule (or whatever, perhaps there's a change in the Xenu version) in the meantime. I'll revise the text on my Link checking page. Thanks again.

      Andi
      ----- Original Message -----
      From: adiscretemale
      To: xenu-usergroup@yahoogroups.com
      Sent: Monday, July 25, 2005 3:57 AM
      Subject: [xenu-usergroup] Re: Wikipedia links not working for Xenu?


      Andi, I checked your page and noted that you say "There is one major
      company URL (http://www.prudential.com/) that will recycle through
      dozens of redirects for Xenu yet behaves normally for a browser so I
      simply omit it as well." Out of curiousity, I tried starting with
      that URL with Xenu and it only went through 3 redirects before
      actually going into the site. The redirects appear to be the same as
      those a browser gets since if you enter http://www.prudential.com/ in
      browser, when the page comes up, the URL shows as
      http://www.prudential.com/index/.

      Investigating further, I determined that
      http://www.prudential.com/
      first redirects to
      http://www.prudential.com/default/
      that, in turn, redirects to
      http://www.prudential.com/index
      and that, again, redirects to
      http://www.prudential.com/index/
      and that returns a normal page.

      Tom
      --- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
      > I've long had this same experience with Wikipedia using other bots
      as well as Xenu. (returns "Wikimedia error") I have always assumed
      that Wikipedia blocks anything that even slightly smells like a bot
      to keep their content from being spidered. Irresponsible bots are
      becoming so numerous that I'd like to do this on my site as well but
      such blocking is more difficult with static pages. I do find that my
      list of sites seeming to block Xenu continually grows (partial list
      here http://www.andilinks.com/linkckg.htm#55000).
      >
      > Andi





      ------------------------------------------------------------------------------
      YAHOO! GROUPS LINKS

      a.. Visit your group "xenu-usergroup" on the web.

      b.. To unsubscribe from this group, send an email to:
      xenu-usergroup-unsubscribe@yahoogroups.com

      c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


      ------------------------------------------------------------------------------



      [Non-text portions of this message have been removed]
    Your message has been successfully submitted and would be delivered to recipients shortly.