Loading ...
Sorry, an error occurred while loading the content.

Wikipedia links not working for Xenu?

Expand Messages
  • frank visser
    hi tilman, do you know why Wikipedia links of the kind: http://en.wikipedia.org/wiki/Alfred_North_Whitehead are not working for xenu? i get a forbidden
    Message 1 of 6 , Jul 7 6:04 AM
    • 0 Attachment
      hi tilman,

      do you know why Wikipedia links of the kind:
      http://en.wikipedia.org/wiki/Alfred_North_Whitehead

      are not working for xenu?

      i get a "forbidden request" message.

      is it becaue the URL has a slightly uncommon format:

      http://en.wikipedia.org/wiki/KEYWORD ?

      tnx,

      frank
    • andi
      I ve long had this same experience with Wikipedia using other bots as well as Xenu. (returns Wikimedia error ) I have always assumed that Wikipedia blocks
      Message 2 of 6 , Jul 7 7:37 AM
      • 0 Attachment
        I've long had this same experience with Wikipedia using other bots as well as Xenu. (returns "Wikimedia error") I have always assumed that Wikipedia blocks anything that even slightly smells like a bot to keep their content from being spidered. Irresponsible bots are becoming so numerous that I'd like to do this on my site as well but such blocking is more difficult with static pages. I do find that my list of sites seeming to block Xenu continually grows (partial list here http://www.andilinks.com/linkckg.htm#55000).

        Andi
        ----- Original Message -----
        From: frank visser
        To: xenu-usergroup@yahoogroups.com
        Sent: Thursday, July 07, 2005 7:04 AM
        Subject: [xenu-usergroup] Wikipedia links not working for Xenu?


        hi tilman,

        do you know why Wikipedia links of the kind:
        http://en.wikipedia.org/wiki/Alfred_North_Whitehead

        are not working for xenu?

        i get a "forbidden request" message.

        is it becaue the URL has a slightly uncommon format:

        http://en.wikipedia.org/wiki/KEYWORD ?

        tnx,

        frank




        ------------------------------------------------------------------------------
        YAHOO! GROUPS LINKS

        a.. Visit your group "xenu-usergroup" on the web.

        b.. To unsubscribe from this group, send an email to:
        xenu-usergroup-unsubscribe@yahoogroups.com

        c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


        ------------------------------------------------------------------------------



        [Non-text portions of this message have been removed]
      • Tilman Hausherr
        ... http://home.snafu.de/tilman/xenulink.html See Nr. 20
        Message 3 of 6 , Jul 7 8:13 AM
        • 0 Attachment
          On Thu, 07 Jul 2005 13:04:39 -0000, frank visser wrote:

          >do you know why Wikipedia links of the kind:
          >http://en.wikipedia.org/wiki/Alfred_North_Whitehead
          >
          >are not working for xenu?

          http://home.snafu.de/tilman/xenulink.html

          See Nr. 20
        • Ken Lorimer
          This is an excellent summary of Professor Whitehead. His work with Bertrand Russel on the theory of classes was a breakthrough at that time. Both men are of
          Message 4 of 6 , Jul 14 9:19 PM
          • 0 Attachment
            This is an excellent summary of Professor Whitehead. His work with Bertrand Russel on the theory of classes was a breakthrough at that time. Both men are of high intellect. I studied their writings at university and I am pleased that you cited them.
            Thank you.

            Ken Lorimer.

            Tilman Hausherr <tilman@...> wrote:
            On Thu, 07 Jul 2005 13:04:39 -0000, frank visser wrote:

            >do you know why Wikipedia links of the kind:
            >http://en.wikipedia.org/wiki/Alfred_North_Whitehead
            >
            >are not working for xenu?

            http://home.snafu.de/tilman/xenulink.html

            See Nr. 20





            ---------------------------------
            YAHOO! GROUPS LINKS


            Visit your group "xenu-usergroup" on the web.

            To unsubscribe from this group, send an email to:
            xenu-usergroup-unsubscribe@yahoogroups.com

            Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


            ---------------------------------




            ---------------------------------
            Start your day with Yahoo! - make it your home page

            [Non-text portions of this message have been removed]
          • adiscretemale
            Andi, I checked your page and noted that you say There is one major company URL (http://www.prudential.com/) that will recycle through dozens of redirects for
            Message 5 of 6 , Jul 25 2:57 AM
            • 0 Attachment
              Andi, I checked your page and noted that you say "There is one major
              company URL (http://www.prudential.com/) that will recycle through
              dozens of redirects for Xenu yet behaves normally for a browser so I
              simply omit it as well." Out of curiousity, I tried starting with
              that URL with Xenu and it only went through 3 redirects before
              actually going into the site. The redirects appear to be the same as
              those a browser gets since if you enter http://www.prudential.com/ in
              browser, when the page comes up, the URL shows as
              http://www.prudential.com/index/.

              Investigating further, I determined that
              http://www.prudential.com/
              first redirects to
              http://www.prudential.com/default/
              that, in turn, redirects to
              http://www.prudential.com/index
              and that, again, redirects to
              http://www.prudential.com/index/
              and that returns a normal page.

              Tom
              --- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
              > I've long had this same experience with Wikipedia using other bots
              as well as Xenu. (returns "Wikimedia error") I have always assumed
              that Wikipedia blocks anything that even slightly smells like a bot
              to keep their content from being spidered. Irresponsible bots are
              becoming so numerous that I'd like to do this on my site as well but
              such blocking is more difficult with static pages. I do find that my
              list of sites seeming to block Xenu continually grows (partial list
              here http://www.andilinks.com/linkckg.htm#55000).
              >
              > Andi
            • andi
              Thanks for this Tom. When I wrote that (It has been over a year) and put Prudential on the exeception list it was returning dozens of redirects to the Xenu
              Message 6 of 6 , Jul 26 7:02 AM
              • 0 Attachment
                Thanks for this Tom. When I wrote that (It has been over a year) and put Prudential on the exeception list it was returning dozens of redirects to the Xenu result page. They must have fixed the bug in their rewrite rule (or whatever, perhaps there's a change in the Xenu version) in the meantime. I'll revise the text on my Link checking page. Thanks again.

                Andi
                ----- Original Message -----
                From: adiscretemale
                To: xenu-usergroup@yahoogroups.com
                Sent: Monday, July 25, 2005 3:57 AM
                Subject: [xenu-usergroup] Re: Wikipedia links not working for Xenu?


                Andi, I checked your page and noted that you say "There is one major
                company URL (http://www.prudential.com/) that will recycle through
                dozens of redirects for Xenu yet behaves normally for a browser so I
                simply omit it as well." Out of curiousity, I tried starting with
                that URL with Xenu and it only went through 3 redirects before
                actually going into the site. The redirects appear to be the same as
                those a browser gets since if you enter http://www.prudential.com/ in
                browser, when the page comes up, the URL shows as
                http://www.prudential.com/index/.

                Investigating further, I determined that
                http://www.prudential.com/
                first redirects to
                http://www.prudential.com/default/
                that, in turn, redirects to
                http://www.prudential.com/index
                and that, again, redirects to
                http://www.prudential.com/index/
                and that returns a normal page.

                Tom
                --- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
                > I've long had this same experience with Wikipedia using other bots
                as well as Xenu. (returns "Wikimedia error") I have always assumed
                that Wikipedia blocks anything that even slightly smells like a bot
                to keep their content from being spidered. Irresponsible bots are
                becoming so numerous that I'd like to do this on my site as well but
                such blocking is more difficult with static pages. I do find that my
                list of sites seeming to block Xenu continually grows (partial list
                here http://www.andilinks.com/linkckg.htm#55000).
                >
                > Andi





                ------------------------------------------------------------------------------
                YAHOO! GROUPS LINKS

                a.. Visit your group "xenu-usergroup" on the web.

                b.. To unsubscribe from this group, send an email to:
                xenu-usergroup-unsubscribe@yahoogroups.com

                c.. Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


                ------------------------------------------------------------------------------



                [Non-text portions of this message have been removed]
              Your message has been successfully submitted and would be delivered to recipients shortly.