Loading ...
Sorry, an error occurred while loading the content.

RE: [xenu-usergroup] Spidering stops after a few links

Expand Messages
  • Frank Visser
    Hi tilman, Thanks for trying. For the record: i had these settings: - 10 threads - 999 levels - No excluded URLs ok 627 URLs 78.18%
    Message 1 of 7 , May 16, 2005
    • 0 Attachment
      Hi tilman,



      Thanks for trying.



      For the record: i had these settings:

      - 10 threads

      - 999 levels

      - No excluded URLs




      ok

      627 URLs

      78.18%


      pending

      141 URLs

      17.58%


      cancelled / timeout

      17 URLs

      2.12%


      skip type

      5 URLs

      0.62%


      busy

      10 URLs

      1.25%


      server error

      2 URLs

      0.25%









      Total

      802 URLs

      100.00%



      After about 600 URLs the remaining ones remain pending.



      When I open one of these URLs in a browser, they open almost immediately.



      Does that contradict your idea of network problems/server overload?



      frank



      ========================================

      Frank Visser, Waterpoortweg 279, 1051 pv, Amsterdam

      Author of: Ken Wilber: Thought as Passion SUNY 2003

      Read all about Ken Wilber : http:// <http://www.integralworld.net>
      www.integralworld.net

      ========================================

      _____

      From: xenu-usergroup@yahoogroups.com [mailto:xenu-usergroup@yahoogroups.com]
      On Behalf Of Tilman Hausherr
      Sent: zondag 15 mei 2005 17:52
      To: xenu-usergroup@yahoogroups.com
      Subject: Re: [xenu-usergroup] Spidering stops after a few links



      On Sun, 15 May 2005 11:25:31 -0000, frank visser wrote:

      >hi tilman,
      >
      >when i try to spider this URL:
      >http://www.intel.com/jp/products/index.htm
      >
      >the spidering stops after a few links.
      >
      >i haven't seen this happen before.
      >
      >could it mean the intel.com site has blocked xenu? (or would that
      >prevent even a few links to be spidered?)
      >
      >if so, is there a way to find out if this really is the case?
      >
      >sometimes i also see this happen on a PC, but not on another on the
      >same LAN, so then it's just a local issue.
      >
      >this time, I tried several other PCs, but the spidering stops.
      >
      >i use the latest official version of xenu, 1.2g
      >
      >thanks for any help on this,

      You should really give more details. What max level do you have, what
      URLs did you exclude, how many URLs did it check?



      _____

      Yahoo! Groups Links

      * To visit your group on the web, go to:
      http://groups.yahoo.com/group/xenu-usergroup/

      * To unsubscribe from this group, send an email to:
      xenu-usergroup-unsubscribe@yahoogroups.com
      <mailto:xenu-usergroup-unsubscribe@yahoogroups.com?subject=Unsubscribe>

      * Your use of Yahoo! Groups is subject to the Yahoo!
      <http://docs.yahoo.com/info/terms/> Terms of Service.



      [Non-text portions of this message have been removed]
    • Tilman Hausherr
      ... Yes, somewhat. So my advice is that you contact the server admin to ask him if there s some maximum for a certain user agent. Tilman
      Message 2 of 7 , May 16, 2005
      • 0 Attachment
        On Mon, 16 May 2005 10:20:12 +0200, Frank Visser wrote:

        >After about 600 URLs the remaining ones remain pending.
        >
        >
        >
        >When I open one of these URLs in a browser, they open almost immediately.
        >
        >
        >
        >Does that contradict your idea of network problems/server overload?

        Yes, somewhat. So my advice is that you contact the server admin to ask
        him if there's some maximum for a certain user agent.

        Tilman
      • frank visser
        hi tilman, looks like i found the solution! reading your FAQ page under time outs there was a passage about enabling both HTTP 1.1 and HTTP 1.1 under proxy
        Message 3 of 7 , May 29, 2005
        • 0 Attachment
          hi tilman,

          looks like i found the solution!

          reading your FAQ page under "time outs" there was a passage about
          enabling both HTTP 1.1 and HTTP 1.1 under proxy conditions.

          on my PC only "HTTP 1.1" was enabled.

          i enabled HTTP 1.1 under proxies, nothing worked.

          however, frustrated, I DISABLED both HTTP 1.1 and HTTP 1.1 under
          proxy... and: the xenu reports started running again as they did in
          the past, with no freezing after a few links.

          I am using WindowsXP on a PentiumIII PC. I have this PC since a few
          months, and most of my earlier reports were run on Win95/PentiumI.

          not sure if my memory is correct, but i could run xenu on the new PC
          for a while.

          anyway.

          Can you explain why disabling HTTP 1.1 this solves the problem of
          xenu freeze?

          Are there any consequences of disabling HTTP 1.1 I should know?

          frank




          --- In xenu-usergroup@yahoogroups.com, Tilman Hausherr <tilman@s...>
          wrote:
          > On Mon, 16 May 2005 10:20:12 +0200, Frank Visser wrote:
          >
          > >After about 600 URLs the remaining ones remain pending.
          > >
          > >
          > >
          > >When I open one of these URLs in a browser, they open almost
          immediately.
          > >
          > >
          > >
          > >Does that contradict your idea of network problems/server
          overload?
          >
          > Yes, somewhat. So my advice is that you contact the server admin
          to ask
          > him if there's some maximum for a certain user agent.
          >
          > Tilman
        • Tilman Hausherr
          ... I had the same problem when you asked me. But it s only on that intel site. ... In theory, everything should be *slower*. No idea why it works. As I said -
          Message 4 of 7 , May 29, 2005
          • 0 Attachment
            On Sun, 29 May 2005 17:42:42 -0000, frank visser wrote:

            >hi tilman,
            >
            >looks like i found the solution!
            >
            >reading your FAQ page under "time outs" there was a passage about
            >enabling both HTTP 1.1 and HTTP 1.1 under proxy conditions.
            >
            >on my PC only "HTTP 1.1" was enabled.
            >
            >i enabled HTTP 1.1 under proxies, nothing worked.
            >
            >however, frustrated, I DISABLED both HTTP 1.1 and HTTP 1.1 under
            >proxy... and: the xenu reports started running again as they did in
            >the past, with no freezing after a few links.
            >
            >I am using WindowsXP on a PentiumIII PC. I have this PC since a few
            >months, and most of my earlier reports were run on Win95/PentiumI.

            I had the same problem when you asked me. But it's only on that intel
            site.

            >not sure if my memory is correct, but i could run xenu on the new PC
            >for a while.
            >
            >anyway.
            >
            >Can you explain why disabling HTTP 1.1 this solves the problem of
            >xenu freeze?
            >
            >Are there any consequences of disabling HTTP 1.1 I should know?

            In theory, everything should be *slower*.

            No idea why it works. As I said - ask the sysadmin.

            And consider buying an up-to-date computer, or does intel pay you so
            badly? Some day, Xenu will no longer work on Windows 95. Because
            according to the description for Microsoft Visual C++ 2003 .net,
            compiled programs work only on W98 and higher.

            For 500 Euro, you can get a good office computer with a Intel Celeron or
            AMD Sempron CPU.

            Tilman
          Your message has been successfully submitted and would be delivered to recipients shortly.