Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] Spidering stops after a few links

Expand Messages
  • Tilman Hausherr
    ... Yes, somewhat. So my advice is that you contact the server admin to ask him if there s some maximum for a certain user agent. Tilman
    Message 1 of 7 , May 16, 2005
    • 0 Attachment
      On Mon, 16 May 2005 10:20:12 +0200, Frank Visser wrote:

      >After about 600 URLs the remaining ones remain pending.
      >
      >
      >
      >When I open one of these URLs in a browser, they open almost immediately.
      >
      >
      >
      >Does that contradict your idea of network problems/server overload?

      Yes, somewhat. So my advice is that you contact the server admin to ask
      him if there's some maximum for a certain user agent.

      Tilman
    • frank visser
      hi tilman, looks like i found the solution! reading your FAQ page under time outs there was a passage about enabling both HTTP 1.1 and HTTP 1.1 under proxy
      Message 2 of 7 , May 29, 2005
      • 0 Attachment
        hi tilman,

        looks like i found the solution!

        reading your FAQ page under "time outs" there was a passage about
        enabling both HTTP 1.1 and HTTP 1.1 under proxy conditions.

        on my PC only "HTTP 1.1" was enabled.

        i enabled HTTP 1.1 under proxies, nothing worked.

        however, frustrated, I DISABLED both HTTP 1.1 and HTTP 1.1 under
        proxy... and: the xenu reports started running again as they did in
        the past, with no freezing after a few links.

        I am using WindowsXP on a PentiumIII PC. I have this PC since a few
        months, and most of my earlier reports were run on Win95/PentiumI.

        not sure if my memory is correct, but i could run xenu on the new PC
        for a while.

        anyway.

        Can you explain why disabling HTTP 1.1 this solves the problem of
        xenu freeze?

        Are there any consequences of disabling HTTP 1.1 I should know?

        frank




        --- In xenu-usergroup@yahoogroups.com, Tilman Hausherr <tilman@s...>
        wrote:
        > On Mon, 16 May 2005 10:20:12 +0200, Frank Visser wrote:
        >
        > >After about 600 URLs the remaining ones remain pending.
        > >
        > >
        > >
        > >When I open one of these URLs in a browser, they open almost
        immediately.
        > >
        > >
        > >
        > >Does that contradict your idea of network problems/server
        overload?
        >
        > Yes, somewhat. So my advice is that you contact the server admin
        to ask
        > him if there's some maximum for a certain user agent.
        >
        > Tilman
      • Tilman Hausherr
        ... I had the same problem when you asked me. But it s only on that intel site. ... In theory, everything should be *slower*. No idea why it works. As I said -
        Message 3 of 7 , May 29, 2005
        • 0 Attachment
          On Sun, 29 May 2005 17:42:42 -0000, frank visser wrote:

          >hi tilman,
          >
          >looks like i found the solution!
          >
          >reading your FAQ page under "time outs" there was a passage about
          >enabling both HTTP 1.1 and HTTP 1.1 under proxy conditions.
          >
          >on my PC only "HTTP 1.1" was enabled.
          >
          >i enabled HTTP 1.1 under proxies, nothing worked.
          >
          >however, frustrated, I DISABLED both HTTP 1.1 and HTTP 1.1 under
          >proxy... and: the xenu reports started running again as they did in
          >the past, with no freezing after a few links.
          >
          >I am using WindowsXP on a PentiumIII PC. I have this PC since a few
          >months, and most of my earlier reports were run on Win95/PentiumI.

          I had the same problem when you asked me. But it's only on that intel
          site.

          >not sure if my memory is correct, but i could run xenu on the new PC
          >for a while.
          >
          >anyway.
          >
          >Can you explain why disabling HTTP 1.1 this solves the problem of
          >xenu freeze?
          >
          >Are there any consequences of disabling HTTP 1.1 I should know?

          In theory, everything should be *slower*.

          No idea why it works. As I said - ask the sysadmin.

          And consider buying an up-to-date computer, or does intel pay you so
          badly? Some day, Xenu will no longer work on Windows 95. Because
          according to the description for Microsoft Visual C++ 2003 .net,
          compiled programs work only on W98 and higher.

          For 500 Euro, you can get a good office computer with a Intel Celeron or
          AMD Sempron CPU.

          Tilman
        Your message has been successfully submitted and would be delivered to recipients shortly.