Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] broken link: Reset

Expand Messages
  • Marisa Ciceran
    Chris, We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain
    Message 1 of 10 , Jul 29 2:53 AM
    • 0 Attachment
      Chris,

      We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

      Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

      Thanks,

      Marisa

      Shiner wrote:
      It could be that the webhost is limiting the number of active connections permitted from a single IP.
      Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
      Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

      YMMV

      Chris

      On 15/07/2012 10:38, Tilman Hausherr wrote:
       

      It might work if you retry (CTRL-R), this is a network problem:
      http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

      I tried to access that page myself with a browser, and only a part of it
      was displayed the first time.

      Tilman

      Am 15.07.2012 07:42, schrieb kanzlei_poveda:
      > Hi.
      >
      > I have been a happy user of xenu for some years now.
      > There is only oneerror message, I hvae not been able tofix for a while:
      > Some external links get the following:
      >
      > Broken links, ordered by link:
      >
      > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
      > error code: 12031 (connection reset), linked from page(s):
      > http://www.ra-poveda.de/English/Links_Immigration.html
      >
      > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
      >
      > Thanks.
      >
      >
      >
      >
      >
      > ------------------------------------
      >
      > Yahoo! Groups Links
      >
      >
      >




    • Tilman Hausherr
      Hi Marias, Yes, this is what he suggested. Run Xenu from a fixed IP address (maybe you already have one? Go to http://whatismyipaddress.com/ several days and
      Message 2 of 10 , Jul 29 3:23 AM
      • 0 Attachment
        Hi Marias,

        Yes, this is what he suggested. Run Xenu from a fixed IP address (maybe you already have one? Go to http://whatismyipaddress.com/ several days and see if it changes) and tell tech support to allow unlimited requests from that IP.

        If they don't - try a desperate measure: set the thread count to 1. And be very, very patient. There are broken links, so you'll already have work to do before its done.

        Tilman

        Am 29.07.2012 11:53, schrieb Marisa Ciceran:
        Chris,

        We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

        Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

        Thanks,

        Marisa

        Shiner wrote:
        It could be that the webhost is limiting the number of active connections permitted from a single IP.
        Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
        Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

        YMMV

        Chris

        On 15/07/2012 10:38, Tilman Hausherr wrote:
         

        It might work if you retry (CTRL-R), this is a network problem:
        http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

        I tried to access that page myself with a browser, and only a part of it
        was displayed the first time.

        Tilman

        Am 15.07.2012 07:42, schrieb kanzlei_poveda:
        > Hi.
        >
        > I have been a happy user of xenu for some years now.
        > There is only oneerror message, I hvae not been able tofix for a while:
        > Some external links get the following:
        >
        > Broken links, ordered by link:
        >
        > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
        > error code: 12031 (connection reset), linked from page(s):
        > http://www.ra-poveda.de/English/Links_Immigration.html
        >
        > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
        >
        > Thanks.
        >
        >
        >
        >
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >





      • Shiner
        Marisa, The /firewall may be blocking my IP address for excessive request/ is the same thing essentially. When you run Xenu, go to the options panel and
        Message 3 of 10 , Jul 29 3:31 AM
        • 0 Attachment
          Marisa,

          The "firewall may be blocking my IP address for "excessive request" is the same thing essentially.

          When you run Xenu, go to the options panel and under the "basic" tab, you will see in the top left hand corner of that panel, a "Parallel Threads" section.
          Here you can set it between 1-100.

          This is number of threads that Xenu uses to connect with the remote server.
          If you set it to 100, then essentially this means that Xenu will open up 100 connections from your IP address to your website.

          Whereas your web server can handle many hundreds if not thousands of connections at the same time, some servers restrict the number of connections from a single IP address.

          Try setting it 10, and run Xenu again.. If no errors, increase to 30.
          If you get errors, then decrease to 20.
          If no errors, increase to 25, if errors, decrease to 15..
          i.e. keep doing this until you find a value that works consistently..

          (The fastest way to find the optimum number is actually to set it to 50, and then half the difference each time it passes or fails).  i.e. if no errors, go to 75, if errors go to 25. Now run again, if errors, then drop or increase by 12, next time by 6, then by 3, then by 1.  I usually start at 10 only because most web servers on shared hosting plans tend to have a low limit on the number of connections, and it's very rarely near 50).
          This is called the "Half Split Method" by the way...

          Obviously, when set to 100, Xenu will scan a website quicker than if set to 1.
          Likewise, if set to 1, it will still complete a scan but a lot slower (if a large site)...

          You can also try setting Xenu to run at a "quiet time" for your web host (i.e. between 2am and 6am), but remember it will be the time zone of where your server is based and not necessarily the time zone where you are based.

          If you can set Xenu to run 'overnight', then it doesn't matter if you set the number of threads to '1' as hopefully when you wake up in the morning, it should have completed (unless it's some humongous web site).

          So in summary, it has nothing to do with the speed of your connection (i.e. whether you are on dial up at 9600bps or a fibre connection at Gigabits per second, it's down the number of connections your Xenu/local machine is trying to make to the remote (web) server in parallel (at the same time).  So what you have to do is find a value that your web server is reliably happy with.

          Give me a shout if I've still been too technical, or you are having problems setting it up.

          Regards

          Chris



          On 29/07/2012 10:53, Marisa Ciceran wrote:
           

          Chris,

          We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

          Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

          Thanks,

          Marisa

          Shiner wrote:
          It could be that the webhost is limiting the number of active connections permitted from a single IP.
          Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
          Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

          YMMV

          Chris

          On 15/07/2012 10:38, Tilman Hausherr wrote:
           

          It might work if you retry (CTRL-R), this is a network problem:
          http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

          I tried to access that page myself with a browser, and only a part of it
          was displayed the first time.

          Tilman

          Am 15.07.2012 07:42, schrieb kanzlei_poveda:
          > Hi.
          >
          > I have been a happy user of xenu for some years now.
          > There is only oneerror message, I hvae not been able tofix for a while:
          > Some external links get the following:
          >
          > Broken links, ordered by link:
          >
          > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
          > error code: 12031 (connection reset), linked from page(s):
          > http://www.ra-poveda.de/English/Links_Immigration.html
          >
          > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
          >
          > Thanks.
          >
          >
          >
          >
          >
          > ------------------------------------
          >
          > Yahoo! Groups Links
          >
          >
          >





        • Shiner
          ... I ve just noticed that you are with HostGator, and you should be able to set the number of threads to 25 and definitely not have any errors. (Any higher
          Message 4 of 10 , Jul 29 3:40 AM
          • 0 Attachment
            On 29/07/2012 10:53, Marisa Ciceran wrote:
            > We changed our host server for our website http://www.istrianet.org
            > last month

            I've just noticed that you are with HostGator, and you should be able to
            set the number of threads to 25 and definitely not have any errors.
            (Any higher than 25 and you may see connection errors).

            Chris
          • Marisa Ciceran
            Chris, ... So far in my exchanges with their tech support, all they come up with is instructing me to read their rules of usage, backups, violations and
            Message 5 of 10 , Jul 30 1:44 AM
            • 0 Attachment
              Chris,

              Shiner wrote:
              > On 29/07/2012 10:53, Marisa Ciceran wrote:
              >> We changed our host server for our website http://www.istrianet.org
              >> last month
              > I've just noticed that you are with HostGator, and you should be able to
              > set the number of threads to 25 and definitely not have any errors.
              So far in my exchanges with their tech support, all they come up with is
              instructing me to read their rules of usage, backups, violations and
              restrictions, but avoiding straight answers as to why I experiencing the
              problem with Xenu. We are with them only one month and haven't done
              anything out of order (being a cultural-only site, there is no heavy
              traffic of hits) on their server after the direct transfer of the site
              via cPanel was made by them. All I've done in the past month is try to
              run FTPSync, an old and primitive web synchronization program (I wish I
              could find a newer and still simple alternative) to see if the files
              were in synch with my PC backup and, over the last few days I also tried
              to run Xenu to clear up any broken links - both of these being a
              reasonable alternative, I thought, to their "SWAMP previewing system"
              which is primitive and imprecise and therefore of absolutely no value in
              verifying anything whatsoever of our 11+ Gb site contents.
              > (Any higher than 25 and you may see connection errors).
              Initially, I had it set for 30 and included the check for broken
              external links, then I went down to 20, 10 and finally to 1 and
              excluding external links. Each attempt gave the same results: Xenu
              started off and appeared to be running, then the error I mentioned
              quickly popped in and rendered the results useless. In my latest
              attempts after my posting here, I got more outright "timeout" messages
              along with the error code I previously mentioned.

              Any other thoughts on why HostGator is being so troublesome right from
              the start?

              Marisa
              >
              > Chris
              Thanks for your observations and suggestions. I will try again in a day
              or two to see if perhaps I have been experiencing the month's end
              traffic jams on HostGator server.

              Marisa
              >
              >
              > ------------------------------------
              >
              > Yahoo! Groups Links
              >
              >
              >
              >
            • Tilman Hausherr
              I scanned her whole site this WE with threads set to 1, later 2. That took many hours, but it worked without trouble. Theories about this: - Maybe if one
              Message 6 of 10 , Jul 30 1:59 AM
              • 0 Attachment
                I scanned her whole site this WE with threads set to 1, later 2. That
                took many hours, but it worked without trouble.

                Theories about this:
                - Maybe if one starts with "too many" threads, the client IP address is
                "penalized" for some time, so it doesn't help lowering threads after
                being "flagged" by the server firewall
                - A change on her client side (many people do have a firewall in their
                router; so do I, and initially it blocked Xenu from running)
                - sysadmins are too often underqualified, underpaid and understaffed.
                What you get then are people who just try to find a way to quickly close
                tickets, instead of taking some time to research a problem.

                Tilman

                Am 30.07.2012 10:44, schrieb Marisa Ciceran:
                > Chris,
                >
                > Shiner wrote:
                >> On 29/07/2012 10:53, Marisa Ciceran wrote:
                >>> We changed our host server for our website http://www.istrianet.org
                >>> last month
                >> I've just noticed that you are with HostGator, and you should be able to
                >> set the number of threads to 25 and definitely not have any errors.
                > So far in my exchanges with their tech support, all they come up with is
                > instructing me to read their rules of usage, backups, violations and
                > restrictions, but avoiding straight answers as to why I experiencing the
                > problem with Xenu. We are with them only one month and haven't done
                > anything out of order (being a cultural-only site, there is no heavy
                > traffic of hits) on their server after the direct transfer of the site
                > via cPanel was made by them. All I've done in the past month is try to
                > run FTPSync, an old and primitive web synchronization program (I wish I
                > could find a newer and still simple alternative) to see if the files
                > were in synch with my PC backup and, over the last few days I also tried
                > to run Xenu to clear up any broken links - both of these being a
                > reasonable alternative, I thought, to their "SWAMP previewing system"
                > which is primitive and imprecise and therefore of absolutely no value in
                > verifying anything whatsoever of our 11+ Gb site contents.
                >> (Any higher than 25 and you may see connection errors).
                > Initially, I had it set for 30 and included the check for broken
                > external links, then I went down to 20, 10 and finally to 1 and
                > excluding external links. Each attempt gave the same results: Xenu
                > started off and appeared to be running, then the error I mentioned
                > quickly popped in and rendered the results useless. In my latest
                > attempts after my posting here, I got more outright "timeout" messages
                > along with the error code I previously mentioned.
                >
                > Any other thoughts on why HostGator is being so troublesome right from
                > the start?
                >
                > Marisa
                >> Chris
                > Thanks for your observations and suggestions. I will try again in a day
                > or two to see if perhaps I have been experiencing the month's end
                > traffic jams on HostGator server.
                >
                > Marisa
                >>
                >> ------------------------------------
                >>
                >> Yahoo! Groups Links
                >>
                >>
                >>
                >>
                >
                >
                > ------------------------------------
                >
                > Yahoo! Groups Links
                >
                >
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.