Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] broken link: Reset

Expand Messages
  • Shiner
    It could be that the webhost is limiting the number of active connections permitted from a single IP. Might be worth checking the number of threads that they
    Message 1 of 10 , Jul 15, 2012
    • 0 Attachment
      It could be that the webhost is limiting the number of active connections permitted from a single IP.
      Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
      Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

      YMMV

      Chris

      On 15/07/2012 10:38, Tilman Hausherr wrote:
       

      It might work if you retry (CTRL-R), this is a network problem:
      http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

      I tried to access that page myself with a browser, and only a part of it
      was displayed the first time.

      Tilman

      Am 15.07.2012 07:42, schrieb kanzlei_poveda:
      > Hi.
      >
      > I have been a happy user of xenu for some years now.
      > There is only oneerror message, I hvae not been able tofix for a while:
      > Some external links get the following:
      >
      > Broken links, ordered by link:
      >
      > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
      > error code: 12031 (connection reset), linked from page(s):
      > http://www.ra-poveda.de/English/Links_Immigration.html
      >
      > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
      >
      > Thanks.
      >
      >
      >
      >
      >
      > ------------------------------------
      >
      > Yahoo! Groups Links
      >
      >
      >



    • kanzlei_poveda
      Thanks Chris, thanks Tilman.
      Message 2 of 10 , Jul 16, 2012
      • 0 Attachment
        Thanks Chris, thanks Tilman.


        --- In xenu-usergroup@yahoogroups.com, Shiner <shinerweb@...> wrote:
        >
        > It could be that the webhost is limiting the number of active
        > connections permitted from a single IP.
        > Might be worth checking the number of threads that they have set up in
        > Xenu and lowering it to a value at which the timeouts do not occur.
        > Some web hosts do not terminate the connection gracefully and it leaves
        > the connection hanging until such time it times out (they have to do
        > that to avoid DoS attacks, so these days, rather then reject the
        > connection, it just ignores it which leaves the visitor hanging on until
        > their local machine times out).
        >
        > YMMV
        >
        > Chris
        >
        > On 15/07/2012 10:38, Tilman Hausherr wrote:
        > >
        > >
        > > It might work if you retry (CTRL-R), this is a network problem:
        > > http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html
        > >
        > > I tried to access that page myself with a browser, and only a part of it
        > > was displayed the first time.
        > >
        > > Tilman
        > >
        > > Am 15.07.2012 07:42, schrieb kanzlei_poveda:
        > > > Hi.
        > > >
        > > > I have been a happy user of xenu for some years now.
        > > > There is only oneerror message, I hvae not been able tofix for a while:
        > > > Some external links get the following:
        > > >
        > > > Broken links, ordered by link:
        > > >
        > > >
        > > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
        > > > error code: 12031 (connection reset), linked from page(s):
        > > > http://www.ra-poveda.de/English/Links_Immigration.html
        > > >
        > > > I check the links, they work fine.I tried to use references from the
        > > external sites that also link to these pages, and they also generate
        > > the same error messages. Is there any possibility I can get rid of
        > > those messages?
        > > >
        > > > Thanks.
        > > >
        > > >
        > > >
        > > >
        > > >
        > > > ------------------------------------
        > > >
        > > > Yahoo! Groups Links
        > > >
        > > >
        > > >
        > >
        > >
        >
      • Marisa Ciceran
        Chris, We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain
        Message 3 of 10 , Jul 29, 2012
        • 0 Attachment
          Chris,

          We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

          Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

          Thanks,

          Marisa

          Shiner wrote:
          It could be that the webhost is limiting the number of active connections permitted from a single IP.
          Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
          Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

          YMMV

          Chris

          On 15/07/2012 10:38, Tilman Hausherr wrote:
           

          It might work if you retry (CTRL-R), this is a network problem:
          http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

          I tried to access that page myself with a browser, and only a part of it
          was displayed the first time.

          Tilman

          Am 15.07.2012 07:42, schrieb kanzlei_poveda:
          > Hi.
          >
          > I have been a happy user of xenu for some years now.
          > There is only oneerror message, I hvae not been able tofix for a while:
          > Some external links get the following:
          >
          > Broken links, ordered by link:
          >
          > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
          > error code: 12031 (connection reset), linked from page(s):
          > http://www.ra-poveda.de/English/Links_Immigration.html
          >
          > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
          >
          > Thanks.
          >
          >
          >
          >
          >
          > ------------------------------------
          >
          > Yahoo! Groups Links
          >
          >
          >




        • Tilman Hausherr
          Hi Marias, Yes, this is what he suggested. Run Xenu from a fixed IP address (maybe you already have one? Go to http://whatismyipaddress.com/ several days and
          Message 4 of 10 , Jul 29, 2012
          • 0 Attachment
            Hi Marias,

            Yes, this is what he suggested. Run Xenu from a fixed IP address (maybe you already have one? Go to http://whatismyipaddress.com/ several days and see if it changes) and tell tech support to allow unlimited requests from that IP.

            If they don't - try a desperate measure: set the thread count to 1. And be very, very patient. There are broken links, so you'll already have work to do before its done.

            Tilman

            Am 29.07.2012 11:53, schrieb Marisa Ciceran:
            Chris,

            We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

            Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

            Thanks,

            Marisa

            Shiner wrote:
            It could be that the webhost is limiting the number of active connections permitted from a single IP.
            Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
            Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

            YMMV

            Chris

            On 15/07/2012 10:38, Tilman Hausherr wrote:
             

            It might work if you retry (CTRL-R), this is a network problem:
            http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

            I tried to access that page myself with a browser, and only a part of it
            was displayed the first time.

            Tilman

            Am 15.07.2012 07:42, schrieb kanzlei_poveda:
            > Hi.
            >
            > I have been a happy user of xenu for some years now.
            > There is only oneerror message, I hvae not been able tofix for a while:
            > Some external links get the following:
            >
            > Broken links, ordered by link:
            >
            > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
            > error code: 12031 (connection reset), linked from page(s):
            > http://www.ra-poveda.de/English/Links_Immigration.html
            >
            > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
            >
            > Thanks.
            >
            >
            >
            >
            >
            > ------------------------------------
            >
            > Yahoo! Groups Links
            >
            >
            >





          • Shiner
            Marisa, The /firewall may be blocking my IP address for excessive request/ is the same thing essentially. When you run Xenu, go to the options panel and
            Message 5 of 10 , Jul 29, 2012
            • 0 Attachment
              Marisa,

              The "firewall may be blocking my IP address for "excessive request" is the same thing essentially.

              When you run Xenu, go to the options panel and under the "basic" tab, you will see in the top left hand corner of that panel, a "Parallel Threads" section.
              Here you can set it between 1-100.

              This is number of threads that Xenu uses to connect with the remote server.
              If you set it to 100, then essentially this means that Xenu will open up 100 connections from your IP address to your website.

              Whereas your web server can handle many hundreds if not thousands of connections at the same time, some servers restrict the number of connections from a single IP address.

              Try setting it 10, and run Xenu again.. If no errors, increase to 30.
              If you get errors, then decrease to 20.
              If no errors, increase to 25, if errors, decrease to 15..
              i.e. keep doing this until you find a value that works consistently..

              (The fastest way to find the optimum number is actually to set it to 50, and then half the difference each time it passes or fails).  i.e. if no errors, go to 75, if errors go to 25. Now run again, if errors, then drop or increase by 12, next time by 6, then by 3, then by 1.  I usually start at 10 only because most web servers on shared hosting plans tend to have a low limit on the number of connections, and it's very rarely near 50).
              This is called the "Half Split Method" by the way...

              Obviously, when set to 100, Xenu will scan a website quicker than if set to 1.
              Likewise, if set to 1, it will still complete a scan but a lot slower (if a large site)...

              You can also try setting Xenu to run at a "quiet time" for your web host (i.e. between 2am and 6am), but remember it will be the time zone of where your server is based and not necessarily the time zone where you are based.

              If you can set Xenu to run 'overnight', then it doesn't matter if you set the number of threads to '1' as hopefully when you wake up in the morning, it should have completed (unless it's some humongous web site).

              So in summary, it has nothing to do with the speed of your connection (i.e. whether you are on dial up at 9600bps or a fibre connection at Gigabits per second, it's down the number of connections your Xenu/local machine is trying to make to the remote (web) server in parallel (at the same time).  So what you have to do is find a value that your web server is reliably happy with.

              Give me a shout if I've still been too technical, or you are having problems setting it up.

              Regards

              Chris



              On 29/07/2012 10:53, Marisa Ciceran wrote:
               

              Chris,

              We changed our host server for our website http://www.istrianet.org last month and, so far this year and especially in the last month since our domain transfer, running XENU has become a major ordeal. Apart from openly timing out, we currently are getting error 12151 (invalid response) that essentially aborts the program from the start. The only response I have gotten this morning from our server technical support folks is that their firewall may be blocking my IP address for "excessive request".

              Is this the same issue as you just suggested? I am not a techie so have no clue as to what this error code actually means.

              Thanks,

              Marisa

              Shiner wrote:
              It could be that the webhost is limiting the number of active connections permitted from a single IP.
              Might be worth checking the number of threads that they have set up in Xenu and lowering it to a value at which the timeouts do not occur.
              Some web hosts do not terminate the connection gracefully and it leaves the connection hanging until such time it times out (they have to do that to avoid DoS attacks, so these days, rather then reject the connection, it just ignores it which leaves the visitor hanging on until their local machine times out).

              YMMV

              Chris

              On 15/07/2012 10:38, Tilman Hausherr wrote:
               

              It might work if you retry (CTRL-R), this is a network problem:
              http://www.howtofixcomputers.com/forums/xp-networking/http-error-12031-a-206361.html

              I tried to access that page myself with a browser, and only a part of it
              was displayed the first time.

              Tilman

              Am 15.07.2012 07:42, schrieb kanzlei_poveda:
              > Hi.
              >
              > I have been a happy user of xenu for some years now.
              > There is only oneerror message, I hvae not been able tofix for a while:
              > Some external links get the following:
              >
              > Broken links, ordered by link:
              >
              > http://www.frankfurt.de/sixcms/detail.php?id=317595&_ffmpar[_id_inhalt]=7846492
              > error code: 12031 (connection reset), linked from page(s):
              > http://www.ra-poveda.de/English/Links_Immigration.html
              >
              > I check the links, they work fine.I tried to use references from the external sites that also link to these pages, and they also generate the same error messages. Is there any possibility I can get rid of those messages?
              >
              > Thanks.
              >
              >
              >
              >
              >
              > ------------------------------------
              >
              > Yahoo! Groups Links
              >
              >
              >





            • Shiner
              ... I ve just noticed that you are with HostGator, and you should be able to set the number of threads to 25 and definitely not have any errors. (Any higher
              Message 6 of 10 , Jul 29, 2012
              • 0 Attachment
                On 29/07/2012 10:53, Marisa Ciceran wrote:
                > We changed our host server for our website http://www.istrianet.org
                > last month

                I've just noticed that you are with HostGator, and you should be able to
                set the number of threads to 25 and definitely not have any errors.
                (Any higher than 25 and you may see connection errors).

                Chris
              • Marisa Ciceran
                Chris, ... So far in my exchanges with their tech support, all they come up with is instructing me to read their rules of usage, backups, violations and
                Message 7 of 10 , Jul 30, 2012
                • 0 Attachment
                  Chris,

                  Shiner wrote:
                  > On 29/07/2012 10:53, Marisa Ciceran wrote:
                  >> We changed our host server for our website http://www.istrianet.org
                  >> last month
                  > I've just noticed that you are with HostGator, and you should be able to
                  > set the number of threads to 25 and definitely not have any errors.
                  So far in my exchanges with their tech support, all they come up with is
                  instructing me to read their rules of usage, backups, violations and
                  restrictions, but avoiding straight answers as to why I experiencing the
                  problem with Xenu. We are with them only one month and haven't done
                  anything out of order (being a cultural-only site, there is no heavy
                  traffic of hits) on their server after the direct transfer of the site
                  via cPanel was made by them. All I've done in the past month is try to
                  run FTPSync, an old and primitive web synchronization program (I wish I
                  could find a newer and still simple alternative) to see if the files
                  were in synch with my PC backup and, over the last few days I also tried
                  to run Xenu to clear up any broken links - both of these being a
                  reasonable alternative, I thought, to their "SWAMP previewing system"
                  which is primitive and imprecise and therefore of absolutely no value in
                  verifying anything whatsoever of our 11+ Gb site contents.
                  > (Any higher than 25 and you may see connection errors).
                  Initially, I had it set for 30 and included the check for broken
                  external links, then I went down to 20, 10 and finally to 1 and
                  excluding external links. Each attempt gave the same results: Xenu
                  started off and appeared to be running, then the error I mentioned
                  quickly popped in and rendered the results useless. In my latest
                  attempts after my posting here, I got more outright "timeout" messages
                  along with the error code I previously mentioned.

                  Any other thoughts on why HostGator is being so troublesome right from
                  the start?

                  Marisa
                  >
                  > Chris
                  Thanks for your observations and suggestions. I will try again in a day
                  or two to see if perhaps I have been experiencing the month's end
                  traffic jams on HostGator server.

                  Marisa
                  >
                  >
                  > ------------------------------------
                  >
                  > Yahoo! Groups Links
                  >
                  >
                  >
                  >
                • Tilman Hausherr
                  I scanned her whole site this WE with threads set to 1, later 2. That took many hours, but it worked without trouble. Theories about this: - Maybe if one
                  Message 8 of 10 , Jul 30, 2012
                  • 0 Attachment
                    I scanned her whole site this WE with threads set to 1, later 2. That
                    took many hours, but it worked without trouble.

                    Theories about this:
                    - Maybe if one starts with "too many" threads, the client IP address is
                    "penalized" for some time, so it doesn't help lowering threads after
                    being "flagged" by the server firewall
                    - A change on her client side (many people do have a firewall in their
                    router; so do I, and initially it blocked Xenu from running)
                    - sysadmins are too often underqualified, underpaid and understaffed.
                    What you get then are people who just try to find a way to quickly close
                    tickets, instead of taking some time to research a problem.

                    Tilman

                    Am 30.07.2012 10:44, schrieb Marisa Ciceran:
                    > Chris,
                    >
                    > Shiner wrote:
                    >> On 29/07/2012 10:53, Marisa Ciceran wrote:
                    >>> We changed our host server for our website http://www.istrianet.org
                    >>> last month
                    >> I've just noticed that you are with HostGator, and you should be able to
                    >> set the number of threads to 25 and definitely not have any errors.
                    > So far in my exchanges with their tech support, all they come up with is
                    > instructing me to read their rules of usage, backups, violations and
                    > restrictions, but avoiding straight answers as to why I experiencing the
                    > problem with Xenu. We are with them only one month and haven't done
                    > anything out of order (being a cultural-only site, there is no heavy
                    > traffic of hits) on their server after the direct transfer of the site
                    > via cPanel was made by them. All I've done in the past month is try to
                    > run FTPSync, an old and primitive web synchronization program (I wish I
                    > could find a newer and still simple alternative) to see if the files
                    > were in synch with my PC backup and, over the last few days I also tried
                    > to run Xenu to clear up any broken links - both of these being a
                    > reasonable alternative, I thought, to their "SWAMP previewing system"
                    > which is primitive and imprecise and therefore of absolutely no value in
                    > verifying anything whatsoever of our 11+ Gb site contents.
                    >> (Any higher than 25 and you may see connection errors).
                    > Initially, I had it set for 30 and included the check for broken
                    > external links, then I went down to 20, 10 and finally to 1 and
                    > excluding external links. Each attempt gave the same results: Xenu
                    > started off and appeared to be running, then the error I mentioned
                    > quickly popped in and rendered the results useless. In my latest
                    > attempts after my posting here, I got more outright "timeout" messages
                    > along with the error code I previously mentioned.
                    >
                    > Any other thoughts on why HostGator is being so troublesome right from
                    > the start?
                    >
                    > Marisa
                    >> Chris
                    > Thanks for your observations and suggestions. I will try again in a day
                    > or two to see if perhaps I have been experiencing the month's end
                    > traffic jams on HostGator server.
                    >
                    > Marisa
                    >>
                    >> ------------------------------------
                    >>
                    >> Yahoo! Groups Links
                    >>
                    >>
                    >>
                    >>
                    >
                    >
                    > ------------------------------------
                    >
                    > Yahoo! Groups Links
                    >
                    >
                    >
                  Your message has been successfully submitted and would be delivered to recipients shortly.