Loading ...
Sorry, an error occurred while loading the content.

Not Checking All Links

Expand Messages
  • Michael Hunsaker
    Hello, I have been using Xenu for quite some time on our entire University web site and it works pretty well. We have over 40,000 pages to check and I always
    Message 1 of 10 , Oct 5, 2005
    • 0 Attachment
      Hello,

      I have been using Xenu for quite some time on our
      entire University web site and it works pretty well.
      We have over 40,000 pages to check and I always
      thought Xenu was checking them all until a few weeks
      ago. I implemented a customized 404 error page that
      sent me an email each time a user hit the 404 error
      page. I get about 100 - 200 emails per day telling me
      about broken links.

      Sure, some of the broken links are not "real" broken
      links but a good chunk of them are truly broken links.
      Why isn't Xenu picking these up? I have the maximum
      level at 999; 100 threads; Treat redirections as
      errors; and reporting on Broken links ordered by page.
      I am not restricting any URLs. We do not use that
      much Javascript or Flash - basic HTML for 90% of our
      links

      Sad to say but we are a Microsoft campus so we have
      FrontPage on most of the end-user computers. I've
      found that some "Find and Replace" tools that I have
      used in the past do not look in FrontPage subwebs or
      childwebs. Does Xenu have any difficulty with this?

      Our web site address is http://www.usi.edu Some of
      the broken links that are being skipped occur on all
      levels. I actually received an email from our 404
      page that one of our level 1 links was broken. Xenu
      didn't pick this up. Several of the broken links
      (notified via 404 error email) are deep in the folder
      structure.

      Please let me know if you have any suggestions for me.
      I would appreciate it.

      Mike
    • Tilman Hausherr
      ... Tell me a link that isn t spidered, but it linked from a page that is.
      Message 2 of 10 , Oct 5, 2005
      • 0 Attachment
        On Wed, 5 Oct 2005 06:04:40 -0700 (PDT), Michael Hunsaker wrote:

        >Sure, some of the broken links are not "real" broken
        >links but a good chunk of them are truly broken links.
        > Why isn't Xenu picking these up?

        Tell me a link that isn't spidered, but it linked from a page that is.
      • Michael Hunsaker
        Thanks for your response. Below is a path navigation to a file that does not exist but Xenu is not picking it up: #1 - http://www.usi.edu #2 -
        Message 3 of 10 , Oct 5, 2005
        • 0 Attachment
          Thanks for your response. Below is a path navigation
          to a file that does not exist but Xenu is not picking
          it up:

          #1 - http://www.usi.edu
          #2 - http://www.usi.edu/alumni/index.asp
          #3 - http://www.usi.edu/alumni/connect/index.asp
          #4 - http://www.usi.edu/alumni/connect/wolfe.asp

          #4 is the broken link.

          Here is another example:

          #1 - http://www.usi.edu
          #2 - http://www.usi.edu/science/index.asp
          #3 - http://www.usi.edu/science/about.asp
          #4 - http://www.usi.edu/science/Dean/page2.asp
          #5 -
          http://www.usi.edu/science/Dean/dean/CollegeSeminarSeries.asp

          #5 is the broken link.

          Those are just a couple of examples that I could
          quickly come up with.

          Thanks again,

          Mike


          --- Tilman Hausherr <tilman@...> wrote:

          > On Wed, 5 Oct 2005 06:04:40 -0700 (PDT), Michael
          > Hunsaker wrote:
          >
          > >Sure, some of the broken links are not "real"
          > broken
          > >links but a good chunk of them are truly broken
          > links.
          > > Why isn't Xenu picking these up?
          >
          > Tell me a link that isn't spidered, but it linked
          > from a page that is.
          >
          >
        • Frank Visser
          Hi mike, That s because your server gives back a 200 (=ok) message, and Xenu picks this up: GET /alumni/connect/wolfe.asp HTTP/1.1 Host: www.usi.edu
          Message 4 of 10 , Oct 5, 2005
          • 0 Attachment
            Hi mike,



            That's because your server gives back a 200 (=ok) message, and Xenu picks
            this up:



            GET /alumni/connect/wolfe.asp HTTP/1.1

            Host: www.usi.edu

            User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
            Gecko/20041107 Firefox/1.0

            Accept:
            text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
            0.8,image/png,*/*;q=0.5

            Accept-Language: en-us,en;q=0.5

            Accept-Encoding: gzip,deflate

            Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

            Keep-Alive: 300

            Connection: keep-alive

            Cookie: ASPSESSIONIDQCCTTTAT=FFECGHACMPNAHNMNLHLHGHKO

            Cache-Control: max-age=0



            HTTP/1.x 200 OK

            Date: Wed, 05 Oct 2005 21:03:59 GMT

            Server: Microsoft-IIS/6.0

            X-Powered-By: ASP.NET

            Content-Length: 10174

            Content-Type: text/html

            Cache-Control: private



            It should return a 404 message, so it's a server configuration issue.



            frank



            ========================================

            Frank Visser, Waterpoortweg 279, 1051 pv, Amsterdam

            Author of: Ken Wilber: Thought as Passion SUNY 2003

            Read all about Ken Wilber : http:// <http://www.integralworld.net>
            www.integralworld.net

            ========================================

            _____

            From: xenu-usergroup@yahoogroups.com [mailto:xenu-usergroup@yahoogroups.com]
            On Behalf Of Michael Hunsaker
            Sent: woensdag 5 oktober 2005 18:30
            To: xenu-usergroup@yahoogroups.com
            Subject: Re: [xenu-usergroup] Not Checking All Links



            Thanks for your response. Below is a path navigation
            to a file that does not exist but Xenu is not picking
            it up:

            #1 - http://www.usi.edu
            #2 - http://www.usi.edu/alumni/index.asp
            #3 - http://www.usi.edu/alumni/connect/index.asp
            #4 - http://www.usi.edu/alumni/connect/wolfe.asp

            #4 is the broken link.

            Here is another example:

            #1 - http://www.usi.edu
            #2 - http://www.usi.edu/science/index.asp
            #3 - http://www.usi.edu/science/about.asp
            #4 - http://www.usi.edu/science/Dean/page2.asp
            #5 -
            http://www.usi.edu/science/Dean/dean/CollegeSeminarSeries.asp

            #5 is the broken link.

            Those are just a couple of examples that I could
            quickly come up with.

            Thanks again,

            Mike


            --- Tilman Hausherr <tilman@...> wrote:

            > On Wed, 5 Oct 2005 06:04:40 -0700 (PDT), Michael
            > Hunsaker wrote:
            >
            > >Sure, some of the broken links are not "real"
            > broken
            > >links but a good chunk of them are truly broken
            > links.
            > > Why isn't Xenu picking these up?
            >
            > Tell me a link that isn't spidered, but it linked
            > from a page that is.
            >
            >






            _____

            YAHOO! GROUPS LINKS



            * Visit your group "xenu-usergroup
            <http://groups.yahoo.com/group/xenu-usergroup> " on the web.

            * To unsubscribe from this group, send an email to:
            xenu-usergroup-unsubscribe@yahoogroups.com
            <mailto:xenu-usergroup-unsubscribe@yahoogroups.com?subject=Unsubscribe>

            * Your use of Yahoo! Groups is subject to the Yahoo!
            <http://docs.yahoo.com/info/terms/> Terms of Service.



            _____



            [Non-text portions of this message have been removed]
          • Tilman Hausherr
            ... I see it returns ok. Although the title is 404 error . This is a configuation problem. Here is what I get when telneting to your server on port 80 with a
            Message 5 of 10 , Oct 5, 2005
            • 0 Attachment
              On Wed, 5 Oct 2005 09:30:29 -0700 (PDT), Michael Hunsaker wrote:

              >#4 - http://www.usi.edu/alumni/connect/wolfe.asp

              I see it returns ok. Although the title is "404 error". This is a
              configuation problem.

              Here is what I get when telneting to your server on port 80 with a bogus
              URL:


              HTTP/1.1 200 OK
              Date: Wed, 05 Oct 2005 21:24:38 GMT
              Server: Microsoft-IIS/6.0
              X-Powered-By: ASP.NET
              Content-Length: 10144
              Content-Type: text/html
              Set-Cookie: ASPSESSIONIDQCCTTTAT=KOGCGHACGDDIKDCIGNPNODMP; path=/
              Cache-control: private


              <html>
              <head>
              <title>404 Error</title>
              ....
            • Michael Hunsaker
              Thanks for taking a look at this for me! I appreciate it! Mike
              Message 6 of 10 , Oct 5, 2005
              • 0 Attachment
                Thanks for taking a look at this for me! I appreciate
                it!

                Mike

                --- Tilman Hausherr <tilman@...> wrote:

                > On Wed, 5 Oct 2005 09:30:29 -0700 (PDT), Michael
                > Hunsaker wrote:
                >
                > >#4 - http://www.usi.edu/alumni/connect/wolfe.asp
                >
                > I see it returns ok. Although the title is "404
                > error". This is a
                > configuation problem.
                >
                > Here is what I get when telneting to your server on
                > port 80 with a bogus
                > URL:
                >
                >
                > HTTP/1.1 200 OK
                > Date: Wed, 05 Oct 2005 21:24:38 GMT
                > Server: Microsoft-IIS/6.0
                > X-Powered-By: ASP.NET
                > Content-Length: 10144
                > Content-Type: text/html
                > Set-Cookie:
                > ASPSESSIONIDQCCTTTAT=KOGCGHACGDDIKDCIGNPNODMP;
                > path=/
                > Cache-control: private
                >
                >
                > <html>
                > <head>
                > <title>404 Error</title>
                > ....
                >
                >
                >
                >
                >
              • Tilman Hausherr
                ... See also Nr. 32 on http://home.snafu.de/tilman/xenulink.html, it explains what one IIS admin did to solve what seems to be the same problem. Tilman
                Message 7 of 10 , Oct 6, 2005
                • 0 Attachment
                  On Wed, 5 Oct 2005 15:31:40 -0700 (PDT), Michael Hunsaker wrote:

                  >Thanks for taking a look at this for me! I appreciate
                  >it!

                  See also Nr. 32 on http://home.snafu.de/tilman/xenulink.html, it
                  explains what one IIS admin did to solve what seems to be the same
                  problem.

                  Tilman

                  >
                  >Mike
                  >
                  >--- Tilman Hausherr <tilman@...> wrote:
                  >
                  >> On Wed, 5 Oct 2005 09:30:29 -0700 (PDT), Michael
                  >> Hunsaker wrote:
                  >>
                  >> >#4 - http://www.usi.edu/alumni/connect/wolfe.asp
                  >>
                  >> I see it returns ok. Although the title is "404
                  >> error". This is a
                  >> configuation problem.
                  >>
                  >> Here is what I get when telneting to your server on
                  >> port 80 with a bogus
                  >> URL:
                  >>
                  >>
                  >> HTTP/1.1 200 OK
                  >> Date: Wed, 05 Oct 2005 21:24:38 GMT
                  >> Server: Microsoft-IIS/6.0
                  >> X-Powered-By: ASP.NET
                  >> Content-Length: 10144
                  >> Content-Type: text/html
                  >> Set-Cookie:
                  >> ASPSESSIONIDQCCTTTAT=KOGCGHACGDDIKDCIGNPNODMP;
                  >> path=/
                  >> Cache-control: private
                  >>
                  >>
                  >> <html>
                  >> <head>
                  >> <title>404 Error</title>
                  >> ....
                  >>
                  >>
                  >>
                  >>
                  >>
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >Yahoo! Groups Links
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                • Michael Hunsaker
                  Frank, Thanks for your response! Do you happen to know why these pages are sending out an HTTP 200 status code instead of a HTTP 404? Is there something I
                  Message 8 of 10 , Oct 6, 2005
                  • 0 Attachment
                    Frank,

                    Thanks for your response! Do you happen to know why
                    these pages are sending out an HTTP 200 status code
                    instead of a HTTP 404? Is there something I can do
                    within IIS to fix this?

                    Thanks again,

                    Mike

                    --- Frank Visser <f.visser3@...> wrote:

                    > Hi mike,
                    >
                    >
                    >
                    > That's because your server gives back a 200 (=ok)
                    > message, and Xenu picks
                    > this up:
                    >
                    >
                    >
                    > GET /alumni/connect/wolfe.asp HTTP/1.1
                    >
                    > Host: www.usi.edu
                    >
                    > User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
                    > en-US; rv:1.7.5)
                    > Gecko/20041107 Firefox/1.0
                    >
                    > Accept:
                    >
                    text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
                    > 0.8,image/png,*/*;q=0.5
                    >
                    > Accept-Language: en-us,en;q=0.5
                    >
                    > Accept-Encoding: gzip,deflate
                    >
                    > Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
                    >
                    > Keep-Alive: 300
                    >
                    > Connection: keep-alive
                    >
                    > Cookie:
                    > ASPSESSIONIDQCCTTTAT=FFECGHACMPNAHNMNLHLHGHKO
                    >
                    > Cache-Control: max-age=0
                    >
                    >
                    >
                    > HTTP/1.x 200 OK
                    >
                    > Date: Wed, 05 Oct 2005 21:03:59 GMT
                    >
                    > Server: Microsoft-IIS/6.0
                    >
                    > X-Powered-By: ASP.NET
                    >
                    > Content-Length: 10174
                    >
                    > Content-Type: text/html
                    >
                    > Cache-Control: private
                    >
                    >
                    >
                    > It should return a 404 message, so it's a server
                    > configuration issue.
                    >
                    >
                    >
                    > frank
                    >
                    >
                    >
                    > ========================================
                    >
                    > Frank Visser, Waterpoortweg 279, 1051 pv, Amsterdam
                    >
                    > Author of: Ken Wilber: Thought as Passion SUNY 2003
                    >
                    > Read all about Ken Wilber : http://
                    > <http://www.integralworld.net>
                    > www.integralworld.net
                    >
                    > ========================================
                    >
                    > _____
                    >
                    > From: xenu-usergroup@yahoogroups.com
                    > [mailto:xenu-usergroup@yahoogroups.com]
                    > On Behalf Of Michael Hunsaker
                    > Sent: woensdag 5 oktober 2005 18:30
                    > To: xenu-usergroup@yahoogroups.com
                    > Subject: Re: [xenu-usergroup] Not Checking All Links
                    >
                    >
                    >
                    > Thanks for your response. Below is a path
                    > navigation
                    > to a file that does not exist but Xenu is not
                    > picking
                    > it up:
                    >
                    > #1 - http://www.usi.edu
                    > #2 - http://www.usi.edu/alumni/index.asp
                    > #3 - http://www.usi.edu/alumni/connect/index.asp
                    > #4 - http://www.usi.edu/alumni/connect/wolfe.asp
                    >
                    > #4 is the broken link.
                    >
                    > Here is another example:
                    >
                    > #1 - http://www.usi.edu
                    > #2 - http://www.usi.edu/science/index.asp
                    > #3 - http://www.usi.edu/science/about.asp
                    > #4 - http://www.usi.edu/science/Dean/page2.asp
                    > #5 -
                    >
                    http://www.usi.edu/science/Dean/dean/CollegeSeminarSeries.asp
                    >
                    > #5 is the broken link.
                    >
                    > Those are just a couple of examples that I could
                    > quickly come up with.
                    >
                    > Thanks again,
                    >
                    > Mike
                    >
                    >
                    > --- Tilman Hausherr <tilman@...> wrote:
                    >
                    > > On Wed, 5 Oct 2005 06:04:40 -0700 (PDT), Michael
                    > > Hunsaker wrote:
                    > >
                    > > >Sure, some of the broken links are not "real"
                    > > broken
                    > > >links but a good chunk of them are truly broken
                    > > links.
                    > > > Why isn't Xenu picking these up?
                    > >
                    > > Tell me a link that isn't spidered, but it linked
                    > > from a page that is.
                    > >
                    > >
                    >
                    >
                    >
                    >
                    >
                    >
                    > _____
                    >
                    > YAHOO! GROUPS LINKS
                    >
                    >
                    >
                    > * Visit your group "xenu-usergroup
                    > <http://groups.yahoo.com/group/xenu-usergroup> " on
                    > the web.
                    >
                    > * To unsubscribe from this group, send an email to:
                    > xenu-usergroup-unsubscribe@yahoogroups.com
                    >
                    <mailto:xenu-usergroup-unsubscribe@yahoogroups.com?subject=Unsubscribe>
                    >
                    >
                    > * Your use of Yahoo! Groups is subject to the
                    > Yahoo!
                    > <http://docs.yahoo.com/info/terms/> Terms of
                    > Service.
                    >
                    >
                    >
                    > _____
                    >
                    >
                    >
                    > [Non-text portions of this message have been
                    > removed]
                    >
                    >
                  • frank visser
                    mike, ... sent me an email each time a user hit the 404 error page this error page gives back a 200 instead of a 404, you should report this to your server
                    Message 9 of 10 , Oct 7, 2005
                    • 0 Attachment
                      mike,

                      as you said, the issue started when you added the error page:

                      > I implemented a customized 404 error page that
                      sent me an email each time a user hit the 404 error
                      page

                      this error page gives back a 200 instead of a 404, you should report
                      this to your server administrator.

                      can you see if this page perhaps redirects to a 404?
                      http://www.usi.edu/alumni/connect/wolfe.asp

                      i don't know enough about servers, but it seems a simple fix to me.

                      frank



                      --- In xenu-usergroup@yahoogroups.com, Michael Hunsaker
                      <to_the_hurricave@y...> wrote:
                      >
                      > Frank,
                      >
                      > Thanks for your response! Do you happen to know why
                      > these pages are sending out an HTTP 200 status code
                      > instead of a HTTP 404? Is there something I can do
                      > within IIS to fix this?
                      >
                      > Thanks again,
                      >
                      > Mike
                      >
                      > --- Frank Visser <f.visser3@c...> wrote:
                      >
                      > > Hi mike,
                      > >
                      > >
                      > >
                      > > That's because your server gives back a 200 (=ok)
                      > > message, and Xenu picks
                      > > this up:
                      > >
                      > >
                      > >
                      > > GET /alumni/connect/wolfe.asp HTTP/1.1
                      > >
                      > > Host: www.usi.edu
                      > >
                      > > User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
                      > > en-US; rv:1.7.5)
                      > > Gecko/20041107 Firefox/1.0
                      > >
                      > > Accept:
                      > >
                      >
                      text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/pl
                      ain;q=
                      > > 0.8,image/png,*/*;q=0.5
                      > >
                      > > Accept-Language: en-us,en;q=0.5
                      > >
                      > > Accept-Encoding: gzip,deflate
                      > >
                      > > Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
                      > >
                      > > Keep-Alive: 300
                      > >
                      > > Connection: keep-alive
                      > >
                      > > Cookie:
                      > > ASPSESSIONIDQCCTTTAT=FFECGHACMPNAHNMNLHLHGHKO
                      > >
                      > > Cache-Control: max-age=0
                      > >
                      > >
                      > >
                      > > HTTP/1.x 200 OK
                      > >
                      > > Date: Wed, 05 Oct 2005 21:03:59 GMT
                      > >
                      > > Server: Microsoft-IIS/6.0
                      > >
                      > > X-Powered-By: ASP.NET
                      > >
                      > > Content-Length: 10174
                      > >
                      > > Content-Type: text/html
                      > >
                      > > Cache-Control: private
                      > >
                      > >
                      > >
                      > > It should return a 404 message, so it's a server
                      > > configuration issue.
                      > >
                      > >
                      > >
                      > > frank
                      > >
                      > >
                      > >
                      > > ========================================
                      > >
                      > > Frank Visser, Waterpoortweg 279, 1051 pv, Amsterdam
                      > >
                      > > Author of: Ken Wilber: Thought as Passion SUNY 2003
                      > >
                      > > Read all about Ken Wilber : http://
                      > > <http://www.integralworld.net>
                      > > www.integralworld.net
                      > >
                      > > ========================================
                      > >
                      > > _____
                      > >
                      > > From: xenu-usergroup@yahoogroups.com
                      > > [mailto:xenu-usergroup@yahoogroups.com]
                      > > On Behalf Of Michael Hunsaker
                      > > Sent: woensdag 5 oktober 2005 18:30
                      > > To: xenu-usergroup@yahoogroups.com
                      > > Subject: Re: [xenu-usergroup] Not Checking All Links
                      > >
                      > >
                      > >
                      > > Thanks for your response. Below is a path
                      > > navigation
                      > > to a file that does not exist but Xenu is not
                      > > picking
                      > > it up:
                      > >
                      > > #1 - http://www.usi.edu
                      > > #2 - http://www.usi.edu/alumni/index.asp
                      > > #3 - http://www.usi.edu/alumni/connect/index.asp
                      > > #4 - http://www.usi.edu/alumni/connect/wolfe.asp
                      > >
                      > > #4 is the broken link.
                      > >
                      > > Here is another example:
                      > >
                      > > #1 - http://www.usi.edu
                      > > #2 - http://www.usi.edu/science/index.asp
                      > > #3 - http://www.usi.edu/science/about.asp
                      > > #4 - http://www.usi.edu/science/Dean/page2.asp
                      > > #5 -
                      > >
                      > http://www.usi.edu/science/Dean/dean/CollegeSeminarSeries.asp
                      > >
                      > > #5 is the broken link.
                      > >
                      > > Those are just a couple of examples that I could
                      > > quickly come up with.
                      > >
                      > > Thanks again,
                      > >
                      > > Mike
                      > >
                      > >
                      > > --- Tilman Hausherr <tilman@s...> wrote:
                      > >
                      > > > On Wed, 5 Oct 2005 06:04:40 -0700 (PDT), Michael
                      > > > Hunsaker wrote:
                      > > >
                      > > > >Sure, some of the broken links are not "real"
                      > > > broken
                      > > > >links but a good chunk of them are truly broken
                      > > > links.
                      > > > > Why isn't Xenu picking these up?
                      > > >
                      > > > Tell me a link that isn't spidered, but it linked
                      > > > from a page that is.
                      > > >
                      > > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > > _____
                      > >
                      > > YAHOO! GROUPS LINKS
                      > >
                      > >
                      > >
                      > > * Visit your group "xenu-usergroup
                      > > <http://groups.yahoo.com/group/xenu-usergroup> " on
                      > > the web.
                      > >
                      > > * To unsubscribe from this group, send an email to:
                      > > xenu-usergroup-unsubscribe@yahoogroups.com
                      > >
                      > <mailto:xenu-usergroup-unsubscribe@yahoogroups.com?
                      subject=Unsubscribe>
                      > >
                      > >
                      > > * Your use of Yahoo! Groups is subject to the
                      > > Yahoo!
                      > > <http://docs.yahoo.com/info/terms/> Terms of
                      > > Service.
                      > >
                      > >
                      > >
                      > > _____
                      > >
                      > >
                      > >
                      > > [Non-text portions of this message have been
                      > > removed]
                      > >
                      > >
                      >
                    • Michael Hunsaker
                      I figured it out. Thanks Frank! I appreciate your feedback and help. Mike ... text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/pl ...
                      Message 10 of 10 , Oct 7, 2005
                      • 0 Attachment
                        I figured it out. Thanks Frank! I appreciate your
                        feedback and help.

                        Mike

                        --- frank visser <f.visser3@...> wrote:

                        > mike,
                        >
                        > as you said, the issue started when you added the
                        > error page:
                        >
                        > > I implemented a customized 404 error page that
                        > sent me an email each time a user hit the 404 error
                        > page
                        >
                        > this error page gives back a 200 instead of a 404,
                        > you should report
                        > this to your server administrator.
                        >
                        > can you see if this page perhaps redirects to a 404?
                        > http://www.usi.edu/alumni/connect/wolfe.asp
                        >
                        > i don't know enough about servers, but it seems a
                        > simple fix to me.
                        >
                        > frank
                        >
                        >
                        >
                        > --- In xenu-usergroup@yahoogroups.com, Michael
                        > Hunsaker
                        > <to_the_hurricave@y...> wrote:
                        > >
                        > > Frank,
                        > >
                        > > Thanks for your response! Do you happen to know
                        > why
                        > > these pages are sending out an HTTP 200 status
                        > code
                        > > instead of a HTTP 404? Is there something I can
                        > do
                        > > within IIS to fix this?
                        > >
                        > > Thanks again,
                        > >
                        > > Mike
                        > >
                        > > --- Frank Visser <f.visser3@c...> wrote:
                        > >
                        > > > Hi mike,
                        > > >
                        > > >
                        > > >
                        > > > That's because your server gives back a 200
                        > (=ok)
                        > > > message, and Xenu picks
                        > > > this up:
                        > > >
                        > > >
                        > > >
                        > > > GET /alumni/connect/wolfe.asp HTTP/1.1
                        > > >
                        > > > Host: www.usi.edu
                        > > >
                        > > > User-Agent: Mozilla/5.0 (Windows; U; Windows NT
                        > 5.1;
                        > > > en-US; rv:1.7.5)
                        > > > Gecko/20041107 Firefox/1.0
                        > > >
                        > > > Accept:
                        > > >
                        > >
                        >
                        text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/pl
                        > ain;q=
                        > > > 0.8,image/png,*/*;q=0.5
                        > > >
                        > > > Accept-Language: en-us,en;q=0.5
                        > > >
                        > > > Accept-Encoding: gzip,deflate
                        > > >
                        > > > Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
                        > > >
                        > > > Keep-Alive: 300
                        > > >
                        > > > Connection: keep-alive
                        > > >
                        > > > Cookie:
                        > > > ASPSESSIONIDQCCTTTAT=FFECGHACMPNAHNMNLHLHGHKO
                        > > >
                        > > > Cache-Control: max-age=0
                        > > >
                        > > >
                        > > >
                        > > > HTTP/1.x 200 OK
                        > > >
                        > > > Date: Wed, 05 Oct 2005 21:03:59 GMT
                        > > >
                        > > > Server: Microsoft-IIS/6.0
                        > > >
                        > > > X-Powered-By: ASP.NET
                        > > >
                        > > > Content-Length: 10174
                        > > >
                        > > > Content-Type: text/html
                        > > >
                        > > > Cache-Control: private
                        > > >
                        > > >
                        > > >
                        > > > It should return a 404 message, so it's a server
                        > > > configuration issue.
                        > > >
                        > > >
                        > > >
                        > > > frank
                        > > >
                        > > >
                        > > >
                        > > > ========================================
                        > > >
                        > > > Frank Visser, Waterpoortweg 279, 1051 pv,
                        > Amsterdam
                        > > >
                        > > > Author of: Ken Wilber: Thought as Passion SUNY
                        > 2003
                        > > >
                        > > > Read all about Ken Wilber : http://
                        > > > <http://www.integralworld.net>
                        > > > www.integralworld.net
                        > > >
                        > > > ========================================
                        > > >
                        > > > _____
                        > > >
                        > > > From: xenu-usergroup@yahoogroups.com
                        > > > [mailto:xenu-usergroup@yahoogroups.com]
                        > > > On Behalf Of Michael Hunsaker
                        > > > Sent: woensdag 5 oktober 2005 18:30
                        > > > To: xenu-usergroup@yahoogroups.com
                        > > > Subject: Re: [xenu-usergroup] Not Checking All
                        > Links
                        > > >
                        > > >
                        > > >
                        > > > Thanks for your response. Below is a path
                        > > > navigation
                        > > > to a file that does not exist but Xenu is not
                        > > > picking
                        > > > it up:
                        > > >
                        > > > #1 - http://www.usi.edu
                        > > > #2 - http://www.usi.edu/alumni/index.asp
                        > > > #3 - http://www.usi.edu/alumni/connect/index.asp
                        > > > #4 - http://www.usi.edu/alumni/connect/wolfe.asp
                        > > >
                        > > > #4 is the broken link.
                        > > >
                        > > > Here is another example:
                        > > >
                        > > > #1 - http://www.usi.edu
                        > > > #2 - http://www.usi.edu/science/index.asp
                        > > > #3 - http://www.usi.edu/science/about.asp
                        > > > #4 - http://www.usi.edu/science/Dean/page2.asp
                        > > > #5 -
                        > > >
                        > >
                        >
                        http://www.usi.edu/science/Dean/dean/CollegeSeminarSeries.asp
                        > > >
                        > > > #5 is the broken link.
                        > > >
                        > > > Those are just a couple of examples that I could
                        > > > quickly come up with.
                        > > >
                        > > > Thanks again,
                        > > >
                        > > > Mike
                        > > >
                        > > >
                        > > > --- Tilman Hausherr <tilman@s...> wrote:
                        > > >
                        > > > > On Wed, 5 Oct 2005 06:04:40 -0700 (PDT),
                        > Michael
                        > > > > Hunsaker wrote:
                        > > > >
                        > > > > >Sure, some of the broken links are not "real"
                        > > > > broken
                        > > > > >links but a good chunk of them are truly
                        > broken
                        > > > > links.
                        > > > > > Why isn't Xenu picking these up?
                        > > > >
                        > > > > Tell me a link that isn't spidered, but it
                        > linked
                        > > > > from a page that is.
                        > > > >
                        > > > >
                        > > >
                        > > >
                        > > >
                        > > >
                        > > >
                        > > >
                        > > > _____
                        > > >
                        > > > YAHOO! GROUPS LINKS
                        > > >
                        > > >
                        > > >
                        > > > * Visit your group "xenu-usergroup
                        > > > <http://groups.yahoo.com/group/xenu-usergroup> "
                        > on
                        > > > the web.
                        > > >
                        > > > * To unsubscribe from this group, send an email
                        > to:
                        > > > xenu-usergroup-unsubscribe@yahoogroups.com
                        > > >
                        > >
                        > <mailto:xenu-usergroup-unsubscribe@yahoogroups.com?
                        > subject=Unsubscribe>
                        > > >
                        > > >
                        > > > * Your use of Yahoo! Groups is subject to the
                        > > > Yahoo!
                        > > > <http://docs.yahoo.com/info/terms/> Terms of
                        > > > Service.
                        > > >
                        > > >
                        > > >
                        > > > _____
                        > > >
                        > > >
                        > > >
                        > > > [Non-text portions of this message have been
                        > > > removed]
                        > > >
                        > > >
                        > >
                        >
                        >
                        >
                        >
                        >
                      Your message has been successfully submitted and would be delivered to recipients shortly.