Loading ...
Sorry, an error occurred while loading the content.

Re: [xenu-usergroup] My URL will not work

Expand Messages
  • Tilman Hausherr
    Hello again Deno, I tried the software fiddler , which is a http debugging proxy http://www.fiddler2.com and found out the cause. Your website DOES return 404
    Message 1 of 4 , Jul 22 11:13 AM
    • 0 Attachment
      Hello again Deno,

      I tried the software "fiddler", which is a http debugging proxy
      http://www.fiddler2.com
      and found out the cause. Your website DOES return 404 for
      http://www.adtenterprises.com , but it returns a huge HTML body which
      shows a whole online shop. This may work with a browser (404 custom
      pages are ok), but not with me. A 404 is a 404, and I don't look at the
      body.

      Tilman

      On Tue, 21 Jul 2009 22:57:38 -0400, Deno Tufares, Jr. wrote:

      >Xenu will not search my site - gives me a "Not Found" error...
      >
      >http://www.adtenterprises.com
      >
      >This does not work either...
      >http://www.adtenterprises.com/cgi-bin/store/
      >It starts then gets to http://www.adtenterprises.com/cgi-bin/store/commerce.cgi and then fails with the "Not Found" error...
      >
      >I am able to use Xenu with other sites and other commerce.cgi sites with no problem - but my site does not work. I have even tried it from a different computer at a different location with the same results. I have notified the technical people for the web hosting server and they state that there are no blocks on the server. My audit logs show that "4 visitors use Xenu Link Sleuth 1.3c as a web browser".
      >
      >
      >I have read through all the notes and searched this user group with no idea as to what to do. My site has a ".htaccess" file REWRITE for my site to go directly to the CGI Shopping Cart - something like:
      >
      >RewriteEngine on
      >RewriteCond %{REQUEST_FILENAME} !-f
      >RewriteRule ^.*(\.html$|\.htm$) /cgi-bin/store/commerce.cgi
      >RewriteRule ^index\.html$ /cgi-bin/store/commerce.cgi
      >
      >
      >Please help
      >
      >Thanks
      >
      >Deno
      >-----------------------------------------------------
      > FREE Shipping & Handling
      > http://www.adtenterprises.com
      >What You Need for the 21st Century
      >-----------------------------------------------------
      >
      >_______________________________________
      >No viruses found in this outgoing message
      >Scanned by iolo AntiVirus 1.5.7.1
      >http://www.iolo.com
    • Chris
      ... You ve got an error in your .htaccess file which is forcing a request for your main index.html/.htm etc to be forwarded to your
      Message 2 of 4 , Jul 22 2:49 PM
      • 0 Attachment
        --- In xenu-usergroup@yahoogroups.com, "Deno Tufares, Jr."<deno@...> wrote:
        >
        > Xenu will not search my site - gives me a "Not Found" error...
        >
        > http://www.adtenterprises.com
        >
        > This does not work either...
        > http://www.adtenterprises.com/cgi-bin/store/
        > It starts then gets to http://www.adtenterprises.com/cgi-bin/store/commerce.cgi and then fails with the "Not Found" error...
        >
        > I am able to use Xenu with other sites and other commerce.cgi sites with no problem - but my site does not work. I have even tried it from a different computer at a different location with the same results. I have notified the technical people for the web hosting server and they state that there are no blocks on the server. My audit logs show that "4 visitors use Xenu Link Sleuth 1.3c as a web browser".
        >
        >
        > I have read through all the notes and searched this user group with no idea as to what to do. My site has a ".htaccess" file REWRITE for my site to go directly to the CGI Shopping Cart - something like:
        >
        > RewriteEngine on
        > RewriteCond %{REQUEST_FILENAME} !-f
        > RewriteRule ^.*(\.html$|\.htm$) /cgi-bin/store/commerce.cgi
        > RewriteRule ^index\.html$ /cgi-bin/store/commerce.cgi
        >
        >

        You've got an error in your .htaccess file which is forcing a request for your main index.html/.htm etc to be forwarded to your /cgi-bin/store/commerce.cgi

        The problem is, rather than return the code 200 AND the content, it returns the content and the 404 error code.

        This is the header that is being returned to a GET (shown first)

        GET / HTTP/1.1
        Host: www.adtenterprises.com
        User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
        Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5,application/x-tsmxml
        Accept-Language: en-gb,en;q=0.5
        Accept-Encoding: gzip,deflate
        Accept-Charset: UTF-8,*
        Keep-Alive: 300
        Connection: keep-alive


        Response from your server

        HTTP/1.x 404 Not Found
        Date: Wed, 22 Jul 2009 21:40:16 GMT
        Content-Type: text/html; charset=ISO-8859-1
        Connection: keep-alive
        Server: Apache
        Set-Cookie: cart_id=1248298817.419; expires=Wed, 22-Jul-2010 21:40:16 GMT; path=/cgi-bin/store; domain=www.adtenterprises.com;
        Content-Length: 18461

        I can't remember the correct .htaccess redirect method, but if I were you I'd go searching around.

        Any visitor will see a 404, but worse still ALL search engines and spiders will also see the 404 error code.
        So despite you web server sending out the contents of /cgi-bin/store/commerce.cgi, I would hazard a guess that they all see the 404 and leave assuming that the page was not found...
        (Some search engines might crawl the 'supposed' 404 looking for other pages, but will assume that the page at www.adtenterprises.com does not exist.

        But, even worse, if I then do a

        GET /cgi-bin/store/0/12_Volt_Travel_Blanket.htm HTTP/1.1

        (looking for http://www.adtenterprises.com/cgi-bin/store/0/12_Volt_Travel_Blanket.htm)

        Your web server still sends out a 404

        HTTP/1.x 404 Not Found

        Even though the 404 is returned, the web server still sends out the content...

        If I can remember the .htaccess stuff, I'll come back and let you know.

        Regards

        Chris
      Your message has been successfully submitted and would be delivered to recipients shortly.