Loading ...
Sorry, an error occurred while loading the content.

Re: crawling a large site

Expand Messages
  • brandonmbyers01
    Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows an Out of memory warning. Since I d turned off the fail domain option, it
    Message 1 of 5 , Jun 25, 2009
    • 0 Attachment
      Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows an "Out of memory" warning. Since I'd turned off the "fail domain" option, it showed many of these warnings, and then Xenu abruptly disappeared.

      I restarted, and tried opening the file but hitting pause immediately. Then I tried sorting the listings, but nothing would happen. (I'd wait awhile & come back to the computer, just in case) I also tried exporting to a tab-delimited file, but nothing would happen. (again, I waited)

      I suppose I just need to subdivide the site further.

      Still, I can't complain; it's still the best program I've used for this, and all the other sites I've crawled have worked out fine (usually 180-230,000 URLs in those).

      -Brandon
    Your message has been successfully submitted and would be delivered to recipients shortly.