Re: crawling a large site
- Having upgraded to 1.3c, it runs for awhile on the 4 GB machine, but soon shows an "Out of memory" warning. Since I'd turned off the "fail domain" option, it showed many of these warnings, and then Xenu abruptly disappeared.
I restarted, and tried opening the file but hitting pause immediately. Then I tried sorting the listings, but nothing would happen. (I'd wait awhile & come back to the computer, just in case) I also tried exporting to a tab-delimited file, but nothing would happen. (again, I waited)
I suppose I just need to subdivide the site further.
Still, I can't complain; it's still the best program I've used for this, and all the other sites I've crawled have worked out fine (usually 180-230,000 URLs in those).