2997Webcrawlers and Yahoo Groups search
- Nov 7, 2004Hi,
just started looking at my Webalizer output, which resulted in my
joining this group :-)
I built a Fedora Core 2 system a few months back, and recently opened
Port 80 on the NAT firewall to allow external access to the web site.
Webalizer reports a very wide range of sites - .nl, .nz, .it, .rr.com
and loads of .net sites all visiting my website.
As the site isn't publicised anywhere (and only has a couple of dumb
java script frames) I presume that the access is by web crawlers
which are just ploughing their way through IP addresses looking for
The number of accesses seem to be climbing also, but no massive
attack (hmmm...cool name for a group) so I presume they aren't
finding anything useful.
The reason for the title of this post - I tried to use Yahoo Groups
search for articles on web crawlers but the search returned all the
postings in the group.
I cross checked searching for 'bumfluff' (well, you never know) which
as expected turned up no postings.
So as I can't seem to search the archives, does anyone have any
useful information on these sites? (or why I can't search
for 'crawlers' in this group)
Top user agent seems to be Kostiki Client 2.20.40120.0
Top host for October with 26 hits in 5 visits was 18.104.22.168
which doesn't have a DNS reverse lookup.
All direct requests, no referrers.
I do like the stats from Webalizer - I presume there is a patch or
similar to Apache (sorry, httpd) which picks up
the 'http://localhost/usage' URL as I just blasted my old website
onto the new box then pointed web root somewhere in the middle.
Nevertheless Apache still serves the usage page just fine.
Any info. gratefully received.
- Next post in topic >>