Loading ...
Sorry, an error occurred while loading the content.

2997Webcrawlers and Yahoo Groups search

Expand Messages
  • smutterbuggler
    Nov 7, 2004

      just started looking at my Webalizer output, which resulted in my
      joining this group :-)

      I built a Fedora Core 2 system a few months back, and recently opened
      Port 80 on the NAT firewall to allow external access to the web site.

      Webalizer reports a very wide range of sites - .nl, .nz, .it, .rr.com
      and loads of .net sites all visiting my website.

      As the site isn't publicised anywhere (and only has a couple of dumb
      java script frames) I presume that the access is by web crawlers
      which are just ploughing their way through IP addresses looking for
      anything interesting.

      The number of accesses seem to be climbing also, but no massive
      attack (hmmm...cool name for a group) so I presume they aren't
      finding anything useful.

      The reason for the title of this post - I tried to use Yahoo Groups
      search for articles on web crawlers but the search returned all the
      postings in the group.

      I cross checked searching for 'bumfluff' (well, you never know) which
      as expected turned up no postings.

      So as I can't seem to search the archives, does anyone have any
      useful information on these sites? (or why I can't search
      for 'crawlers' in this group)

      Top user agent seems to be Kostiki Client 2.20.40120.0

      Top host for October with 26 hits in 5 visits was
      which doesn't have a DNS reverse lookup.

      All direct requests, no referrers.

      I do like the stats from Webalizer - I presume there is a patch or
      similar to Apache (sorry, httpd) which picks up
      the 'http://localhost/usage' URL as I just blasted my old website
      onto the new box then pointed web root somewhere in the middle.

      Nevertheless Apache still serves the usage page just fine.

      Any info. gratefully received.

      Dave R
    • Show all 9 messages in this topic