Grouping BOT IPs
- Grouping BOT IPs can improve visibility as to who is visiting your
site. We've had a large increase in site visits. After aggregating
entries in the Sites table I'm better able to see what proportion is
due to increase in search engine crawler/spider visits.
Post #3405 correctly pointed out it's best to use wildcards and not
list individual IP addresses in a range, this example does that.
I used SpiderTrack.org and IPLists.com to build the list. The
following lines go into your Webalizer CONF file.
GroupSite 66.249.* GoogleBot
GroupSite 216.239.36.* GoogleBot_Translation
GroupSite 216.239.38.* GoogleBot_Translation
GroupSite 65.54.* MSNBot
GroupSite 64.242.* LookSmart_Crawler
GroupSite 66.147.154.* IBM_Almaden
GroupSite 68.142.* Yahoo_Slurp