6/28/2007

Yahoo Spider Problems– Too many IPs!

Filed under: — Mike @ 11:01 am

So a suggestion I made over at the Yahoo suggestions board seems to be growing legs, so I figured I would post it here.

The long and short of it is that Yahoo has changed the way they crawl websites. Instead of a nice, slow meandering pace from limited originating IPs, Yahoo is now using a different IP for nearly every page request when crawling websites.

This is leading to increased and overly-inflated guest counts on nearly all scripts that utilize guest-visitor totals, or guest visitor tracking. It is also causing undue load upon many servers.

Here is my Yahoo suggestion: http://suggestions.yahoo.com/detail/?prop=SiteExplorer&fid=31431

Here is the Digg: http://www.digg.com/software/Yahoo_Spider_wrecking_havoc_on_website_guest_tracking

Leave a Reply