I just realized a Bot has been crawling this Blog excessively. This bot identifies itself as Twisted PageGetter. During this month only Twisted PageGetter has made 11.714 hits on this URL. It cost my bandwidth a lot and wasting server resources, even though I have unmetered bandwidth but excessive hits obviously adds to server loads and it could reduced the server stability.
You can check your server log to figure out this problem either. If you have similar problem with Twisted PageGetter, you can add these lines to .htaccess files on your server.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "Twisted PageGetter" [NC]
RewriteRule ^.*$ - [F,L]
Those code will give Twisted PageGetter bot a 403 Forbidden Error and it will stop the bot to crawl deeper on your site. Hopefully the Bot “get” my message and stop accessing my site anymore.
1 comment for this entry ↓
1 Blogulate // Oct 7, 2007 at 12:14 pm
Hmmm .. havent heard of that before ..
Leave a Comment