ZOMDir > Blog

Monday, 9 September 2013

Crawl speed to low?

As stated in Taming the bots, I have to slow down the bots. This evening I concluded that Googlebot is again visiting ZOMDir.com multiple times a second. The result is that the site is down again.

I have changed my robots.txt and for Googlebot I have the following setting in Google Webmaster Tools according to this instruction


0.02 requests per second (50 seconds between visits)



The crawlrate in Google Webmaster Tools (in Dutch)
Due to the fact there are 24*60*60 = 86400 seconds in a day, I expect that Googlebot will retrieve a maximum of 1728 pages a day, with this setting.

May be Google thinks this makes no sense, and may be they are right. I will change this setting to 0.2 (actual 0.198) requests per second and hope that Googlebot respect this setting.


Note: I will keep the setting in robots.txt the same for this moment (Crawl-delay: 50) for other bots.


I am pretty sure that Googlebot is visiting ZOMDir.com. The IP addresses used by this bot are for example: 


66.249.73.151

66.249.73.142
66.249.73.158
66.249.73.137
66.249.73.138
66.249.73.157

Verifying these adresses give via Reverse DNS Lookup results like this:



66.249.73.142 resolves to
"crawl-66-249-73-142.googlebot.com"
Top Level Domain: "googlebot.com"
Country IP Address: UNITED STATES

The only hope I have at this moment is that someone wrote that it can take weeks before Googlebot is following your instructions. I hope that that person was wrong ...

No comments:

Post a Comment