I was glad when googlebot visited the website, but ... the load of googlebot was to heavy. In practice I will be able to handle circa 30000 pages a day. So it shouldn't be a surprise that googlebot sliced the website down.
It wasn't easy to find where to set the crawl rate in Google Webmaster Tools. Initial I was looking at the menu at the left while the option is available under the "settings icon" shown at the top right. The next hurdle was that it was not clear for me that this setting is only available for the domain name zomdir.com and not for the subdomains www.zomdir.com or thedarksideof.zomdir.com.
After changed the crawlspeed I concluded that in process of time Googlebot is indeed slowing down. I hope that the other bots will follow these instruction in the robots.txt file:
User-agent: *
Crawl-delay: 50
At this moment it seems that it worked to tame the bots. The last days the site continues to respond while several bots are visiting ZOMDir.com. I hope it stays that way.
No comments:
Post a Comment