Googlebot Unleashes On ultramookie

august 20, 2005

Googlebot unleashed on ultramookie this morning. I took a glance at my server logs and saw that googlebot was pounding the server over the bookmark app that I just put up. Googlebot looked like it was all confused with the tags.php file, it was trying to crawl all permutations of the tags that I had put into the system -- not that many since I have only 70 or so bookmarks in there. Over the period of 11.5 hours, googlebot hit the tags.php file 31535 times! That's almost 2800 hits an hour. I put a Disallow into my robots.txt file to hopefully stop googlebot from indexing that file. I have also started to temporarily reject packets from Google at the firewall. Hopefully when I turn off the firewall rule later on in the day, Google will read the robots.txt and stop trying to index like a cornfused retard.