Relentless

october 30, 2003

Search engine crawlers are relentless at their jobs. Heh. Ever since I turned on search friendly URLs, the bots have been crawling the site like crazy. Check out the usage stats to see how the traffic has increased. Though, with the crawling, there are a lot of people who are coming into the site. I did have to block the search engines from the photo gallery since they were just sucking bandwidth like never before with all the pictures. I tried to use a robots.txt file to tell crawlers not to crawl some stuff, but it doesn't ever seem to work. So I resorted to banning those domains from the directories I don't want crawled. Last ditch effort, I guess...