I would think rate-limiting would be good. Crawlers are not patient enough to operate at the speed of a real human user.