The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked. If I didn't block them, I would need to pay for a beefier server. All the while this doesn't give me or my regular visitors any benefit, it's just large corporations driving up my hosting costs.
> The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked
What are they downloading, like heavy videos and stuff? Initiating heavy processes or similar?
It takes 125,000 4MB requests to use up 0.5 TB bandwidth, which is the lowest offered by Vultr. I could see this being an issue for personal sites that include photos.
