I had the same issue with TikTok/ByteDance. They were using almost 100gb of my traffic per month.
I now block all ai crawlers at the cloudflare WAF level. On Monday I noticed a HUGE spike in traffic and my site was not handling it well. After a lot of troubleshooting and log parsing, I was getting millions of requests from China that were getting past cloudflare's bot protection.
I ended up having to force a CF managed challenge for the entire country of China to get my site back in a normal working state.
In the past 24 hours CF has blocked 1.66M bot requests. Good luck running a site without using CloudFlare or something similar.
I now block all ai crawlers at the cloudflare WAF level. On Monday I noticed a HUGE spike in traffic and my site was not handling it well. After a lot of troubleshooting and log parsing, I was getting millions of requests from China that were getting past cloudflare's bot protection.
I ended up having to force a CF managed challenge for the entire country of China to get my site back in a normal working state.
In the past 24 hours CF has blocked 1.66M bot requests. Good luck running a site without using CloudFlare or something similar.
AI crawlers are just out of control