For the "good" bots which at least respect robots.txt you can use this list to g... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		jsheard 11 months ago \| parent \| context \| favorite \| on: Nepenthes is a tarpit to catch AI web crawlers For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

breakingcups 11 months ago | [–]

Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1]

1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...

zcase 11 months ago | | [–]

For those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-blo...

maeil 11 months ago | | [–]

This seemed to work for some time when it came out but IME no longer does.

taikahessu 11 months ago | [–]

Thanks, will look into that!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact