Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rul...

		mrkramer on March 19, 2022 \| parent \| context \| favorite \| on: If Google sucks then why is everyone still using i... Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect robots.txt.