I notice they have updated their robots.txt to only allow user agents they have ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		xd on March 25, 2011 \| parent \| context \| favorite \| on: How I got sued by Facebook (2010) I notice they have updated their robots.txt to only allow user agents they have approved. http://www.facebook.com/apps/site_scraping_tos.php

historical on March 25, 2011 [–]

I noticed that a while back when it had the effect of removing thefacebook.com from the Wayback Machine. There used to be some fascinating reading in their old TOS and Privacy Policy. Wish I'd kept a copy.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact