Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

News websites are asking if I'm a robot instead of showing the headlines.Is it common? https://postimg.cc/47xNhT1g


Unfortunately yes. They see too many requests from paperdelivery.co and block requests originating from that domain - they only want the google bot to crawl their site :) I think I have two options:

1. Respect their decision to not allow PaperDelivery to fetch that page.

2. Use a set of proxies and headless browsers or outright impersonate the Google bot to trick the news websites into allowing PaperDelivery to fetch that page.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: