Writing your own may take some time but it's definitely worth it.

I use Python + AsyncIO + lxml. If you're are new to this and don't have time to learn new things, I suggest you try import.io

Write your own. For me, it's the most efficient (and you will learn a lot).

Absolutely, the web is such a horrid place that scraping is a necessity.

You scrape as an alternative to browsing in some cases? That's interesting, is it to avoid advertising / tracking / poor UI / something else?

I love Nokogiri. It is a ruby library for parsing XML/HTML