I do a big recrawl around once every 10 weeks, but I have a RSS crawler that doe... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		marginalia_nu 9 months ago \| parent \| context \| favorite \| on: Marginalia – A search engine that prioritizes non-... I do a big recrawl around once every 10 weeks, but I have a RSS crawler that does daily visits and fetches new items. But that should be very low volume traffic.

stackghost 9 months ago [–]

Interesting, does your RSS crawler auto-detect feeds? That's a neat solution I'd never considered.

marginalia_nu 9 months ago | [–]

The big crawler tries to find feeds, and the RSS crawler acts on what the big one has found.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact