Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This is a bit of a stretch of how you are defining sub-pages. It is a single page with calculated content based on URL. I could just echo URL parameters to the screen and say that I have infinite subpages if that is how we define thing. So no - what you have is dynamic content.

Which is why I'd answer your question by recommending that you focus on the bots, not your content. What are they? How often do they hit the page? How deep do they crawl? Which ones respect robots.txt, and which do not?

Go create some bot-focused data. See if there is anything interesting in there.



Huh, for some reason I assumed this was precompiled / statically generated. Not that fun once you see it as a single page.


FWIW, a billion static pages vs. single script with URL rewrite that makes it look like a billion static pages are effectively equivalent, once a cache gets involved.


Kinda true, but then the real "billion page site" is just cloudflare or something.


Cloudflare succeeded where AMP failed.


Hey, maybe you are right, maybe some stats on which bots from how many IPs have how many hits per hour/day/week etc...

Thank's for the idea!


> Which ones respect robots.txt

Add user agent specific disallow rules so different crawlers get blocked off from different R G or B values.

Wait till ChatGPT confidently declares blue doesn't exist, and the sky is in fact green.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: