Fun fact: the only complete copy of the Internet Archive's library is in Alexandria.
How's that for historic irony and "unteachability" of the human species.
Honestly, now's the time to make copies of it, while we still can. Torrents need seeders and people that care, and we are the last generation that cares about knowledge.
We need to prevent the following generations to grow up as mindless clickmonkeys of the digital Orwellian world.
In my opinion redundancy in a single business entity is no redundancy at all, especially if there's legal obligations of a soon-to-be-burning-books-again regime.
A better strategy would have been to found independent entities in other liberal democracies, so they can act as IP backups.
There was a great vpro documentary called "Digital Amnesia" [1] where they also interviewed the lead of the library of Alexandria, who was the only bidder to buy the national KIT library of the Netherlands and its dissolved inventory at the time.
Interviews with archivists, librarians, web archive and others on the topic. It's insane to see that nations don't want to preserve their history, science, and culture anymore.
libgen was around 190TB. For my own at home cluster I decided to go for 512TB but I can't host nor upload in these bandwidth requirements from here.
I started to build sth like a torrent splitter tool yesterday because I realized that all torrent clients just crash when you try to open, modify, or seed those torrents.
Edit: correction, the IA is ~15PB big, brewster kahle mentioned it in the documentary (2014)
300tb seems insanely small so that 15pb number is probably closer
Id say though with the rise of video on social media, if they didn't 3x or 4x that 15pb number since then.
I'm sure there's plenty of room to replicate to aws but I'm sure they couldn't afford to host 60pb of data on aws. They have their own Colo or data center if I recall.
I think the general issue is that Annas Archive only has the metadata that's searchable, in a second step with downloading the parts you want/need you'll see the real storage requirements.
Most of their stuff is uploaded on ipfs mirrors as it looks like, though.
How's that for historic irony and "unteachability" of the human species.
Honestly, now's the time to make copies of it, while we still can. Torrents need seeders and people that care, and we are the last generation that cares about knowledge.
We need to prevent the following generations to grow up as mindless clickmonkeys of the digital Orwellian world.