From my recalling, a few years at most. There was, and apparently still exists, distributed.net which was aimed at brute forcing DES (easy), RC5-56 bits and then RC5-64 bits by establishing a web of personal computers (via a client one had to install). Thus it was well known brute forcing was achievable in a reasonable time.
PGP (1991) was considered secure as it was considered not brute forceable. With 128 bits, it was considered military grade at the time and the US had an export restriction due to that. That might have been an incentive for GNU Privacy Guard. In France you had to give your private key to the government authority if an encryption system used anymore than 56 bits (as I recall, I don't remember the exact number).
Copyright IS an issue to Debian, and always has been for the last 30 years or so. They are, rightfully so, extremely picky when it comes to respecting copyright and licensing. It is the 0.01%.
> Who the f*ck cares who made it? A monkey could have made it for all I care. If it does what it claims to do, and I understand how, it's all good.
Copyright laws do care. As an example one can send a patch claiming its their own but because they had do it under their employer duty, the copyright might well be associated to their employer rather than them individually. Does the patch does what it claims to? Surely. Is that a copyright infringement? DEFINITELY SO.
The copyright rationale for the first proposition (Choice 1: Ban LLM contributions from Debian via Social Contract):
> 1. Copyright
> -------------
>
> LLM output has very unclear legal status: it may be possible to copyright on its own merits, or not; it may be affected by all of the licenses and copyrights in the training data, or not.
> Debian Policy and the DFSG require absolute clarity for licensing and copyright[1][2]. Software and other contributions written conventionally by humans with unclear copyright or license status are not allowed in Debian; LLM output should not have a special exception to this.
This rationale states if there is doubt about the copyright of the code, it not suitable for inclusion. Until I guess LLM output get a clarification regarding who is the author of its output.
Maybe but I think you are underestimating the achievements Fabrice has accomplished. Among others:
- Improved an algorithm to compute Pi, ran it on a *personal laptop* and broke the world record. That achievement is not even listed on his personal homepage, and it a single line of facts with Zero bragging involved https://www.bellard.org/pi/pi2700e9/
- a PC emulator in vanilla javascript, boot the Linux Kernel in a browser and get a virtual terminal also implemented from scratch
- QuickJS, embeddable, self contained (no libs) and fast JavaScript engine matching almost entirely ES2025
- NNCP, a Neural Networks driven lossless data compression system
I have been referring to his page for decades as an example of one can have a huge respect without having a fancy web page and no bragging at all. He is a genius :-)
Not at all. I mean, regardless of him not having a fancy web page or an Instagram, he is anyway an Internet geek celebrity we all know and respect. My point is that I believe there are many similar but noname engineers whose achievements stayed and will stay behind corporate proprietary walls.
I do not understand why the scrappers do not do it in a smarter way: clone the repositories and fetches from there on a daily or so basis. I have witnessed one going through every single blame and log links across all branches and redoing it every few hours! It sounds like they did not even tried to optimize their scrappers.
> I do not understand why the scrappers do not do it in a smarter way
If you mean scrapers in terms of the bots, it is because they are basically scraping web content via HTTP(S) generally, without specific optimisations using other protocols at all. Depending on the use case intended for the model being trained, your content might not matter at all, but it is easier just to collect it and let it be useless than to optimise it away⁰. For models where your code in git repos is going to be significant for the end use, the web scraping generally proves to be sufficient so any push to write specific optimisations for bots for git repos would come from academic interest rather than an actual need.
If you mean scrapers in terms of the people using them, they are largely akin to “script kiddies” just running someone else's scraper to populate their model.
If by scrapers in terms of people writing them, then the fact that just web scraping is sufficient as mentioned above is likely the significant factor.
> why the scrappers do not do it in a smarter way
A lot of the behaviours seen are easier to reason if you stop considering scrapers (the people using scraper bots) to be intelligent, respectful, caring, people who might give a damn about the network as a whole, or who might care about doing things optimally. Things make more sense if you consider them to be in the same bucket as spammers, who are out for a quick lazy gain for themselves and don't care, or even have the foresight to realise, how much it might inconvenience¹ anyone else.
----
[0] the fact this load might be inconvenient to you is immaterial to the scraper
[1] The ones that do realise that they might cause an inconvenience usually take the view that it is only a small one, and how can the inconvenience little them are imposing really be that significant? They don't think the extra step of considering how many people like them are out there thinking the same. Or they think if other people are doing it, what is the harm in just one more? Or they just take the view “why should I care if getting what I want inconveniences anyone else?”.
Because that kind of optimization takes effort. And a lot of it.
Recognize that a website is a Git repo web interface. Invoke elaborate Git-specific logic. Get the repo link, git clone it, process cloned data, mark for re-indexing, and then keep re-indexing the site itself but only for things that aren't included in the repo itself - like issues and pull request messages.
The scrapers that are designed with effort usually aren't the ones webmasters end up complaining about. The ones that go for quantity over quality are the worst offenders. AI inference-time data intake with no caching whatsoever is the second worst offender.
Because they don't have any reason to give any shits. 90% of their collected data is probably completely useless, but they don't have any incentive to stop collecting useless data, since their compute and bandwidth is completely free (someone else pays for it).
They don't even use the Wikipedia dumps. They're extremely stupid.
Actually there's not even any evidence they have anything to do with AI. They could be one of the many organisations trying to shut down the free exchange of knowledge, without collecting anything.
So the easiest strategy to hamper them if you know you're serving a page to an AI bot is simply to take all the hyperlinks off the page...?
That doesn't even sound all that bad if you happen to catch a human. You could even tell them pretty explicitly with a banner that they were browsing the site in no-links mode for AI bots. Put one link to an FAQ page in the banner since that at least is easily cached
When I used to build these scrapers for people, I would usually pretend to be a browser. This normally meant changing the UA and making the headers look like a read browser. Obviously more advanced techniques of bot detection technique would fail.
Failing that I would use Chrome / Phantom JS or similar to browse the page in a real headless browser.
I guess my point is since it's a subtle interference that leaves the explicitly requested code/content fully intact you could just do it as a blanket measure for all non-authenticated users. The real benefit is that you don't need to hide that you're doing it or why...
You could add a feature kind of like "unlocked article sharing" where you can generate a token that lives in a cache so that if I'm logged in and I want to send you a link to a public page and I want the links to display for you, then I'd send you a sharing link that included a token good for, say, 50 page views with full hyperlink rendering. After that it just degrades to a page without hyperlinks again and you need someone with an account to generate you a new token (or to make an account yourself).
Surely someone would write a scraper to get around this, but it couldn't be a completely-plain https scraper, which in theory should help a lot.
I would build a little stoplight status dot into the page header. Red if you're fully untrusted. Yellow if you're semi-trusted by a token, and it shows you the status of the token, e.g. the number of requests remaining on it. Green if you're logged in or on a trusted subnet or something. The status widget would links to all the relevant docs about the trust system. No attempt would be made to hide the workings of the trust system.
I was collecting UK bank account sort code numbers (to a buy a database at the time costs a huge amount of money). I had spent a bunch of time using asyncio to speed up scraping and wondered why it was going so slow, I had left Fiddler profiling in the background.
“world” is derived from “were” + “eald” (old), and meant “the age of humans”, which was distinguished from the age of the Gods, when the Æsir and Vanir dominated, and the age of the Jötnar.
I find it interesting how the term shifted from a (mythical) temporal concept to a spatial concept, to now often a social concept (e.g. the Fourth World).
Why add the complexity of having to maintain an Ansible installation, a logging stack, deal with their upgrades and whatever python issue one might encounter. I had the issue of Ansible builtin `shell` not doing the right thing (sh vs bash) or it being unnecessarily slow when uselessly looking up `cowsay`.
Adding layers and layers of tooling is often overkill and it is hard to bit the simplicity of 33 lines of shell when the use case is a single person doing the code, deployment and maintenance.
I’m with you on the usecase. Simple server deployment on a VM, bash script is fine, in fact I recommend it. It’s when you start dealing with 5+ VMs that I would start looking into using a tool like Ansible.
@unixispower , you might consider adding the site to TheOldNet webring ( https://webring.theoldnet.com/submit ). I have discovered it from a yesterday post about ucanet (a DNS for retro site). Your site would be an excellent ring member!
His position was about moderating legal content. He thinks moderating legal content is censorship. He is in favor of taking down illegal content. If people think currently legal content should be taken down they should appeal to change the laws and it shouldn't be part of at least their platform to judge the legal content. That's his position, not my position.
That's what he says his position is. He bans people who post the movements of his private jet. He also has had Twitter file lawsuits against people merely for saying mean things about Twitter.
Of course, it's also possible that he's a complete idiot and has no idea what the First Amendment actually permits in speech, like he stopped paying attention after Schenck v US.
Yeah, that is hypocritical. He should be congruent in his views and in any ambiguity in content disfavoring him he should lean towards giving benefit of the doubt to prove lack of bias. Although in this case there didn't seem to be any ambiguity, he should have not banned the account.
Filing lawsuits is entirely consistent with a position that the courts, not private companies, should regulate speech online.
Then there's Jack Sweeney, the guy tracking Elon and his private jet - who, by the way, was accused of facilitating stalking by Taylor Swift. For him, X made a policy against any account "doxxing real-time location info of anyone".
Does anyone here actually want to argue that tracking real-time location info should be allowed?
Real-time tracking of a person or their ground transport: not ok.
Real-time tracking of a person's plane: ok.
And the reason for such a distinction is that planes can only land on specific ground slots. That also means that real-time tracking of a person's helicopter falls under "not ok". And by extension, the same will hold for flying taxis, once they take off[tm].
We don't (yet?) live in a world where shoulder mounted surface-to-air missiles, outside of war zones, are a realistic threat.
He just banned posting the identity of pseudonymous accounts which is definitely not illegal. He also banned posting public information about the movements of his private jet which is also definitely not illegal.
This stuff would easier to take seriously he was consistent about it. At this point it’s kind of insulting
This is one where “won’t someone thing if the children” (underage and non-consensual) is relevant. It tends to be a lot more… universally agreed upon limits to whatever you think free speech is.
> Elon Musk twitted about moderation being censorship
And instead, X has implemented hellbanning, where nobody outside X knows who is being censored and why. People just slowly figure out that, actually, no one sees their posts. At least with outright <scare-quote> "moderation" you would know that you had been cancelled.
I work for Wikimedia and I have been one of the few weirdos assisting in its creation back in the early 2000's. The first time I got exposed to wikis predates Wikipedia and my stance was:
- a site editable by anyone on the internet? That is never going to fly
Few months later I followed a link to Wikipedia and that clicked, we can definitely build an encyclopedia online and the hope (I was in my early 20's) or bet was that more people were wiling to write article than people willing to deface it. I guess we won that bet by a large margin :-]
Even if I was younger, I was not clueless. I was well aware some people would deface it and it did happen. I have also been involved in two very long and tedious fights with editors having a political agenda, which diverting me from actually writing articles. I think that is the real danger: shifting the focus of people from writing articles toward pointless long discussions.
That burning up of resources of the good guys is exactly why a lot of stuff eventually derails. Just like the fight against the online crime rings that hold people's data ransom and that deface and destroy: the defenders have to succeed all the time, the attackers only have to succeed part of the time to be successful. So over a long enough run the attackers have an edge. Wikipedia is an exception, so far. Enough people cherish it that they are willing to put in the effort. But the day enough of them blink at the same time the assholes will take over. I hope that day will never come, but I'm not sure it will not. On a human scale Wikipedia is still very young.
PGP (1991) was considered secure as it was considered not brute forceable. With 128 bits, it was considered military grade at the time and the US had an export restriction due to that. That might have been an incentive for GNU Privacy Guard. In France you had to give your private key to the government authority if an encryption system used anymore than 56 bits (as I recall, I don't remember the exact number).
reply