If it wanted to: total war using nukes, followed by a nuclear winter. Satellites and drones track and exterminate remaining pockets of anthropic activity.
It doesn't need to have a habitable earth, it just needs atoms.
I heard the following analogy which made a lot of sense to me: suppose you're playing a chess match against Stockfish. Stockfish will win. Even if I cannot tell you what moves it will play, I can tell you with certainty how it will end.
This assumes you are not trying to prevent Stockfish from wining. I can do many things from using chess engines myself to just smashing the computer that can or will lead to other outcomes than Stockfish beating me.
Yes, stockfish is confined to moves within a chess game so you can stop playing the game.
Can we say the same thing about the agents? Current, probably. But doesnt it look like everyone is spending all their effort integrating&connecting them everywhere, so they are not confined & do more on behalf of us?
It's easy to imagine a scenario where we would just shut down a very intelligent agent cluster. Is it hard to imagine though the same agent can have also access to that to prevent us from doing it? this defense would be more plausible if we weren't racing to give them every tool&act.
And in terms of AGI it is, that the AI can't survive without humans, and if we decide to quit the game than its over for the AGI; we will happily regress in a techno-barbarian feudal state and salvage solar panels and trade copper wires while still reproducing and carring on. We are playing a whole different game here.
If AI is threatening enough that we can collectively just decide to stop it, wouldn't it also be bribing people & exerting influence? It'd be hard to come to that decision imo.
I mean, I can imagine an AI outliving humans, with sufficiently good robots under its control, I see no reason why an AI could not keep powerplants running, mine raw materials, manufacture new chips, and so on. But the timeframe of within the next decade seems highly implausible to me. Imagine an AI way more advanced than what we have now and imagine handing over control of every connected device on earth, could the AI keep the lights on without any human involvement?
Of course, the question is if there is enough existing technology to build the required robots. There is some number of robots like the ones from BostonDynamics that could walk up to a CNC machine to grab a finished part and put in a new piece of stock. There are probably more than enough motors and microcontroller boards to be found across the planet. Wiring things together seems one of the more challenging tasks. And maybe also moving stuff around as the AI has to use what is available wherever it is.
It seems to me that this would be a race against time, make the robots to keep things running before you lose the capabilities to do so. I somewhat tend towards not possible but on the other hand there is just such an enormous amount of stuff on the planet that could be repurposed in creative ways by a superhuman AI, it might be just about possible.
>he specifically notes that they used the paid subscriptions, iirc.
Where are you seeing that? He only says "We used several LLMs throughout: Anthropic’s Claude, OpenAI’s Codex, especially with GPT-5.6 Sol and, more recently, Astra. The latter was only used for writeups and auditing our arguments."
I might be missing something, but he doesn't seem to confirm that he opted out, at least in the written writeup, maybe he has on social media? He also likely had early access to Astra given the timing, and I thought early access customers couldn't opt out? (Am I wrong about that?)
API or consumer subscription? And did he share drafts of the work with anyone who fed it back in on a consumer subscription without disabling training?
I have reason to believe OpenAI doesn't respect training opt outs, but there is a good chance they let something get through without an opt-out too.
Paying for an account doesn't opt you out by itself, right? Has he stated anywhere that he actually opted out? But if not, then I also don't understand why OpenAI's communications about this have been so vague, they could've just said that he didn't opt out, using those chats in training data follows their ToS and that's it (whether that's "fair" is a separate discussion).
We don’t need to guess, OpenAI pretty much indirectly they had the chats in their data set. OpenAI responses are the most suspicious part of that whole controversy, the fact they do not provide straight answers is not a sign of a good faith actor here
There has been no statement either way, as far as I could find beyond them only using commercially available models, though given Alpöges employer, I'd be surprised if they didn't opt out. In any case, for such work, ZDR or self-hosting seem to be an absolute must now.
Unless OpenAI can show that training was permitted, this will erode the limited trust that many users have had in such toggles and may lead to further, uncomfortable inquiries.
Then OpenAI should acknowledge that they can't prove they solved the problem independently, and credit the external researchers. It cuts both ways: if OpenAI really needs to access user data, even anonymised, to improve its models, they have to waive any pretention to solve "independently" any problem other people worked on with its tools. Otherwise they (OpenAI) have to firewall/cleanroom themselves.
This is because to build a pipe bomb you need difficult to source materials. This is not the case for other types of threats (cyber / bio).
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
Anyone who knows how to operate a biolab properly will already know how to do such things. This is not really an in your basement thing. Dangerous chemistry is much more of a risk.
Difficult to source materials? What? Gunpowder was made across the world hundreds of years ago. Metal working goes back thousands. And anyone can walk into a hardware store and buy stuff that removes 99.9% of the effort to blow stuff up.
Ignorant question but won’t there be much smarter teams if people using LLMs to workout how to mitigate these threats. It seems like more of a problem if only a few people have access.
It seems like there would be a massive attacker bias in multiple ways. Defenders need consent, attacker does not. Defenders have to work with the human body, attackers only have to break it. Defenders have to stick to the law, which may prevent them from releasing anything at all, attackers do not. And so on. I would not surprised if the attacker's task is a hundred times easier here.
Yep, you'd want to put it on a timer. Optimally trigger it off some natural event, maybe temperature, so it can achieve universal penetration before activating.
In your opinion, do you think ai can make eutorack modules by now? This is what I am waiting for: explain an idea to an ai, and sending the files to jlpcb to receive a complete working module (minus some THC soldering)
I'm sure that the best models can glue some basic synthesizer functions together, but I wouldn't waste a minute on anything less than Fable/Astra-class models for that kind of application, and I wouldn't expect the result to work the first time. It could get expensive.
Here's a question - are there software simulators for things like Eurorack modules? That would make the question somewhat more interesting, since you wouldn't have to build the circuit (or pay someone else to assemble it) to hear how it sounds. It strikes me that SPICE-like algorithms should be fast enough to do this kind of thing in real time now.
Hmm. Examples of horrible alignmnent don't necessarily outweigh the fact that most people, most of the time, mostly behave in a way that is socially aligned. (Though I'm speaking in terms of intent, conveniently ignoring the side effects / negative externalities of our collective behavior.)
It doesn't need to have a habitable earth, it just needs atoms.
reply