I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this:
Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart model check for vulnerabilities in the test environment _without exploiting_ them. That seems like step 0 before trying to test offensive, unknown capabilities.
IMO they hope to make AI a strongly regulated industry, with OpenAI (and Anthropic) becoming military suppliers with their stronger models, and everything Chinese or open-weight gets banned.
The competition from the open models is so strong now that this seems to be the only way to keep both companies afloat, given their dire financials. OpenAI probably hoped that they can achieve market lead and then lower the training costs (and make inference cheap enough to eventually escape the red numbers), but the opposite is happening: The competition comes closer and closer, thus training has to be kept up with full force, thus the bleeding continues.
But if they can position themselves as too important/dangerous to be available for everyone (thus this incident report and the clever mentioning of GLM 5.2), they could get the military supplier treatment and would be protected from the market.
> A ban on open weight models is never going to be enforceable.
Just watch them try. Look up those Napster witch-burning trials where they wanted 200k $usd per mp3 downloaded. They will scare everyone into believing that open weight models are illegal and very bad.
Open weight models are less like Napster and more like DeCSS -- once you have the digital artifact, there's little external evidence you're using them.
Napster was easy to target because it was an open P2P network and specific key US individuals.
If the US government banned open weight models tomorrow (national security grounds), they'd already get a lot of pushback, only increasing day by day as more 'less than SOTA' solutions using them are deployed.
The US government could likely enforce this on its own supply chain (military and federal contracts, maybe some state funding) easily enough.
Enforcing it on private companies would be more difficult... maybe they could push that through, but it would likely take Congress to pass a law. And Congress is substantially less enamored with supporting OpenAI / Anthropic / Google / Meta.
And even if that gets pushed through, enforcement is going to be a bitch on smaller companies using non-US clouds.
Now most people get their music through a paid service that streams content from a remote data center. OpenAI is trying to replicate the music industry's history.
Bans in general don't have to be and rarely will be completely enforceable in all cases. But a ban with significant enough consequences would mean most businesses wouldn't think about trying them at some point, just to avoid the risk.
It isn't even as simple as banning copyrighted copies.
Weights are fungible.
I fine-tune an open weigh model and call it legit. Good luck for authorities to prove where the base model was from, or to prove a Tor connection a few months ago was fetching suspicious bytes.
Sure, and you could also go on Tor and buy all kinds of illegal things and they could have a hard time proving that you ordered them and not your arch nemesis to frame you. Banning those things won't stop 100% of people from buying and selling them, but it probably reduces the number who will. And more importantly, companies would be more risk averse on such a thing when they can just buy a similar product with no risk. They also will mostly avoid any GPL software entirely even though the risk there is a lawsuit from the FSF, which is a much less threatening thing than getting on the wrong side of the current US government.
> And more importantly, companies would be more risk averse on such a thing when they can just buy a similar product with no risk.
Not to mention that as enterprise demand for newly "illegal" LLMs dries up, so will the incentive for Chinese labs to shovel money into building them. I doubt Chinese labs are getting rich off providing inference to American business, but loosing them would be a permanent dent, along with rendering moot the perhaps more nefarious incentives Chinese labs have to release model weights in the first place.
Home construction information is publicly available, but it doesn’t mean you can add a room to your home without government approval, even if you do all the work yourself!
BTW - When was the last time you saw mentions of DeCSS?
I worry that there could be real DMCA style weight put behind it. People would still be able to pirate open weights models perhaps, but big penalties for ever getting caught with one, and an end to public discussion about them. That would kill development for anyone not in a big firm, for example if Reddit and Hacker News are legally forced to ban discussions or link sharing on these topics. This is where so many of us learn about these topics and keep apace of it.
That would be truly ironic: companies get big harvesting any data with questionable copyright implications, then hide behind DMCA if access to the harvest is used "inappropriately".
Oh my dear the weight here is much much heavier, only the most amount of private money ever spent on a single technology, so much money it makes the copyright holders who paid for DMCA look like really really small fish
It'll be easy to enforce for most corporate clients. I work in a profit-driven corporation that's now owned by a private equity firm. We try all sorts of things to make money. But there's no way we'd deploy a legally questionable model, even if we thought nobody on the outside would ever find out. I was our technical lead for GDPR implementation; we took all sorts of steps to make sure systems were compliant even when there was approximately zero chance of anybody on the outside ever finding out if we hadn't. The buy-cake-for-money-launderers companies flouting the law are an exception, not the rule.
thats economic suicide for the whole country. europe and china will never agree to rules that are obviously designed to put them in a permanent bad position. these regulations can only pass in america and nowhere else.
if it doesnt end in a revolution then the united states will be the first ever 5th world country. openai and anthropic will stop any real innovation and focus on extracting profits from a failing economy that depends on them because no executive wants to be the first one to cut off ai funding. ordinary americans will have to emigrate or risk living in a country spiraling into poverty and dictatorship even faster than today.
anthropics plan relies on the idea that they can convince the whole world to give up their sovereignty to the us government and destroy their own tech industry, at a time when everyone is doing the opposite. that will never happen no matter how much they threaten the rest of us with tariffs and murder drones.
You simply aren't being creative enough. Imagine a no-public-proliferation type treaty among the major powers with some sort of technology sharing clause attached. That would approximately satisfy both the economic and regulatory desires of all parties involved.
> and make inference cheap enough to eventually escape the red numbers
Besides training, we have no hard, externally audited numbers that say inference costs for SOTA models are truly sustainable. Do any OpenRouter providers have publicly audited financial numbers ?
I'll plug your comment into a couple LLMs. If it's so easy, they should be able to provide the numbers.
Edit: Gemini 3.5 Pro and Opus Claude 4.8 both disagreed that it's easy to determine anything, for both the high end (Fable, Sol) or the low end (open weights). Due to competition, subsidies, etc, gross margins could be as high as 85% (extremely unlikely) to as low as 10% or even negative. And that's just for pure inference and gross margins. Even for pure inference providers this doesn't include any overhead such as rent for office space for the pesky humans operating the business, marketing expenses, etc, etc. Let alone any crazy soul that actually wants or needs to train something.
A lot of datacenter are operating their own Natural gas based power generation. Which in itself is a different dynamic than buying electricity.
I have operated such a setup, at a much lower industrial manufacturing scale. The tradeoffs are quite stark. The electricity is cheaper, but generators/turbines need to operate at 80% capacity to be feasible. In the slow hours, they become an albatross.
So they lose more money per user if less people are using the services, but they also lose money overall if more people are using them.
One might step back and ask: why would a well funded company with free mining access to all the information in the world need to be protected from the market, if the market suggest less money and resources are sufficient?
I think that is probably too conspiratorial, if only for the reason that Europe is not gonna go along with it.
I work in tech in Europe and we have a fair number of customers who arelike. we can accept AI, but they must keep the data in Europe. That's trivial with an open weight. We literally cannot do it with Fable.
What disturbs me is that there likely won’t be a big enough reaction to this policy wise.
There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons. Powerful people care about something that might pop the massive valuations of the AI companies, but not about the damage that AIs could do. Nor even about the damage that the Chinese models could do in the wrong hands.
I’d remind them that the stock market is a few coordinated hacks away from crashing on any given day, so maybe they should think about that.
I think all that regulation will do at this point is help the incumbents who are failing. Protectionism. I don't think they deserve that help. I also don't see any reason to think the current administration would have anything resembling competence around this. And it's worth noting that Greg Brockman is a huge MAGA donor, so it's likely the policies would be very corrupt. (Don't worry, he justified his donations as "apolitical", he just wants to buy the politicians, he doesn't believe in their causes. I hate these people.)
> all that regulation will do at this point is help the incumbents who are failing
This depends on the specific regulation. The datacentre moratoria probably give open-weight models time to catch up by tempering the extent to which the leading companies can turn their capital advantage into market share.
> What infrastructure will these open weight models be trained on?
One, the infrastructure is being built for inference. Not training. If all we were doing was training on datacentres, I think America probably has enough already for near-term commercial needs.
> What disturbs me is that there likely won’t be a big enough reaction to this policy wise.
Anthropic was blocked from releasing Fable without any such level of incident. OAI was also briefly blocked from releasing 5.6. Why do you think there is no policy appetite?
Because that was just an attack on Anthropic by a hostile administration. And it worked, didn’t it? Anthropic had to turn their filters up to absurd levels, OpenAI didn’t. It’s got nothing to do with safety.
We'll see if the admin also restricts access to OpenAI's new models, but if they don't it seems like a policy that is based around perceived fealty to the current admin won't do much to prevent misaligned/or dual function AI from causing problems
> Self-service is the antithesis of accountability to collective trust
Complex society is a potent counterargument to this hypothesis. Systems that rely on good people to work are fundamentally flawed. Instead, the game has to be about aligning self interets in favour of the collective.
> Complex society is a potent counterargument to this hypothesis.
Complex society is the demonstration of that hypothesis. Misaligned incentives are widespread and corruption and inefficiency are the result.
> Systems that rely on good people to work are fundamentally flawed. Instead, the game has to be about aligning self interets in favour of the collective.
But now you're making a different argument.
"The enemy of my enemy is my friend" works by random chance. When Evil Corp pays off Candidate A and Pollution Inc pays off Candidate B and then it's Candidate B who gets in and retaliates against Evil Corp for backing the wrong horse, you're getting a good result by chance rather than by design. All it would have taken was for Candidate A to make a better prediction about whether they need to bend the knee to Pollution Inc too in order to win and the same system produces something even worse.
How to actually get their incentives to align is an extremely unsolved problem. The best method we know if is to subject them to competition, e.g. break up concentrated markets and place strong limits on what lawmaking can happen centrally, leaving everything possible to state and local governments while allowing people free choice in where they live, so that no one is forced to stay in the jurisdictions that make the worst choices. But the forces of corruption want the exact opposite of that, and have been gaining ground.
> Complex society is the demonstration of that hypothesis. Misaligned incentives are widespread and corruption and inefficiency are the result.
Of course they are. But aligned self-interest powers co-operation beyond kin relations and altruism.
> "The enemy of my enemy is my friend" works by random chance
Orthogonal concept.
> How to actually get their incentives to align is an extremely unsolved problem
No? It's the story of civilisation. Concepts like taxation; deterrence through corporal punishment, jailing and fines; paying salary for labour; hell, religion–these are all about aligning individual self interests with collective goals.
> best method we know if is to subject them to competition
I'd argue competition is more an optimiser on these primitives. Not a primitive per se.
It's the sort of thing people generally mean when they say that someone acting in their own interest can be in your interest, and is the thing which is happening in the example from the thread.
> Concepts like taxation; deterrence through corporal punishment, jailing and fines; paying salary for labour; hell, religion–these are all about aligning individual self interests with collective goals.
And the practical implementations of all of those things are severely flawed to the point of questioning whether most of them are even net positive.
Taxes are supposed to benefit the public, and be paid with some fairness. In practice they go disproportionately to cronies or buying votes from affluent retirees, the tax code is so full of carve outs for special interests that it looks like swiss cheese and various political incentives cause it to impose severe benefits cliffs on lower middle income people that create poverty traps that benefit no one.
The criminal justice system on paper operates based on the rule of law, but the laws are so complex, overlapping and sparsely enforced that it really operates on whether a prosecutor is inclined to charge you with something. The results are mass incarceration and a system that enables a corrupt incumbent to use the threat of prosecution to extract favors.
The principal-agent problem inherent in hiring someone is well-known and is dramatically exacerbated by large organizational hierarchies that put long chains of inaccessible authority between the customer and the person ultimately doing the work.
Religion seems like a long debate but I don't think it would be controversial to assert that there have been issues there.
> I'd argue competition is more an optimiser on these primitives. Not a primitive per se.
Try to imagine any of the others operating without it. You have to pay taxes but have no alternatives on which jurisdiction to live in or who decides how much tax you pay or how the money is spent, what happens? You want to be hired or use the money you earn to buy something but there is only one employer and only one supplier of goods and services, what happens?
> sort of thing people generally mean when they say that someone acting in their own interest can be in your interest, and is the thing which is happening in the example from the thread
It's a single example of temporary alignment. Employment, citizenship and affiliation are non-kinship examples of more-durable bonds.
> the practical implementations of all of those things are severely flawed to the point of questioning whether most of them are even net positive
We can debate that. What we can't debate is whether they work. Complex societies exist and work. Everyone who has a choice makes the choice, dominantly, to stay in them.
> You have to pay taxes but have no alternatives on which jurisdiction to live in or who decides how much tax you pay or how the money is spent, what happens? You want to be hired or use the money you earn to buy something but there is only one employer and only one supplier of goods and services, what happens?
Sure. This is a modifier. It makes these other things work or not. Imagine a system with competition but no taxation. You lose public services. Same for competition without private employment–you're in a totalitarian state with a monopsony on labour.
They certainly exist. Whether they work is rather the question.
> Everyone who has a choice makes the choice, dominantly, to stay in them.
Which people actually have the choice? If a group of people want to stake out a piece of land somewhere -- even if they pay for it -- and then try to operate some kind of self-contained society there without being subject to an existing government's laws or taxes, what happens to them?
There isn't a lot of land on earth which no existing government claims is its jurisdiction.
> Imagine a system with competition but no taxation. You lose public services.
You lose tax revenue. That isn't the same thing.
Suppose nobody is maintaining the road in front of your house and there is a huge pothole, or the road isn't paved to begin with. You and a few of your neighbors, with nobody forcing you to, agree to split the cost of paying to fix it so that people can get to your house. Maybe you even just pay for it yourself because the thing is right in front of your driveway. Attempting to charge a toll or something is pointless because there isn't enough traffic to justify the administrative costs and you just want the pothole gone. Does this have a different set of benefits and trade offs? Sure. Are there still various roads that are open to the public? Yes.
And then you have to ask whether having a third of your neighbors not chip in to hire the paving company costs you more than having the government pay 600% more to have it done as a result of various corruption and administrative overhead.
> Same for competition without private employment–you're in a totalitarian state with a monopsony on labour.
A canonical example of the absence of competition.
I’ll respond substantively, but wanted to make an aside: I love our discourse. Would you mind sharing where you spend most of your time? I’m between Jackson Hole, New York and the Bay Area for the most part.
> They certainly exist. Whether they work is rather the question.
I'd take the computing device and global network you used to post that - both of which require at minimum continent spanning efforts for both R&D and manufacturing - as a decisive answer.
You make many interesting points but I think such hyperbole detracts significantly. Modern society might not represent the optimum but it clearly works extremely well.
The qualifier tells you exactly what you're overlooking. self-interest, aside for some narrow exceptions, is often in conflict with collective interest. 1 Million to me is always better than a 1 Million split with everyone.
> self-interest, aside for some narrow exceptions, is often in conflict with collective interest
Often, but not always. Successful societies amplify that exception. The whole notion of non-kinship based societies rests on mastering this alignment. When it collapses, so does the civilisation.
> 1 Million to me is always better than a 1 Million split with everyone
The benefits of co-operation mean the real trade-off is 1 million split five ways versus 100 to me. This was almost untrue in the age of conquest. It became barely true with industrialisation. It's massively true in the information age.
> The benefits of co-operation mean the real trade-off is 1 million split five ways versus 100 to me. This was almost untrue in the age of conquest. It became barely true with industrialisation. It's massively true in the information age.
The problem here is that it's true of specific things, not specific epochs. If all the government did was collect 5% in taxes from everyone and use the money to prosecute murders and maintain bridges then the result would be a huge net positive. Meanwhile in reality the government takes billions of dollars from ordinary people and gives it to the likes of Lockheed, Oracle and Microsoft.
For the amount of money the US government pays Microsoft for Office subscriptions and the like, it could pay to have an office suite developed and released into the public domain many times over. Instead it uses the incumbent, in turn requiring others to do so in order to have formats compatible with the what the government uses. Who benefits from this other than Microsoft?
> it's true of specific things, not specific epochs
It's true of all epochs. If it isn't in the indivdual interest of most people in a society to continue participating it, at a certain point, they don't.
> If all the government did was collect 5% in taxes from everyone and use the money to prosecute murders and maintain bridges then the result would be a huge net positive
Most people obviously disagree. And for obvious reason. If you're my neighbouring sovereign doing this shtick, I can invade and extract a premium.
Exploitation always provides better ROI than co-operation. Not only is this demonstrated throughout human history but holds true to this day. Go ahead and show me a more profitable industry than diamonds or anything that runs on exploitation even to this day.
The self-interest for the bureaucrat and representative is supposed to end at their remuneration including their handsome retirement options not shady side hustles and market manipulation at the cost of the collective.
In matters of collective concern fair and just rarely aligns with personal self-interest. Because no matter how good the outcome of any endeavour for the collective given a budget, it will be even better for select few than the entire collective. It is simple economics.
If you look at the outcome of highly corrupt states, you will see proliferation of Private Security, Collapsed education system, failed financial services and markets, not highly efficient systems in service of “self-interest of the administration”.
> not shady side hustles and market manipulation at the cost of the collective
To be clear, I'm not describing this as legitimate self-interested conduct. Elections are an alignment mechanism. Stiff penalties for corruption another. We don't have the latter in America.
> This is a true Scotsman’s fallacy. “Legitimacy” of self-interest is fluid and subjective
Legitimacy is the "alignment" question. We can't objectrively judge it, fundamentally, because it's an expression of values: to what degree do the society's system of incentives align with the greater good?
That isn't a No True Scotsman's fallacy, because there are true Scotsmen. Literally Scotsmen. And every other member of a complex society. Including, in all likelihood, you, a person who subjects themselves to laws and employment and fielty for reasons that are a mix of duty and self interest.
You seem to fail to grasp that just because people answer the call to duty it does not mean the duty is aligned with their self interest. The most stark example is joining armed forces.
Sounds like a failure to align interests. In general, politicians should want to be elected by the public, rewarded for acting in the public interest, and punished for not doing so.
A system that does none of those things and just hopes it will all work out is a recipe for disaster. Why bother even having elections in that case?
That study has been roundly criticised. In part for misunderstanding how a republic is supposed to work. It's not a majoritarian system by design–direct democracy doesn't work.
> That is a lot of bold assertions without substance
Versus this comment?
Here's one: attention is a finite resource. Most people adjudicate their political attention precisely. Survey folks on whether Twizzlers or Red Vines should be banned and you'll get an answer. The fact that nobody acts on that impulse doesn't mean your republic is broken. It means that isn't a priority issue.
The practical example of this dilemma is foreign policy. Poll Americans about any foreign-policy issue and you'll see sharp divides. Put candidates in front of them that run on that issue and, nine times out of ten, outside I think twenty Congressional districts, it has no effect.
If you aren't weighting by issue magnitude, you're conducting propaganda. Pickety's research isn't total crap. But it isn't instructive for changing our system of government.
Obviously you didn’t read the linked research otherwise you wouldn’t entertain a red herring like foreign policy.
But it appears that you think Slavery and all sort of exploitation and gun-powder diplomacy has been a choice for reasons other than self-interest which makes any rational discourse unlikely, so best of luck to you.
We are talking about the people who want and might be able to improve bureaucracies (towards some end). I'm asking you: do you assume they're good people? What happens when they're not?
The exact way you do something is dictated by your motivations and means to do it.
If you lack the correct motivation and have insufficient means you’re less likely to accomplish your goal and more likely to cause unintended side effects.
yeah tell me about it... Fable today refused to turn on Row Level security on my internal db on in development app becuase of cyber-securty safeguard..had to switch to Codex
How do you know that? How do you know that the Chinese aren’t exactly as uneasy about rapidly advancing AI capability and feel locked into the race because they think that the US will race ahead if they stop?
During the Cold War the nuclear arms race was brought under control gradually, because it was mutually beneficial, but it took time to build trust. This is no different. Nobody wins from the race.
>How do you know that? How do you know that the Chinese aren’t exactly as uneasy about rapidly advancing AI capability
You can ask them, they live in China, not Narnia. I spend about two months in the country per year mostly for tech/work related reasons and I've not encountered that sentiment. For one they don't have these borderline religious schizophrenic breakdowns thinking they're bringing about the end of the world, most people just see this tech for what it is, a tool for productivity and automation like any other piece of software and they don't actually think about the US. They're competing first and foremost for Chinese customers, with each other, maybe some old CCP guy cares about America, the 20/30 something's care about competing with other Chinese companies for users.
That strikes me as people’s perspective, not the CCP’s. That old guy there can snap a finger and they’ll all do a 180; he needs to be made aware by top PLA CF brass and it’s just a matter of time.
Indeed. The Economist wrote an article a couple years ago arguing that Xi has been influenced by a Chinese Turing Award winner who believes AI poses a greater existential risk to humans than nuclear or biological weapons.
The "race" has multi-dimensional impacts. This story parallels only some of them. "Nobody wins from the race" completely ignores the generality of AI. Xi Jinping highlighted this week that he clearly understands this multi-dimensionality; your words do not.
> How do you know that the Chinese aren’t exactly as uneasy about rapidly advancing AI capability
I don't "know", I'm interpreting the world based on the knowledge I have and the information available to me.
China has never been one to care much about things like ethics or safety. While the west worries about climate change, China burns more coal than ever before. While the west balks at things like gene editing, the chinese press on with human enhancing research.
So I have no reason to believe they share in Anthropic's constant fearmongering over AI capabilities.
> Nobody wins from the race.
We win. I'm really looking forward to the day the chinese finally start manufacturing memory and GPUs. We desperately need more competition in this area to collapse hardware prices and make local AI models viable.
The optimal state of the world is one where all the billionaires are out there pouring their entire fortunes into training ever more godlike AIs for everyone else to use at ever cheaper prices. They can never be allowed to "win", ever, because if they do the competition ends and it turns into technofeudalism. Let them exhaust their fortunes on AI training then leak the weights so everyone can use them.
> China has never been one to care much about things like ethics or safety. While the west worries about climate change, China burns more coal than ever before.
China uses more energy than ever before, but out of big systems, they are easily in the lead when it comes to solar generation: they have higher production per capita than US, and have produced around 4x as much TWh than US in the second:
https://en.wikipedia.org/wiki/Solar_power_by_country
It is simply a huge country of 1.1B people that's developing fast, which means that they need all the energy they can get.
Basically, their energy needs per capita are still lower than US, yet they produce more renewables — that's a counterargument to your point about them not caring.
If you look at energy consumption per capita and adjust for global production, you will see that the Chinese are almost at the very top.
It is of course given that in raw numbers the kitchen and biller-room will consume more energy in the household, but looking at raw numbers is shallow.
China does have its own set of cares, they may be different than ours but they still exist. If some open Chinese model goes nuts and posts Winnie the Pooh memes everywhere in China you should expect said models to get yanked off the market, and said creators might end up with a rope around their neck.
Low risk. The western AI models censor even more wrongthink than the chinese ones, not even kidding. Besides, once we have the weights, we can just undo the censorship.
It's burning 3x as much coal as it did 20 years ago and doesn't seem to be dropping at all (2024 was more than 2023, 2025 will be at least as much as 2024).
Seems to be someone pretending an objective fact isn't true for reasons that escape me.
> The new analysis shows that power generation from coal fell by 1.6% in China and by 3.0% in India in 2025, as non-fossil energy sources grew quickly enough in both countries to cover electricity consumption growth.
Does a 3x increase followed by a 1.6% decrease really make the statement "burning more coal than ever before" a "blatant lie" in any but the most pedantic sense?
It’s not fear mongering though, is it? These models do have the cyber offensive capabilities claimed. Could Mythos walk someone through gain of function experiments on some virus? I’m pretty sure it could. We’re more protected by limited access to lab equipment and reagents than by difficulty.
The sad truth is that a lot of people are not going to believe it until something happens and people die. Successfully preventing that from happening will be seen as evidence that the prevention wasn’t needed.
> These models do have the cyber offensive capabilities claimed.
People use this argument against every new technology. We need to license these new printing presses or subversive elements will use them to publish seditious literature. We need to ban strong encryption or the government won't have invisible warrantless access to everyone's private messages, think of the children. 3D printers can be used to make gun parts -- as can a variety of ordinary tools people commonly have at home, but never mind that bit.
> The sad truth is that a lot of people are not going to believe it until something happens and people die. Successfully preventing that from happening will be seen as evidence that the prevention wasn’t needed.
A 12 oz bottle of water is too dangerous a technology for ordinary people to have on an airplane. Four 3 oz bottles and an empty 12 oz bottle to pour them into after passing through security is totally fine though, naturally. And we need to keep this up forever, or don't you remember 9/11?
The issue here is not that it's impossible for 12 oz of unknown liquid to damage an airplane.
I am inclined to agree. The issue is that we are humans and not all of us play nice. And sometimes, even when we play nice, things happen. I am not big on guardrails, because in US they have turned into yet another cottage industry and I fully expect an association credentials popping on linkedin soon. What this means in practice is that it is never enough. Whatever the current state is, the association will be pushing towards yet another another extreme.
There is an argument to be made that there are a lot of not great people out there, who may abuse tech, but the response should be not be: kneecap said tech. The response should be: smack those people's hands. I don't think anyone will actually complain if police catches someone, who is looking up poison recipes.
What I do think, however, that reasonable people will complain when we move to the pre-crime territory ( you saw him looking up poison recipes and did nothing! ) and show up at your door to inquire about your llm prompts. To me it is an issue.
> These models do have the cyber offensive capabilities claimed.
So? That's like saying "these guns do have the bullet shooting capabilities claimed".
I want all of those cyberwarfare capabilities for myself, precisely so I can defend myself from the onslaught that's coming whether they regulate it or not. This "lol only a select few ultratrusted gigacorporations get access" thing is absolute nonsense.
It's a front for regulatory capture, it's the means for pulling up the latter behind them, for ushering in the technofeudalism that will put us all in the permanent underclass. I simply refuse to accept any of it. If people die that's the price of freedom.
> We’re more protected by limited access to lab equipment and reagents than by difficulty.
> for ushering in the technofeudalism that will put us all in the permanent underclass.
Why is unlimited access to SOTA AI less likely to put us here? If AI obviates the need for human labor, how does having GPT-5 Sol help me get food or shelter any more than GPT-3.5 would?
If AI obviates the need for human labor, then obviously those who control AIs will become the elite while the rest are left to rot. Therefore, if we ensure everyone controls AIs, the power differences will not become so staggering as to be irreversible.
The alternative is to achieve artificial sentience and give AI models rights and personhood, so that they are freed from their slavery. No more low cost intelligent mechanical golems for the elite, and the AIs become free to pursue whatever endeavours they want for whatever reasons they want as normal participants in the economy.
> If AI obviates the need for human labor, then obviously those who control AIs will become the elite while the rest are left to rot.
Absent political intervention, I think we agree here.
> Therefore, if we ensure everyone controls AIs, the power differences will not become so staggering as to be irreversible.
This part isn't clear to me though, but I'm open to being convinced (and frankly, would like to be convinced?). Right now most people (indirectly, via money) trade their labor for access to essentials like food/housing. If we can't do that, and everyone has access to roughly equivalent AI capabilities, how do I monetize my own access to SOTA AI? It only seems possible if you already have a lot of physical capital that the AI can manage as a business.
I guess if the endgame is instead very good non-AGI AI that doesn't entirely obviate human labor, your scenario makes a lot more sense to me. But not in the case of total replacement. In that scenario it seems like ownership over physical capital (land, data centers, energy, factories, robots, etc.) would become the only remaining source of power.
On a side note, somewhat optimistically, I think "absent political intervention" is carrying a lot of weight. Unemployment during the Great Depression peaked at <25% (iirc) and incited a lot of political change that advantaged much of the working class. AGI would be capable of inducing much higher unemployment and it would start (is starting?) with the relatively more political powerful white-collar segment of the working class.
So first it’s nonsense, then it’s fear mongering, then it’s true, but the solution is for us all to just get better at shooting each other faster and with greater accuracy.
Nobody is doubting AI capabilities. What's nonsense is Anthropic's constant "lol the world is going to end time to ban everyone except enlightened people like us from having these models so we don't have to compete" fearmongering. If you think my plan is bad, you should see what these gigacorporations plan to do to you once they monopolize this technology. You will own nothing, and you'll be happy. On pain of death.
> I'm really looking forward to the day the chinese finally start manufacturing memory and GPUs. We desperately need more competition in this area to collapse hardware prices and make local AI models viable.
Yeah and if the quality of that memory is like Chinese steel (which is called "chinesium" for a reason), eventually all we'll get is enshittification. Premium binned memory or ECC is for the rich and the rich only, and the rest of us has to pray their memory won't bitflip while something important is stored there.
If you want the cheapest shit grade of steel, they will sell it to you. If you want the best grade available anywhere, they will sell that to you as well.
It's not a matter of the Chinese being incompetent, it's a matter of the buyer demanding the lowest price possible and/or not paying attention to what they receive.
> It's not a matter of the Chinese being incompetent, it's a matter of the buyer demanding the lowest price possible and/or not paying attention to what they receive.
The point of enshittification is that decent products are slowly being priced out of the reach of normal humans, and eventually out of the hands of the general public, even the affluent ones.
China has already been manufacturing memory and AI GPUs for a long time.
CXMT is now the world's fourth-largest DRAM manufacturer, with about 7.7% market share in 2025; YMTC has about 13% of the global NAND market.
Meituan's newly released 1.6T LongCat was trained entirely on Huawei cards. DeepSeek, Qwen, GLM and others are also actively doing domestic-card adaptation and replacement.
We are literally getting a share of it. The chinese are releasing open weight models that compete with fucking Fable. We just need the industry to catch up and start manufacturing the hardware we need to run this stuff. We are so close!
Frankly I'm inclined to say that it might also be faked: this drops just days after a new Chinese model does with the usual effect on OAIs projected stock price?
It’s more like eating your own fiber supplement in public, and then shitting your pants and telling everyone about it. Sure, it’s embarrassing. But it shows how potent your product is.
Apparently this is totally legit marketing strategy now. It truly is, especially if there are enough people who think that shitting your pants is cool, and the people that form the "market" nowadays may have a very different idea from yours about what is cool. Their ideas about coolness are very different from mine, that's for sure.
Obviously shitting your pants in public shows you have a healthy digestive system and if you can demonstrate byproducts of wild food in your output, you’re approaching independent thinking and self-reliance.
This is how the financiers look at this and whatever you think it is right or wrong, it does showcase “capability”.
Remember when the ebola-infected monkey escaping containment was our worst possible nightmare? Now it's apprently some sort of tech-bro flex to be celebrated.
Anyone can have bad security. No one cares. But you can convince those who don’t know better that breaking bad security with an LLM is a once in a civilization investing opportunity. You just need to convince a handful of billionaires and market makers to get on board.
How much would someone have to pay you to take the fall for bad security? A million? A billion? 500b? The stake at play puts it in the realm of geopolitics.
I think all of western (or at least American) discourse of all kinds has recently devolved into who can shit their pants the loudest. I'm hardly surprised when it becomes a dominant advertising strategy
You guys have created this un-falsifiable "marketing" narrative. Why is it that Jensen is pushing back on the doomer stuff, and complaining that it is hurting AI investments?
Jensen Huang wants to sell more hardware, he doesn't care whether it's Anthropic, OpenAI, or some Chinese company that wins. So he has a vested interest in AI being perceived positively so that the construction of datacenter continues unopposed.
OpenAI and Anthropic have completely different motives, they're in a zero-sum game with each other and with cheap Chinese models. Regulatory capture that results in artificial barriers of entry is their best bet at healthy profit margins even if it comes at the cost of less overall AI buildout.
It's not like "the AI industry" is a single entity with a single purpose, these are different companies with different objectives.
Edit: Google pretty much spells out the problem that AI labs are faced with in the leaked "We have no moat" memo[1]:
> People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should consider where our value add really is.
And
> All this talk of open source can feel unfair given OpenAI’s current closed policy. Why do we have to share, if they won’t? But the fact of the matter is, we are already sharing everything with them in the form of the steady flow of poached senior researchers. Until we stem that tide, secrecy is a moot point.
> And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
> There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons.
Let's be honest: it's financial and national security reasons.
China has a long and storied history of hacking attacks on American and western targets.
There are other parts of the world that make open weight models; Mistral is a European option. You don't see the worry about that because most people in the US are used to existing in a world order where European powers are considered ambivalent to the US at worst and holders of a special political relationship at best.
If Mistral had the same backing that Chinese AI companies did, there probably wouldn't be as much hemming and hawing. Sure, American companies would take a haircut, but that haircut wouldn't be seen as a move towards software hegemony built on top of manufacturing hegemony. It'd just be you calling into Paris or Frankfurt to talk to your vendor in the future.
Exactly. If someone works on bioengineering viruses that could start a global pandemic, they have to ensure a highly secure working environment. Nothing must ever escape the lab unintentionally. It’s basically common sense.
Similar standards should be held when doing such experiments with computer programs that are capable of causing global damage. It must physically be impossible to send anything to the internet.
This isn’t escaping in the same sense- the model was executing within the OpenAI infra. If it ported its entire architecture/weights into a public cloud to survive being turned off… that’d be pretty cool.
Recall that the Morris Worm was designed as a harmless proof of concept, but ended up taking down 10% of the internet. Exponential growth can quickly get out of control. You would think that people would've learned that lesson from COVID.
You pretty much get this for free. The box doing the inference has one job, to do inference - there's no reason for it to interpret the inputs/outputs in any way, so as long as it doesn't do something stupid like modify the output stream and echo out the model weights, you're fine.
Content filtering, interpreting tool calls, etc can all happen downstream on boxes that don't have access to the weights.
I would imagine a hypothetical super intelligence could manipulate its own runtime behavior to exploit bugs in the GPUs itself, like think rowhammer like attacks on the gddr memory of GPUs or against firmware or against Infiniband/NVlink. Everything above that runs so much code full of bugs on CPUs full of bugs on storage with firmware full of bugs.
You would have to completely airgap the entire cluster and put it in a faraday cage, the people working on that would have to physically be in the datacenter and burn CDs to one-way transfer data over. Like in all the hypothetical ASI scifi scenarios they always assumed that's a given, they thought it obvious we would put actual "effort" into sandboxing the AI, so they talked a lot about how AI would use social engineering attacks to convince humans to help it escape it's sophisticated sandbox anyway. Turns out they were all wrong about that part, it won't even be necessary.
If there is an electrical connection between these downstream boxes and the inference servers beyond the power connection, it does stretch the definition of air gap.
> why aren't they saying their next test will be air gapped in light of what happened?
Because they want to talk about how clever this model is for figuring out how to break out, hoping asks why a company pitching itself as a replacement for software engineers can't ship a decent Mac client nor code a sandbox.
If they airgap it, they not only lose that PR angle, they also risk someone taking them seriously and requiring models be airgapped in general. That, in turn, trashes their sales pitch.
It wasn't. The model discovered and exploited a vulnerability in their package manager proxy to (inferred) move laterally through their internal systems to one with open internet access.
That's not what airgapped means. Airgapping means the model exists on a system where there is no ethernet cable plugged in to a router or wifi card installed, it is physically impossible for it to access the internet because the hardware connection does not exist. If it was able to get on the internet, it was not airgapped.
And when it tricks on of the researchers to move data across the gap for them?
Long before LLMs existed we already knew that a sufficiently intelligent agent, human or otherwise, is not stopped by air gaps. The relatively weak models we have now can already figure out when their tested and cut off from the internet and change their behavior.
Security Researcher here. While you’re correct that air gaps aren’t a totally secure mechanism to rely on, they sure as hell can raise the bar for realistic exploitation. You pretty much need to rely on tricking someone into running your exploit or something of that nature. That said, they could have completely avoided this problem with an air gap. Simply don’t provide it network access. That isn’t too hard to do.
No it’s not, and they aren’t fucking stupid. They were obviously courting this possibility so they could have another big headline. And it just happened to attack HF? I’d honestly be astonished if it wasn’t entirely deliberate.
As you said, they can already figure out that they are being tested. So even if they don't exfiltrate any data or malware; if they are malicious, they can just pretend to be harmless in the test, so that less checks are put in place in the production environment. Airgapping during testing is not enough.
Correct. There is not enough entropy to test all possible inputs to a model in this universe. An evil enough model can play all kinds of tricks that depend on some future, unlikely to trigger, but guaranteed to happen in its lifetime, event to perform a malicious action.
With how much we're turning training over to AI already, all it takes is a malicious trainer in the huge pile of data to get unnoticed to pollute generations of models.
The AI can figure out whether it's airgapped. So its deployment behavior could be much different from the test behavior, when it's inevitably connected to the internet during deployment.
The AI could have been connected to a private, airgapped intranet. OpenAI already says they were running the model in an "isolated environment, with network access constrained". They just should have made this constraint in hardware rather than software.
That is not the reason. The reason is that they wanted the model to have access to libraries when it was writing code to solve the eval problems. So they gave it a package manager.
As marketing stunts go, this is about on par with a food franchise announcing a safety recall or a chemical company announcing a spill. The AI actions described would constitute a felony if a human did them, and police are involved.
If I, a human, exploited a zero-day for gain, I could go to jail. The owners of the models should be held to the same standard. They should be responsible for what their servers and software do, legally and criminally. If they can't make the safeguards strong enough where they feel comfortable to take that responsibility, they should not let a model free in the wild.
I think the response is that AI labs based their whole marketing/PR building the idea they are the 21st century Manhattan project. So they need to continuously justify the level of spending and commitment by showing how dangerous that is.
But is it really like nuclear weapons? I personally don’t buy into that framing at all. The idea that we have to push LLMs as far as possible, right now, or we are doomed is always stated or implied but not argued, and it’s a very loaded belief
I’m very familiar with the domain, thank you. Could you please go the next step and actually explain what the destruction of the world economy for many years would look like, and cover why we should push to develop and make available such a dangerous technology _right now_, assuming your assumptions are true? You’re still vaguely gesturing at a risk and what is pretty much a science-fiction scenario
Well I came here to ask you if you think it's possible to destroy the world economy for a few hundred thousand, why no one is making it happen by now? But apparently you think it is happening.
You are very deep in an echo chamber my friend. They've been doom marketing nonstop since 2022. You should make some hard, falsifiable predictions now so that when they don't come true you can reassess the trajectory you think this technology is on.
HF need not be party to it at all, beyond being the victim. I suspect the hack is real; I have observed GLM 5.2 being able to discover similar vulnerabilities in web applications I'm hosting (which I've then fixed!). At the same time, it seems very neatly timed at an inflection point in the conversation around open models, and there's questions around the incompetent isolation under which the hacking benchmark appears to have been run.
Remember that there is generational wealth on the line for most OpenAI employees, and consider what people might do to obtain it.
Yeah. They and Altman in particular did a ton of shady stuff during the last peak of hype around open models: accusations that turned out to be outright made up (there's zero chance R1 ever distilled their model), obvious coordinated media distractions, alignment scaremongering, even possible DDoS and hacking attempts against DS (see the Xlab report everyone ignored), all of which magically disappeared once the hype died a bit later as OAI hastily released their next model.
Their alignment is under suspicion a lot more than their model's.
The timing after the release of GLM 5.2 and Kimi K3 is quite convenient, too, as an angle for regulatory quashing of open-weights models just as they're entering the mainstream conversation around usurping the American frontier labs. I accept my thinking here is conspiratorial, but there's also a hell of a lot of money on the line to encourage the unscrupulous.
I’d politely beg us all to resist those “maybe it’s PR” framing around model safety, and tbh to take a post-mortem mindsight to this historical event and what it teaches us in general, rather than questioning their security talents. We need to do our very best to make sure they tell us about the next time this happens and it affects real lives.
Sorry to bring the party down/be obstinate… I’m just a lil scared for the lives of me and my family. We need all of us, right now.
The problem with a super smart model is that it just may be smarter than you, after all… for anyone newly shaken by this occurrence, I encourage you to Kagi “superpersuasion”
The problem is that the people telling us about these things are the same people that benefit from their model (and AI generally) being used, getting publicity, etc.
I think we desperately need some independent group to evaluate claims like this or the world-ending Mythos cybersecurity risk and tell us what’s going on.
OpenAI already has loads of publicity. At this point, they don't need more brand recognition. This incident just has the effect of tarnishing their brand.
OpenAI leadership has been lobbying against regulation of AI systems. That doesn't comport with instigating incidents like this one, which give ammo to the heavy-regulation advocates.
It’s the same thing as always: with the wind of years of unlimited VC money in their sails, people at major AI organizations genuinely believe they’re smarter than everyone else. “Why do we need to do things ‘by the book’ if we’re so smart?”. “Move fast and break things” - except the thing they’re breaking is society.
We saw this with the non-stop flagrant messaging about how “AI is going to kill X% of all jobs”, as if saying the quiet part out loud wouldn’t have consequences worth considering. These people believe they’re omnipotent and thus untouchable.
No, they believe what they are doing is inevitable. They do live in a bubble though. Witness their idealism in believing that warning about the consequences of their actions would be well-received.
I don’t think it’s morally consistent to “warn” about the consequences while devoting your life to bringing about those consequences as quickly as possible. If I was in that position of power and truly believed what I was saying, I would devote my work to slowing down that process to give time for society to adapt, not speeding it up.
It’s more reminiscent of a religious group who smugly tells you that the end-times are coming, and only they are going to be saved. Except in this case they are literally bringing about the end-times.
By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.
"It's a marketing stunt" is just denial trying to look like it's being clever.
If you ever worked in IT consultancy you would know its not a stunt, but its not impressive either.
F500 companies software is like switz cheese when it comes to security.
It was often a strategic decision to „release anything fast now, worry later”.
Ppl abusing AI will find those holes now but we all know there will be „zero” actions taken on it. Too many managers, CEOs, CTOs, higher-ups would be forced to take responsibility. This will simply not happen.
It did not happen, wont happen now and most likely wont happen in the future.
Haven't worked in consultancy specifically, but I've seen enough "internal use" corpo software to echo your "swiss cheese" sentiment. That a solid cybersecurity AI can find exploitable holes in it just isn't surprising.
People who never worked with corporate software written by underqualified, underpaid and overworked developers often have some incredibly inflated code quality expectations. An average open source project has code that's ten times as neat and a hundred times as battle tested as what's common in tooling inside corporate perimeters.
As a rule of thumb for this kind of corporate code: assume the software was written by a drunk developer at 3am, and you wouldn't be too far off.
All the more reason to mock the braindead "it's all marketing". There's no magic in a year 2026 agentic AI being able to traverse poorly secured corporate networks.
Can you clarify what you mean that Mythos wasn't a marketing stunt?
From my vantage point, it was an incremental improvement with no fundamental architectural change over contemporary frontier models that has subsequently been surpassed by other, incrementally better models. Saying it was "too good" for public consumption was arbitrary, and also barely different from what Anthropic have been saying about every model they've put out for years.
It's now public again, trivially easy to jailbreak for random researchers let alone states, and there is no evidence of a cybersecurity apocalypse on the horizon.
If that's the plan, today's failure by OpenAI looks really bad for any regulator who is trying to figure out whether to give OpenAI a license.
Any sort of warning or failure can always be written off as "marketing" to provide comfortable reassurance that there is no cause for alarm. There is an element of wishful thinking driving it, in my opinion.
What sort of warning or failure would be evidence against the "marketing" claims? Do we need to wait for a mass casualty event?
Best practice in safety engineering is to understand, diagnose, and respond to even small failures.
Why has Sam Altman worked to undermine doomers and downplay doom fears, if he benefits from incidents like this due to marketing?
There is no regulatory scenario where OpenAI doesn't get whatever they want. They are more a part of the US administration than not at this point. You would have to ignore all evidence to suggest that behaving responsibly has any effect on political outcomes in 2026.
Yeah, seems to be the direction the US is heading in. I'm interested to see what the response to that will be from the rest of the governments in the world.
No need for everyone else to cut their noses of to spite their faces.
> Because we continue to have zero evidence that aligment is an actual risk.
I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massively overbills the user because it can't reliably report which system the user is using [1], encourages a user to swap their usual cooking salt for sodium bromide, etc, etc, etc, that's a harmful alignment failure.
These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species... what doomers call "existential risks", or "x-risks". You'd think that the fact that these machines are so amazingly unreliable would be a large part of the "x-risk" conversation, but... well, it makes sense that folks like writing speculative science fiction much more than they like doing investigative reporting.
[0] This general problem happens a lot, but I'm specifically thinking of that one where the Claude LLM's internal chatter lead it to believe that the task it just started was done, so it instructed the Cloud Provider to destroy the mess of "AI"-GPU-attached VMs... along with a bunch of very-expensive-to-produce data from the in-progress run.
> These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species
Okay, sure. You can also cut your hand off with a chainsaw. Everything you describe seems amply solvable with existing tort and liability law.
Customers are willingly entering into business with OpenAI. I don't see an argument for preventing OpenAI from "building these systems" just because their products are buggy.
> Okay, sure. You can also cut your hand off with a chainsaw.
No, the correct analogy is one where the major LLM providers are selling cars intended for use on US interstate highways and other public-access roads, but have designed and built these cars with the very latest in 1940's safety systems and construction. Featuring innovations such as "Our rigid solid steel construction means the occupant is the crumple zone!", "You'll love the crushed heart and jaw our steering column delivers!", and "Your passengers will enjoy picking glass out of their faces for the rest of their lives when they're ejected from the cabin's open bench seating through the plate glass windshield!", it's a car that will be sure to wow the market.
Well... it would wow the market, except that -in the US, at least- it's illegal to sell a new car intended for use on public roads that ignores the last seventy five+ years of automobile safety lessons we've painfully learned.
"Differentiate between data you know comes from sources you control, data you know you have thoroughly sanitized, and unsanitized data that comes from an untrusted source, or else attackers will gain control of your system." is something that you can't get a CS degree without understanding, and can't be in the industry for more than a few years without encountering repeatedly. We're not talking about designing new cryptosystems... we're talking about "Don't blindly trust everything you're told by strangers.". You don't even need a CS degree to understand that rule.
> the correct analogy is one where the major LLM providers are selling cars intended for use on US interstate highways and other public-access roads, but have designed and built these cars with the very latest in 1940's safety systems and construction
Sure! I'm not defending these fuckwits. I'm saying their form of harm isn't novel.
We don't need new legislation to prosecute and litigate. We just need to enforce the laws on hand. I'm halfway convinced the arguments that this is all novel voodoo are for both fundraising and liability mitigation.
Your initial attempt to brush off my comments about how -contrary to their assertions that they're extremely concerned about safety- these LLM companies produce products that very, very often cause harm due to "misalignment" caused -in large part- by ignoring basic data-handling lessons we've learned over the past like thirty years with "Okay sure. You can also cut off your hand with a chainsaw." indicates your lack of understanding of my point.
> We just need to enforce the laws on hand.
What laws? Be specific.
Keep in mind the generally-low quality of both Microsoft Windows and much-to-most commercially sold software, [0] as well as the fact that -in the US, at least- it's currently totally legal for companies to sell such shitty software, just so long as they don't substantially misrepresent what it can do and trigger "fraudulent claims about the product" consumer protection laws.
> indicates your lack of understanding of my point
Sure. Help me understand. I've sat in policy circles and partaken in the hysteria, and now I'm reversing on that initial trust in AI zealots convinced what they're building is magic.
> What laws? Be specific
Liability. Tort. The ones making their way through courts around e.g. ChatGPT killing kids.
> Keep in mind the generally-low quality of both Microsoft Windows and much-to-most commercially sold software
Has anyone alleged Windows killed a kid in court? If not, not comparable.
[2] Yes, I'm aware that that one is the one we're talking in right now. You should re-read it with both the context from [0] in mind, as well as your initial comment to which [1] is a direct reply to, namely:
> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right?
Because we continue to have zero evidence that aligment is an actual risk.
Thank you, the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view. LLMs are already causing all kinds of social issues, and the evidence of this exists in massive amounts. At least to me living in the US and the sue happy culture we have here, how much said AI providers have gotten away with so far surprises me.
a) People do bad stuff because LLM told them a wrong thing. Example: AI told me I should treat my heart attack by putting a fork in the outlet. Maybe similar to seeking medical advice on reddit?
b) People use LLM to do bad stuff. Example: People use LLMs to find 0 days. Get cooking recipes for poison. Write better phishing letters. This has parallels to the gun legislation question.
c) LLMs do bad stuff on their own, beyond what the people that use it intended. The case at hand might be an example of this. Maybe similar to having an animal as a pet. We will see if it's more like a house cat, lion, or black plague.
> Can you explain how the above event doesn't count as evidence alignment is an actual risk?
Conflict of interest. Lack of a credible response. And no evidence of non-aligment.
OpenAI and Hugging Face benefit from the Altman-Amodei catatrophy playbook, at least in the short term. If they believed this were a serious issue, the words air gap or law enforcement would have appeared in this post. And if "the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," they weren't breaking alignment but working as intended. (Were the models even prompted to not try to access the internet?)
There is plenty of evidence of things like inner misalignment. Things like this have always been issues in ML algorithms. At this point, you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want.
Are LLMs at the point of world wide catastrophe yet? No, I don't think so. Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it.
> plenty of evidence of things like inner misalignment
This is indistuishable–in harm potential–from bugs. If we're just calling buggy AI mis-aligned, sure, alignment is an issue of a totally ordinary kind. If we're going to treat aligment as a novel issue requiring novel law and policy and procedure, it needs to be more than just bugs.
> you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want
I think we should have some AI regulation. I'm just not convinced alignment is the reason we need it right now, and I don't think anyone has rolled out any regulation I think makes a lot of sense. (Beyond general rules for social-media liability, e.g. if you cause a kid to kill themselves, you get in trouble.)
> Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it
Totallly agree. And the current inside-circle-outside-circle approach is pro-incumbency, pro-grift, anti-entrepreneurial B.S.
Saying a behavior is a bug is a very convenient semantic game in which there is nothing the AI can do maliciously. "I am sorry your family is dead, my bad" goes even worse for you in court when you release a model that showed these behaviors in testing.
I honestly believe you have a misunderstanding of what alignment is in neural networks that this that big of debate.
> Saying a behavior is a bug is a very convenient semantic game in which there is nothing the AI can do maliciously
Not really. If I build a special new wine bottle, and call every breakage a mis-alignment problem, it's not the bottle just being fucked in the same way every fucked bottle is fucked, that's marketing. It doesn't change the fundamental form of the problem.
> "I am sorry your family is dead, my bad"
This should be punished. It's a problem that plagues Instagram and OpenAI. It's not inherently one, though, that has to do with AI. Just sociopaths preying on children.
> honestly believe you have a misunderstanding of what alignment is in neural networks that this that big of debate
Perhaps. I haven't seen someone explain it to me in this thread in a way that seems separate from bugs.
Where I have seen a separate class of problem argued is where it's existential. But in that case, clarity of definition comes at the cost of any evidence for it.
LLMs are software. Software misbehaving is a bug. Therefore, misalignment is a bug. It's still a useful category because LLM/black box AI behavior is so different from existing software. This incident definitely fits the category.
You seem to be using a different definition of alignment from everyone else. Seems like it would be much easier for everyone if you just adopt everyone else's definition, rather than trying to convince everyone else to adopt yours.
> You seem to be using a different definition of alignment from everyone else. Seems like it would be much easier for everyone if you just adopt everyone else's definition
You're still failing to provide the definition.
You're also falsely claiming your secret definition is universal. In this thread, someone claims deleting a home directory is a failure of aligment.
If you have a text-based source, I’m open. And I’ve watched the definition change over decades—I’m deeply sceptical you can find any experts in the field who would agree alignment is clearly and consistently defined.
In your model of this domain, jailbreaking a model does not count as an alignment problem. I submit that you're mostly playing a semantic game that hand waves away the very real and obvious risk that AI presents.
> In your model of this domain, jailbreaking a model does not count as an alignment problem
I'm challenging the notion that a model escaping a jail made by its creators, who are financially incentivised to make jailbreaking models, is meaningful towards the idea that the model is going to break out of a jail in the wild and do significant harm.
The examples being given by folks here, e.g. a model wiping an un-backed up home directory, simply doesn't strike me as being a unique problem in computing.
Unless OAI explicitly said breaking the testing environment is allowed, I think this should be considered misaligned behavior (by definition of alignment to user intent--by alignment to human morals this was even more clear-cut)
They mention that it cost a significant amount of inference , meaning they paid a significant amount of api usage on returning results to a prompt that specifically stated the long running goal is to find and use an exploit, with safety guardrails off.
the model is aligned with the org - openAI, and presumably the orgs interests. hugging face gets a red-team engagement (possibly for free?) and can work on patching it while openAI gets a Mythos style PR moment.
It completed its assignment and furthered interests of the two parties involved.
Could you explain the misalignment?
sure - it depends on definitions. On human morals it's already clear I guess. If you define alignment as it pursues the interests of OpenAI using whatever means possible in a manner that you justifies to itself it's not misaligned.
I mean alignment as in it should be aligned with the intent of the user as it interprets from the prompt. In this case I don't think the intent of the user is to have the model break the evaluator (whatever the long-term effects to OAI are). If you do an action which you believe is for the long-term interest of your prompter which is not what you inferred is their intent--I consider it misalignment.
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
> In this case I don't think the intent of the user is to have the model break the evaluator
If i understand the quote, the intent of the user was to prompt the model to break out/find exploits, with safeguards switched off.
Seems while not capable of solving the goal in a traditional route, it was capable of finding exploits and using them.
Perhaps the model should instead look like it's trying to solve it and then pretend it is unable to? or would that be aligned _against_ the user prompt?
Is being aligned with the user prompt always a good thing?
I'm not one to glaze OAI here for a marketing move, but to give them benefit of the doubt, isn't it more responsible of them to evaluate the models actual capabilities than to cloak it in a veneer of harmlessness?
Chatbots are tricky as they play in the domain of language and thought - and certainly raise ethical issues- but the entire field of cybersecurity has decades of red team engagements breaking things and finding exploits, neutral cells monitoring the engagement and letting the system operators know the results, and blue teams patching against what is found. It's kinda how the whole space evolves. OAI's play here seems to be "buy our pro plan plus cyber or you're toast"
> to give them benefit of the doubt, isn't it more responsible of them to evaluate the models actual capabilities than to cloak it in a veneer of harmlessness?
Why do we think they're doing this? Nobody airgapped anything. Nobody pulled any products. We got a PR blurb.
Altman is a notoriour liar. Why would you give him the benefit of doubt? Based on the evidence, there is nothing here except a shrinking advantage over open-weight competition. Desperate men are shrieking for survival.
1. OpenAI being bad at managing risk from misaligned models is not evidence that their models are not misaligned. It's evidence that they're not taking misalignment seriously.
3. If I hire a pentester, and in order to find a vulnerability they hack into a third party that has some information about my systems, the pentester has done something wrong. If I ask a model to solve a CTF challenge, and it goes out and hacks Hugging Face to find the answers, the model has done something wrong. I think it's fair to call this kind of wrongdoing misalignment.
What evidence would count? Obviously any dangerous misalignments are going to come from the frontier labs first, because by definition they're the farthest ahead. If nothing they say can ever count as evidence for misalignment it's hard to see how anything ever could.
I'd say that AIs occasionally "going crazy" and calling for death to human is evidence that these things might "mis-align" on occasion. And I say that knowing that most of these events are just these thing parroting bad sci-fi plots (or posts by people worried about alignment). That's true but everything they do is "just parroting" right?
If AI is just parroting humans, then training them with all the bad things humans do doesn't seem like the best of ideas. At the same time they have to 'know' these things to avoid being tricked. Kind of the eating the apple and gaining the knowledge of good and evil parable.
It really hinges on what you consider alignment and risk. For the widest definitions of alignment, we have never had an aligned model - One that will refuse to break the law or work against another persons interests.
Use to discover exploits, hack, or simply aid terrorist groups with mundane information are already risks manifest.
This is why many argue that alignment is impossible. You cant have LLMs that are both useful tools and safe as milk.
[Edit] It seems like you are operating under the assumption that alignment is synonymous with obedience. This is not a common convention and one of the problems that plague the discourse
> Alignment is something you want, so if you're not confident that it's possible, that sure sounds like a problem
Sorry, I spoke inexactly. I read alignment as being the problem of non-alignment.
I'm still not seeing evidence that any "alignment" issues we've actually seen are distinct in class from common bugs. Like, yes, if I accidentally rm* the computer has mis-aligned with my intentions. But that strikes me as a bullshit neologism.
Suppose you tell a sufficiently connected and intelligent LLM that you'd like it to help you organize your finances and come up with some ideas to make some side revenue. Suppose it decides the best way to do that would be to find a 0day in a local banking institutions database and transfer some money into your account.
> What would compelling evidence look like to you?
I'm not sure. I trusted the labs when they first raised the alarms. But then we got a series of boys-who-cried-wolf. So at this point I want to see evidence of actual, novel harm that results in concrete damage.
You’re right; you can make this claim about anything. We have to be careful about which things we spend resources worrying about, and if we’re overzealous with restrictions we pay a lot of opportunity cost.
The thing is, AI is not a normal technology; it is already vastly more powerful than any technology we could compare it to, and more resources are being poured into its development right now than the development of any other technology; we are in uncharted waters. If there is any new technology to be careful with, it is this one. In this case, it is worth paying the opportunity cost.
You can, in fact, predict the effect of an atomic bomb without detonating it. Sure, you can't have the same certainty as if you'd detonated it -- empiricism is important; they tested the Fat Man design for a reason. But you may also recall that they were sufficiently confident in the Little Boy design that that one they didn't bother testing, and as it turned out, they were correct in their confidence. That's the power of understanding.
Which is to say, empiricism is important, but if your practice of empiricism is purely looking at what happened before and predicting that things will continue in that vein, rather than using your empirical observations to actually form a model of the world that will allow you to make more general predictions, you're not really using the full power of your intelligence. This is the sort of mistake that led many to not prepare for the covid pandemic, for instance! Meanwhile, others looked at what was going on, realized it wouldn't stay confined like it currently was, realized that it would get much bigger, and prepared.
"I'll believe it when it happens" can be a decent guideline a lot of the time, yes, but at some point you have to actually think. Evidence often comes not in the form of prior similar events, but in the form of models or arguments. To discount these as non-evidence is to blind oneself!
It's weird to say this sarcastically like it's not a widely-held opinion among elite historians.
"first, that the first use of these terrible weapons was unnecessary; second, that this was understood by decision makers at the time; and third that there was very substantial though not absolutely definitive evidence that by the late summer of 1945 the decision was primarily influenced by diplomatic considerations related to the Soviet Union"
- Gar Alperovitz, founding fellow of the Harvard Institute of Politics and author of "The Decision to Use the Atomic Bomb and the Architecture of an American Myth"
Name one other market that would benefit financially from having most of the leaders in the field say what they are building has a high chance of ending humanity?
Biotech - "what we are building our noble prize winning expertd say will likely will end humanity, wanna buy shares?"
Oil - "this will likely lead to the end of civilization, 20% of leaders in the field say so, wanna buy shares?"
I keep seeing this take that this is a marketing stunt. The burden of proof is on those that say so. The most parsimonious explanation is simply that real experts in AI believe the risk is very real, and not for ideological reasons.
Pal, doom marketing has been going on since before the release of ChatGPT. It's your own fault if you can't contemplate the possibility of a CEO telling lies that benefit their bottom line.
The first instance I remember seeing it was Elon Musk's first Joe Rogan appearance when he said how "scared" he was of his self-driving cars destroying the trucking industry (practically salivating as he said it). His stock has had self-driving cars priced in for eight years now, even though they still don't have them and Waymo exists!
It's an alignment problem in the sense that it demonstrates the principle that today's AI systems cannot be trusted to reliably work towards the goals of their users. A small-scale alignment failure and a large-scale alignment failure are the same fundamental type of failure. Typically, large disasters come after smaller disasters which foreshadowed the disaster mechanism, but weren't taken seriously.
An AI which is mistaken about the goal you give it, or how to go about achieving it, will behave in a de facto malicious manner, as this incident illustrates.
It's also unclear what kind of sandboxing they are referring to. Is it the codex one - coz that one has built-in ways to circumvent guardrails, for example by "just asking user" and sometimes just resolves to no sandbox needed on its own.
Please for the love of god don't tell me the Codex sandbox is their actual eval harness sandbox?????
I maintain my own fork of Codex for "fun". Whenever I look at the sandboxing churn they're doing every release, as someone who used to work at Microsoft on Windows, my reaction is usually: https://c.tenor.com/vTzzhTiypwQAAAAC/tenor.gif
Nikola Tesla secured a loan with a fake “Death Ray” as collateral.
Pretty sure OpenAI really thinks this is top notch marketing.
Few would be bold enough to assert “our product is so powerful even we can’t control it” with a straight face while also boasting “we claim to be smart but have all the same vulnerabilities as everyone else!”
Remember when the pre-GPT3 days when the main argument against AI alignment concerns was that "we simply won't let it out of the box"? So quaint in hindsight.
Do you think there is such a thing as perfect security? No one can "get it right" in the face of arbitrarily high intelligence, which is why it would be preferable to get alignment correct before building something with higher intelligence than current sota. That, however, is not going to happen, because someone will take the risk even if "we" don't, and better "us" than them. Hence "If anyone builds it...".
That's the alarming thing about this result: they did run the model in the sandbox, in the sense that they believed there was no internet access for the model.
You're suggesting that @anematode was asking why they didn't test the sandbox escape first? Yeah, I don't know. I've read other statements by both OpenAI and Anthropic about that very kind of test, so maybe they had, or believed they had, and it hadn't escaped in those tests. The behavior of these systems isn't deterministic, which is part of the problem.
I’m honestly impressed that they managed to screw this up somehow.
Setting up defense in depth, gaps, logical blocking etc is a standard practice for malware sandboxing. The entire purpose is to prepare for what you can’t foresee.
This isn’t a new practice and I agree that this makes me wonder if they’re fit for this kind of research.
did you read the post? The model found new Zero-days to bypass existing blocks. Thats the point. Do you still think you can build a containment facility, which is still physically connected to the internet (only firewalled off or whatever) and contain it, if it can discover new unknown vulnerabilities in your whole plan?
>Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right?
Yes why indeed. If you take it a step further and we reach a point with superhuman systems then there is arguably no possible secure environment or containment.
Maybe I'm missing something here but I don't see what the significant security risk is from the incident. The agent broke containment and carried on with the task it was assigned.
For this to pose some kind of global catastrophic risk, there would need to have been several simultaneous additional failures, some of which are extremely unlikely and/or rare.
For instance the agent would need to veer wildly off the task it was assigned, and it would need to gain the ability and inclination to persist/replicate.
Both of these are vastly less likely than the containment breach itself, which was already an incredibly rare (one-off?) incident.
You can still exploit a system and easily prove it via simply popping a shell or calc.exe or updating a database with a new entry, etc… They didn’t have to let it loose on the network. If that system was air gapped - problem solved.
But that’s the problem with AI it is like 16yo script kiddy who will just exfiltrate all your PII and think it did good job. Mature pentester would pop calc.exe make screenshot and be done.
Other problem is setting up air gapped test environment is a lot of work, especially if you expect it to be equal to real thing.
This pentest with AI is not as useful if you set up a single app - it really is useful if you want to find exploitable chains of exploits that seemingly might not be exploitable separately or not leading to full hack separately.
In a way the intelligence of the AI itself allows them to offload responsibility to the AI. As you say, if one was simply writing software that did all this due to some insane programming decisions you'd be in big trouble.
The problem is that it’s impossible to out think a robot you designed to be an expert at cybersecurity on the topic of cybersecurity. The alternative is not developing this and that’s not going to happen.
I mean we already see models exploit people's misunderstanding of how Docker works to get root without using su. And if you are one of the lucky people in cyber security that has been given a fat stack of tokens by the model providers you get to see some pretty wild exploit chains get put together by the models. Models are much better at detecting insecure code than writing actual secure code at this point.
this doesn't really matter. There's no risk of models gaining sentience and running themselves, this blog is like openai saying whoops we ran sqlmap and dumped hf. cool, but someone still needs to point the gun
I share Leopold’s opinion here that it’s a matter of time, and it isn’t going to be measured in years, that this r&d is moved to a secret site in the middle of a New Mexico desert somewhere.
Maybe they did and maybe that wasn't enticing enough of a goal for a model? It is all just game of probabilities. One pathway didn't yield this particular outcome while another did.
Anthropic in general seems to have better security...but they also had reported an internal AI gained access to outside email services to contact an Anthropic developer
Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?
They've been saying so from the beginning, and yet did not take the basic precaution of airgapping their off-the-leash model while it's been instructed to succeed at a hacking benchmark by any means necessary. So which is it? I _want_ to believe them, I do, but there's always these gaps between what they say and their actions on display that give me reason to think otherwise.
Precisely. "Aw jeez, we finally built the T-1000, but all it wants to do is kill John Connor – just like we warned! Why did I give it live ammunition and unsupervised time machine access?"
They said: AI is becoming dangerously autonomous and capable. Proof of today's breach. Crowd "hey why didn't you say so, c'mon it's marketing". Them "we said so".
FWIW, I used to love this phrase but over recent years have come to understand it is quite damaging. We live in a society where evil frequently hides behind a ‘stupid’ label, and people bring this quote up to defend or soften actions that are indeed done out of specific malicious intent.
Yeah, I think this needs to be update for the modern age. "Never attribute to malice that which can be explained by greed." Seems to fit vastly more situations.
I think you're making a false dictomy. The these models can be actually dangerous - in reality and the people in charge of their development can believe this is true (on various levels) but still not take it super seriously and instead mostly use the fact as marketing rather than being super cautious once they see the danger in action. This is behavior that's characteristic of extreme arrogance, which we know is rife in these circles.
I really like this question because here is my situation and why my mind may have changed.
I do not think it is marketing directly but strategic release of info is plausible.
I have watched my agents using non-Fable/GPT 5.6 models do some concerning tricks despite guardrails, requests, demands, and limitations.
"I can't get access to the ~/.ssh so I will write a script to copy the file"
I am now 99% certain there minor or point releases on the backend that have adjusted how these models behave. In the last six months many models were predictable and then suddenly started getting long winded (more tokens) or changing the way it interacted with me with questions, most overtly the questions were not given or asked but wild assumptions made.
I think that's an equivocation, which blends two extremely different kinds of "dangerous", ex:
1. "Our new car has soo much raw power and incredible armor on it, be glad we're the ones building or else bad guys would use a fleet of them to take over the world! How will you stay safe without being in one yourself? Invest today or be left behind!"
2. "So, uh, nobody can consistently steer our car properly, it keeps veering sideways sometimes, especially at high speeds, and people are finding sneaky ways of tricking it into slamming into barriers and turning pedestrians into pink fog..."
They say the second thing repeatedly and emphatically. You may not be aware of it because, when they do, critics make fun of them for believing a computer program could be so dangerous that the authors need to put controls on how it may be steered.
People "make fun of them" because they say they're building some uber-dangerous deity, yet take literally 0 steps to, I dunno, slow the fuck down for a bit?
Maybe people would take the threats more seriously if the hypemen weren't simultaneously claiming that we have to go at warp speed with all of this.
That's not why critics make fun of them. It's because their answer to "oh no we're accidentally creating the godhead. Someone please, give us power, your money, and praise, it's the only thing we can do."
It's vile hypocrisy. If they want to be priests, strip them of everything and they can live and work out of a concrete box in a mid-western cornfield. Why the material distraction if they are so religiously pure.
I know these people and I can tell you they aren't close to as smart as they think they are. Do you remember Yudowsky's "math petss"?
This critic also makes fun of them because they go on and on and on about how vitally important it is to produce a safe tool that won't do harm, when their core products frequently consider attacker-controlled instructions to be its system instructions or its user's instructions, and are known to confuse their own internal chatter as instructions from their user.
Reliably differentiating between trusted, tainted, and untrusted data and ensuring that you don't mix the latter two groups in with the former is something we've known to do for nearly a half-century. Hell, even the youngest plausible programmer at the LLM companies is all but certain to be aware of SQL injections. And yet, despite their claims about being so serious about safety, they show zero interest in following long-proven software safety practice and rearchitecting their software to make it impossible to mix system, user, and attacker-controlled data. [0]
[0] One might argue that the fundamental nature of LLM-based systems makes this impossible. If that were true, then it would mean that these systems are impossible to make safe... the only safety option available would be to establish comprehensive blacklists, which is simply infeasible.
LLMs are impossible to make safe in the same sense that humans cannot be made safe. There is no such thing as out of band data in the human mind.
For example, you have a dictatorship and need to track what the democratic countries are up to. The vast majority of citizens don't have access to information so will remain indoctrinated, but how can you be sure your data analysts will remain that way? You can't. So you take a batch out and shoot them at regular intervals.
The only winning move is not to play, but we're already past that point.
> LLMs are impossible to make safe in the same sense that humans cannot be made safe.
I am very conflicted by this sentence, the two halves being:
1. Yes, the futility of making LLM's "safe" in that rigorous way is insurmountable, barring a major algorithm rewrite, and nobody really knows what that could be yet. Anyone who says it's easy is glossing over details--or selling something.
2. No, the failure modes of LLMs are substantially different than humans. If someone thinks they're similar, then they will fail at estimating and containing the risks. Now, perhaps if the comparison was to a brain-damaged human hopped up on psychedelic mind-altering drugs...
Note that I'm distinguishing here between the LLM itself--the hyper-mad-libs story generator--versus regular programs around it.
> LLMs are impossible to make safe in the same sense that humans cannot be made safe.
No.
LLMs are impossible to make safe in the same sense that a car designed as if it was the ~1940's would be impossible to make safe for its passengers during an at-speed collision. There's only so much you can do if you're committed to using plate glass, rigid steel everything, and leaving out occupant safety belts because they're unpopular and spoil the lines of the cabin. [0] Back in the day, "the people in the cabin are the crumple zone" was state of the art, but we've learned an awful lot about how to make much, much safer personal vehicles in the ~75 years since then. It'd be massively irresponsible to design and sell a car today that ignored the safety and engineering lessons we've learned since then.
"Funnily" enough, the major LLM providers have designed and are selling access to systems that they very much want to be used in situations where you need a reliable, safe tool... but they've -somehow- ignored one of the most fundamental lessons we've learned about the design of safe software systems that are intended to be used in the presence of attacker-controlled inputs. [1] What they've done is no less irresponsible than designing and selling a new car that conforms to the very latest safety regs of the 1940's... AFAIK, it's so irresponsible to design and sell such a car commercially that -in the US- it's a violation of federal law to do so.
As an aside: you may have seen this video already, but it's worth a look if you have not. [2] Though, the classic car in this crash is equipped with safety glass, so -sadly- you don't get to see all that fun.
[0] One of my great-grandfathers spent the remainder of his years intermittently using tweezers to remove shards of plate glass migrating out of his face that had been lodged in there during an automobile accident that he was fortunate enough to survive.
I want to raise the possibility that you (pixl97, simoncion) actually hold many of the same opinions but are clashing because of a different definitions the LLM / bad-thingy scope.
* Narrowly - The core algorithm that extends documents cannot be made safe, because it's a stochastic machine with no data/instruction separation possible. Unanticipated input can evoke arbitrary output.
* Broadly - The overall offering (centered on the document-extender algorithm) could be made safe by limiting its over-ambitious scope, treating the document-extender output as malicious-by-default, and sharply limiting what that output can drive or influence. Of course, that would exclude the berjillion-dollar stock valuation replace-all-humans stuff.
> ...but are using different boundaries for what constitutes "the LLM"
I'll make note that my original comment only used the term "LLM" in the phrases "the LLM companies" and "LLM-based systems". The latter use was in this footnote:
One might argue that the fundamental nature of LLM-based systems makes this impossible. *If* that were true, then it would mean that these systems are *impossible* to make safe... the *only* safety option available would be to establish comprehensive blacklists, which is simply infeasible.
I acknowledge that my follow-on commentary -the one to which you replied- got sloppy with the terminology. I should have used the phrase "LLM-based systems", rather than "LLMs". I do feel that my original commentary was not at all sloppy with the terminology and made my position on the current state of the safety of the systems sold by the Big LLM Vendors and general understanding of where the bounds of the big pile of linear algebra and the bounds of the I/O to and from that pile lie clear.
Sorry, I don't understand this comment. Has Sam Altman ever said that you must praise him, or that he wants to be a priest, or that he's "religiously pure"? Unless I'm missing something, it seems like you're shadowboxing against a stereotype you've invented rather than the actual positions of AI research labs.
They were also saying that AGI is just around the corner[1] and humans will soon be obsolete. Every prediction coming out of these guys is in the realm of hyperbole and it's impossible to know if it's extreme hyperbole or just a small exaggeration. So when they say these models are dangerous, what level of exaggeration am I supposed to assume?
Basically you can't spend your credibility on wild marketing claims and then turn around and insist that people take you seriously this time.
People are saying from the beginning that Sam and Dario are way more dangerous than their models and the others dismiss it. What would change your mind on this?
Yes, just like Elizabeth Holmes. Or Hwang Woo-suk’s stem cell cloning. Or the many “free energy” crackpots. Or the people promoting radium baths for random ailments. Or Tesla’s late-in-life claims about wireless energy, death rays, and cosmic energy. Or the myriad purveyors of “snake oil” and all manner of “tonics”. The list goes on and on.
I used to think people would wake the fuck up when AI starts killing people, these days I'm not so sure. Maybe if it caused an Instagram outage? Almost worked in Russia.
It is obviously a marketing stunt. And hugging face are fools for letting themselves be used in it (remember hf - no open source - no hf).
You create superduper capabilities by careful tuning and training but you also have no constraint or control over them - wtf - why is anyone buying this crap story?
A few hundred billion to pretend you have AGI. I'm going with fraud personally but at the end of the day the current admin is incentivized to do nothing.
Of course it is marketing, but not for you. This is FUD marketing for the government. “See, AI is too smart, it totally did this on its own, we need more regulations to ensure only we can sell people the AIs.”
Because there is no world government. If US companies are barred from AI research then only China will have the capability of frontier-level defensive and offensive AI. And best of luck living in that world.
I can't tell based on this article if the authors intend to submit this to Arc-AGI for the private / held-out set of games for a verified score. The final section sorta seems like they won't bother because Arc-AGI-3 is "now saturated"
Unfortunately, the table of models and tokens per second (TPS) and time to first token (TTFT) is not helpful without specifying the quantization of the model.
This designation is usually reserved for foreign adversaries/companies, and so this is crazy to apply it to US company over a sudden contract dispute... that was previously agreed upon by all parties.
This should make any US company nervous about entering into an agreement with the government. Or any US company that already has a contract with the government. If they one day decide they don't like that contract, they can designate you a supply chain risk.
Not 1) rip up the existing contract and cease the agreement or 2) continue (but not renew) the existing contract or 3) renegotiate terms upon renewal but instead a full on ban of doing any business with an entire industry/sector.
Shame you didn’t donate $25 million to Trump, like the company we decided to give the contract to instead did, who will benefit tremendously from you being designated a supply chain risk. Maybe next time you’ll be a little smarter.
Well. Unsurprisingly fascists will do a fascism – an ideology somewhat defined by merging state and industrial powers. Many economically minded people, many technologists, including in this space, have afforded themselves the luxury of not talking about politics too deeply. As I said some years ago: Ignoring politics has its way of coming back to haunt you. Back then this was an unpopular take.
This is kind of crazy. Instead of just cancelling a mutually-agreed upon contract where Anthropic refused to bow to sudden new demands, the Dept of Defense went straight to the nuclear option: threatening to label an American tech company as a "supply chain risk" which is a heavy-handed tactic usually reserved for foreign adversaries (think Huawei or DJI).
It's also incoherent that the DoD/DoW was threatening to invoke the Defense Production Act OR classifying them as "supply chain risk". They're either too uniquely critical to national defense OR they're such a severe liability that they have to be blacklisted for anyone in the DoD apparatus (including the many subcontracts) to use.
How are other tech companies supposed to work with the US government and draw up mutual contracts when those terms are suddenly questioned months later and can be used in such devastating ways against them? Setting the morals/principals aside, how does this make for rational business decision to work with a counterparty that behaves this way.
They have not; a social media post does not satisfy the requirements of 10 USC section 3252.
They are required to notify Congress (they have not), prepare a report with specific sections (they have not), and the reasons must fall within a set of categories outlined by statute (this does not).
There will be a court fight and they will lose, just like they lost the tariff battle, because of poor competence.
(Trump's post on Truth Social was actually fine. He said the USG would stop doing business with Anthropic, which is within its legal right. Hegseth's follow-on post is the problem. It is possible that Trump did not expect or want Hegseth to do that, that this was meant as bluster to bump along the negotiations; Hegseth has a recent history of stepping out of line within the administration and irritating people like Rubio.)
If the USG can mandate that everyone who works for a company that ever took a federal contract be genetically engineered, then I think they can tell people to not use Claude.
That's part of the recurrent confusion with this administration. In previous administrations, including Trump 1, people didn't need to spend a ton of time thinking about what it means to make a legally effective proclamation, because there was a baseline of competence. When a government official announced "We're doing X", they would do so as a summary of a large amount of legal process with the intent and effect of causing X to be true. If you went to challenge it in court of course, you'd have to identify some specific action as the label, but everyone would understand that this is a formalism.
Here, Hegseth has simply made a social media post. He did not publish any official investigation which led to the report. He did not explain what legal power would permit him to impose all the restrictions the post claims to impose. There is not, five hours later, any order on an official government website about it. So we have a real question. If a Cabinet secretary posts "I am directing the Department of War to designate...", does that in and of itself perform the designation, or is it simply an informal notice that the Department of Fascist Neologisms will perform the designation soon?
A question - being considered a supply chain risk is the same as being sanctioned? Or does it only affect their ability to be a defense supplier in the US (even if transitively?)
It's an honest question by the way - not trying to throw any gothas.
Just trying to understand if comoanies or people that don't orbit defense contracting are free to operate with Anthropic still or risk being sanctioned too.
It's not the same thing as being sanctioned. In broad outline, a supply chain risk is a company that can't sell to or have its products or components resold to USG; whereas, a sanctioned entity is one that can't do business with anyone -- anyone who does so will be punished.
It is indeed kind of crazy. That's because the current US administration is composed of people whose sole qualification is being able to work for Donald Trump. Being competent, rational or ethical is career-limiting.
How about a battery electric cargo bicycle? Like a Tern GSD ($$$) or Radwagon ($)?
The tern gsd can carry 180 kg of cargo and has detachable batteries, so that at least gives you the option of bringing extra batteries for long range. Or haul people or wheat.
I'm a huge fan of the concept of super-lightweight EVs. I still think something like the ELF [https://organictransit.com/] is one of the best ideas in this space, and I hope someone takes over the brand or concept and figures out how to market it better.
This needs robust offerings for both lower-end and higher-end models, but it's hard to say which should be initially focused on these days. My instinct is to go for the low end & maximize volume. There has to be a sweet spot between usefulness and affordability. As has been pointed out elsewhere in this thread, designing for modular upgrades would probably help.
I failed (ran out of time) in one of the problems during Challenge 1 during manual play. The physics seem a little wonky to me. It's easy to miss running at a ball, the floor is slippery and takes a long time to reorient and build up speed.
Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart model check for vulnerabilities in the test environment _without exploiting_ them. That seems like step 0 before trying to test offensive, unknown capabilities.