"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear. No other human activity poses this level of danger." - https://x.com/hilbertspaess/status/2097476196791709843#m
"Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman
This forum is fundamentally unserious and complete kneejerk cynicism these days. At one point it was prescient. Now these types of things were discussed years ago in other channels and when we reach a point where finally everyone comes around to the right conclusion based on tangible evidence in the news, a majority of people here go "huh i guess so"
This is exactly how Cloud Computing looked in 2012-2018. Dumping huge $ into computing buildout that wasn't profitable yet. All those co's: Amazon, GCP, Azure paid off immensely and are ridiculously profitable.
No, it wasn't. Cloud computing was almost immediately profitable.
And Amazon was famously "unprofitable" for their first 9 years because they were investing all their very real profits into a form of capital that the US tax code didn't recognize.
You are being downvoted but only a few years ago Google contemplated whether to kill GCP altogether because of lack of traction in the market or revenue goals.
IIRC Amazon chose to reinvest early revenue to grow AWS intentionally, and the revenue curve eventually evened out and obviously surpassed expenses. The problem with the AI buildout is that there's not a ton of evidence that these companies are approaching profitability, we can't even know because they're private.
Amazon's incubation of AWS was methodical and transparent. OpenAI is saying that they don't expect to be profitable until at least 2030, with over a trillion dollars in committed spend before that point. It's the largest "trust me bro" play in human history.
> I can only assume the reasonable people at openai and anthropic were all pushed out long ago
Typical uninformed take on the side of "doomers are crazy".
Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hired chasing $1m compensation packages.
If you're a doomer, wouldn't the "let's try to get there so fast" actions of the companies suggest that, in fact, there are no reasonable people in positions of influence there?
By your own description neither Altman or Amodei are reasonable if their thought process goes: "this is an existential risk, give me hundreds of millions of dollars so I can accelerate it."
GP defined reasonable people as not believing in existential risk from AGI. I think the leadership and alignment teams are way more informed and reasonable than some of the very poorly thought through takes like GP just making things up about the labs. There’s just no evidence of it whereas lab leadership are operating under a mostly informed worldview.
Yes, I don’t think lab leadership are totally reasonable as they were concerned about the risks and yet lack of reason caused them to directly contribute to the problem we’re facing. They’re still not being reasonable when they throw their hands up at the arms race and hope the utopia come, when that’s not our trajectory at all and more lab employees are realizing it
If they truly, truly believed that, would they be speeding towards building it? If yes, that would make them truly insane, right? Not as in a quaint "off their rocker" but more "non compos mentis".
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps. They have publicly stated as such repeatedly. They think the ~10-50% chance of extinction sucks, but that it's going to happen anyway and they believe they can steer it towards something good the best and unlock all the potential positives like infinite life.
I'm not saying they are crazy, I'm saying their predictions have a record of not being accurate, and thus give them no weight compared to others'.
In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.
Before OpenAI was ever even founded, way before any regulatory capture plausible claims:
"Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." -Sam Altman
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps.
How is it not addressed? The company never contained only "reasonable people" that believe AGI is not an existential risk to humanity. At both inceptions were people who believed in AGI x-risk, even the founders. Only after time, did there become more "reasonable people" who were mercenaries and only believed it to be a typical tech job. Today, there are more "reasonable people" than ever there. They haven't been pushed out. We're just witnessing some prescient mercenaries smart enough to Eureka the grave implications of what is actually happening.
> extremely intelligent people make confident predictions about AI capabilities and progress, then be so completely wrong.
The ability for the human mind to rationalize conclusions to maintain denial in the face of a very scary future is immense. Genuinely grappling with the implication of where we're headed is usually very crushing. It's not easy to engage with the possibility, and very intelligent people will use those smarts to feel safe.
> Genuinely grappling with the implication of where we're headed is usually very crushing.
As someone currently prepping for various AI doom scenarios and who has been dealing with AI-related nightmares for years this is very relateable.
Although, I don't personally think it's this. In my experience the opposite is more true – the majority of high probability doomers seem rather laid back about considering what they believe will happen to the people they love in a few years. Equally I don't get the sense those who don't have such extreme predictions are worried at all. If anything there's not enough emotion.
In my opinion people just don't reason well when it comes to exponentials and are ignorant about things they don't have good mental models of. At least I know I struggle with this.
Yes a couple of elite mathematicians working on the problem for a year, which AGI solved in a fraction of the time. What about everyone else 100IQ? What about as the models are even better 1 year from now, 2 years? The trajectory hasn't abated.
I don't know one way or another but there is a credible allegation that the "AGI" was training on the (very extensive) test set that these two mathematicians produced.
If that is true then this seems to be, again, a case of AI producing an interpolation over data it has seen before. Everything about openAI's behavior indicates that they were using the transcripts as input. Why not have the AGI choose a different Millenium prize problem?
If ~~someone gives you a hint about an approach~~ you steal someone’s notes about a promising approach, and then you hire 10,000 people to brute force the problem basically everyone would consider that “shitty behaviour”, “theft”, and “poor form”.
Well if you do the math, the number of agent-compute time in total, given the insane number of agents thrown at the problem, might end up being comparable in time, if not for the budget.
The goalposts for the singularity have always been that AI improves itself fully autonomously. AFAIK OpenAI is heavily using AI but still employs human researchers and developers.
Uh I'm pretty sure the "singularity" always presupposed a lot of previously unthinkable technologies becoming part of daily life, and was not ever limited to just computer stuff or math problems.
"Moving the goalposts" as shallow dismissal doesn't work if e.g. someone points out AI hasn't even built a new type of spaceship yet in response to a claim that AI is on the verge of building a Dyson sphere.
Zitron implied 2 years ago that OpenAI would collapse by now. How's that bubble popping going? All NVDA+memory co+frontier lab numbers are accelerating
The thing about bubbles is that everyone who thinks they’re going to pop look crazy until they pop.
And to be clear, a bubble popping doesn’t mean that AI goes away forever.
What it does mean is that we’ll see some kind of economic crash or recession, and we’ll probably see at least one big company fail or go bankrupt/restructure.
OpenAI is the company in most obvious peril.
I happen to think that Nvidia is in a more perilous position than they appear. Their hardware advancement pace is relatively weak and they’re in a crypto-like hardware bubble where they’re one technology breakthrough away from a complete collapse in demand for their AI data center solutions. They’re also doing a lot of sketchy hardware financing schemes.
The thing with bubbles is that this is a near permanent state. I heard as far as 2017 that there is a bubble and, well if the bubble popped now the economy would tank to a level a lot higher than it was 2017.
The more interesting thing is the needle and if the bubble will run into one.
Comparing to other bubbles in the past, this one could still have something like 2 years left. Patrick Boyle has a video on the topic, in which he's also very careful to point out he could easily be wrong.
By all means, keep betting on this being a wild success and we'll see who's right in a little while. If you don't think this is a bubble then you're in for a ride.
Look if we are going to raid pensions and 401Ks to prop up the valuation targets of Anthropic and OpenAI longer, then the bubble can stretch further and further, but eventually datacenters have to get built and powered. It's the power generation part that no one talks about. We simply don't have enough power in the United States to scale at the rate that Anthropic and OpenAI need to prop up their absurd valuations. AI is real, but these valuations are not.
TL;DR: Ed is directionally correct, but it's anyone's guess as to the exact timing.
In the meantime I'm not going to complain about subsidized credits from the big labs. :-)
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
A book if you will: https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
reply