Tbh, the answer is simple: if we truly get AGI, the government would nationalize it because it's a matter of national security and prosperity for that matter. Everything will change forever. Agriculture, Transportation, Health... Breakthrough after breakthrough after breakthrough. The country would hold the actual key to solve almost any problem.
when you write it out like that, it sounds unfathomably… silly.
I'm not a tinfoil hat skeptic, and i'd like to think i can accept the rationale behind the possibility. But I don't think we're remotely close as people seem to think.
> Tbh, the answer is simple: if we truly get AGI, the government would nationalize it
If you truly get AGI, you are highly unlikely to be able to reliably control it let alone nationalise it. And it is highly unlikely that only a single country would reach it. Chances at least one other country would. And AGI would be eventually weaponised against other countries successfully or not.
the changes of AGI causing huge damage in the world would be very real. Unlike a WMD, the damage isn't necessarily visible, immediate or obvious.
As technology changes over history, governments tend to emerge that reflect the part of the population that can maintain a monopoly of violence.
In the Classical Period, it was the citizen soldiers of Rome and Greece, at least in the west. These produced the ancient republics and proto-democracies.
Later replaced by professional standing armies under people like Alexander and the Ceasars. This allowed kings and emperors.
In the Early to Mid Medieaval time, they were replaced by knights, elites who allowed a few men to defeat commoners many times their number. This caused feudalism.
Near the end of the period, pikes and crossbows and improved logistic systems shifted power back to central governments, primarily kings/emperors.
Then, with rifles, this swung the pendulum all the way back to citizen soldiers between the 18th and early 20th century, which brought back democracies and republics.
Now the pendulum is going in the opposite direction. Technology and capital distribution has already effectively moved a lot of power back to an oligarchic elite.
And if full AGI combined with robots more physically capable than humans, it can swing all the way. In principle a single monarch could gain monopoly of violence over an entire country.
Do not take for granted that our current undertanding of what the government is, is going to stay the same.
Some kind of merger between capital and power seems likely, where democratic elections quickly become completely obsolete.
Once the police and military have been mostly automated, I don't think our current system is going to last very long.
In the AGI Succeeds scenario, the situation is unprecedented and it's not clear how it ever gets better.