> The second generation AI would happen as soon as some subset of the AI travels too far for real time communication at the speed of light.
Not necessarily. It's very easy to add error correction codes to make a computer not change if you really don't want it to even in the presence of radiation-induced bit-flips.
(There's also the possibility of an ASI finding a solution to the alignment problem before making agents of its own; I would leave that to SciFi myself, just as I would proofs or disproofs of the Collatz conjecture).
Also: what does "real time" even mean in the context of a transistor-based mind? Transistors outpace biological synapses by the same ratio that wolves outpace continental drift, and the moon is 1.3 light-seconds from the Earth.
Not necessarily. It's very easy to add error correction codes to make a computer not change if you really don't want it to even in the presence of radiation-induced bit-flips.
(There's also the possibility of an ASI finding a solution to the alignment problem before making agents of its own; I would leave that to SciFi myself, just as I would proofs or disproofs of the Collatz conjecture).
Also: what does "real time" even mean in the context of a transistor-based mind? Transistors outpace biological synapses by the same ratio that wolves outpace continental drift, and the moon is 1.3 light-seconds from the Earth.