It doesn't have to be a conscious AI god with malicious intent towards humanity to cause actual harm in the real world. That's the thought that concerns me, much more so than the idea that we accidentally end up with AM or SHODAN on our hands.

This bing stuff is a microcosm of the perverse incentives and possible negative externalities associated with these models, and we're only just reaching the point where they're looking somewhat capable.

It's not AI alignment that scares me, but human alignment.