I completely agree with you — we should definitely decouple our conception of worth from what we “do”. And if the only issue with AI is that it gets good at doing what we want it to do, then I also agree with you.
But if AIs become superintelligent and plot an overthrow of humanity, it won't matter what our conception of identity or dignity is, the AI will still kill us. The AI doesn't ascribe dignity or worth to humans. It only pursues goals.
But if AIs become superintelligent and plot an overthrow of humanity, it won't matter what our conception of identity or dignity is, the AI will still kill us. The AI doesn't ascribe dignity or worth to humans. It only pursues goals.