I had this set for my old AWS work account, but unlike a good WebAuthn implementation I'm pretty sure AWS only allowed me to a set a single key.

I tolerated that because a work account administrator can let me back in if I lose the key, but this is very much a second class implementation and I think AWS ought to do better.

It's true. You can only set one 2FA factor on an IAM account. As a work around, I ended up making myself two IAM accounts: one tied to primary Yubikey and another to my backup. Certainly not ideal.