I thought the point of secure enclaves is to protect against attacks by someone with access to the hardware.

Therefore requiring physical assess is still low complexity in context.