The point is not about "stifling EV adoption", it's reducing problematic levels of over-dependence on cars in general. EVs don't solve all the issues that cars raise.
Noise is a relevant factor in that discussion, not compared to internal combustion engines[1], but compared to fewer cars in general.