The server-side swift ecosystem is incredibly small, and that's by far the most important thing for adoption.

You're not really going to share backend source with your iOS apps, so there's nothing to really gain there. Languages are so homogenized these days that moving between the popular few is pretty simple and requires fairly minimal rampup.