Unless you're famous enough that everything you post gets to the top of Google, no one will read your content unless you post it to a social media platform with a curated audience. Literally no one but bots and AI crawlers. I mean, you have a whole list of crossposting links at the bottom of your own blog posts, including to HN and Mastodon, so you clearly understand. You wouldn't lose anything by limiting comments to just those curated social media platforms except spam and bots.
Looking through my recent comments, the majority are from people filling in the box on my posts. So, yes, I would lose a lot if I switched them off.
You'll also notice that I use WebMentions to import those 3rd party discussions (or links to them) into my blog.
That way users aren't subject to the advertising on platform A, the poor moderation on platform B, or the difficulty of even finding the comments on platform C.
I was actually going to suggest importing comments from a Fediverse account like Mastodon as a solution to that problem but I saw you were already doing something along those lines.
I'm trying Mastodon comments on my own blog, that I'm not posting here because it isn't really ready, from my own Mastodon account. I'd never even consider public comments anymore, it just seems like inviting trouble.