Would you not eject someone from you property if they were harassing your other guests?
It's not an exact analogy, and there are features like mute and block that are less severe than removing someone from a platform, but people generally don't want to individually deal with every person who decides that being miserable to others is the best use for social media.
Moderation, either top-down curation or bottom-up mute/block, is not a solution. Either way, you're going to haphazardly create echo chambers, which aren't great at maintaining free speech.
It's not an exact analogy, and there are features like mute and block that are less severe than removing someone from a platform, but people generally don't want to individually deal with every person who decides that being miserable to others is the best use for social media.