Haven't used XMPP much so can't comment on that but in case of Matrix, I use conduit (server) + cinny (client) and it's much lighter and better setup than the synapse and element. Conduit isn't super feature rich nor cinny but if you aim for rich text messages it works well.
Cinny's latest update seemed really good and juicy, with lots of important features finally added. Haven't tried it proper yet tho.
My issue with anything not Synapse (for now), is migration and atability. Conduit is still somewhat a young project in comparison, and I'm not entirely confident in the upgrade paths yet. They also use RocksDB (or sled), which furthers this "fear", perhaps irrationally so.
I'd love to host my conduit server with confidence though, but I'd also need Matrix to properly support account migration. IMO it's one of the key features still missing and it's damaging its "decentralized" image.
