Indeed, and you can go even further: with e2ee there is no need for central server beyond dumb distribution of opaque blobs. So the history can be exchanged by the whole network and the encryption keys shared recipient by recipient on a need-to-know basis. That's what bitmessage is doing, for example

(At least Threema does it this way): Server doesn't really store the messages long term; only until the receiver picks it up. With just two devices - sender and receiver - you have a guarantee that once the single receiver picks it up, there's none else to do the same and can be safely dropped.