I don't have a guide on-hand but my advice, if you're new to Drupal, would be to take a few weeks and really explore the power of it (via contributed modules and some of your own development; if possible). The same thing scared me away initially (that was 8 years ago), but I've learned to love the power and complete flexibility that this platform offers.
Since we're talking about messaging, try out the Private message module. Right off the bat, you'll get a much cleaner form than the ugly node form. Private messages are also fieldable so you can add additional fields like attachments, labels, etc. Using the API (hook_form_alter()), you can do anything you want to any form at all. Then once you introduce a good theme, the possibilities are as endless as any other solution.
I'd also take a look at some of the Drupal 8 backported modules for Drupal 7 (if you're going to use 7). They introduced a number of great UI changes and made some of them available for Drupal 7 via modules.
Thanks - I used Drupal a lot from v4.4-4.7 and then was responsible for the security of a handful of 4.5-4.7 and v6 sites. Neither task endeared me to it, but I know Drupal 8 is going to be pretty different, so seems a good one to look at.
The flexibility's great, but the code hooks felt like a maintainability nightmare ("just what's affecting this now? It's 6 months since I last looked at this module..."). A tradeoff I guess. I was also burned by broken modules (submitted a couple of patches, but it did defeat the point of off-the-shelf features), but Wordpress et al are no better at that either!
Continuing with my earlier point, I don't have specific examples to give, but I've built so many complex web applications using Drupal with beautiful interfaces/UIs that you wouldn't even be able to recognize what was powering it. The complete allure of Drupal to me is that you can be literally anything (for the most part), and you can do it incredibly fast.
Since we're talking about messaging, try out the Private message module. Right off the bat, you'll get a much cleaner form than the ugly node form. Private messages are also fieldable so you can add additional fields like attachments, labels, etc. Using the API (hook_form_alter()), you can do anything you want to any form at all. Then once you introduce a good theme, the possibilities are as endless as any other solution.
I'd also take a look at some of the Drupal 8 backported modules for Drupal 7 (if you're going to use 7). They introduced a number of great UI changes and made some of them available for Drupal 7 via modules.