The main problem with running a mail server is properly keeping it up, secure, spam free, and trusted by peers to actually send email. These are not exclusively _software engineering_ problems with the existing (SMTP/RFC822-based) infrastructure. Rather, there's a large management component here, that is unlikely to be solved by software alone (open or closed source).
This will only change with a radical infrastructure change, along the lines of djb's Internet Mail 2000 proposal. Until then, if you want your mail server to remain reliable without working hard yourself, you'll have to delegate it (to google or hundreds of other available services).
> They are letting Google occupy whole the internet by not creating an alternative for Gmail.
Now that statement makes absolutely no sense.
They have ~20 million users after 10-18 years (depending if you count from the original Thunderbird or the original Netscape 2.0 which added a mail client). Gmail has > 500 million accounts, let's say ~400 million users.
With 5% of the users, they're not making a difference either way.
Furthermore, it's still an apples-to-oranges comparison: I suspect at least 50% (if not 90%) of those mozilla users have at least one Gmail/GoogleApps mail account set up in their TB.
The main problem that gmail solves for most people, and which TB can't offer (being a software product rather than a service), is a reliable mail service. (I wrote that already, but you seem to have not noticed that).
> They are letting Google occupy whole the internet by not creating an alternative for Gmail.
Services available for you that are not google: fastmail.fm, hotmail.com/live.com, mail.com.
Mail clients that are available for you that are not gmail: Opera Mail, Claws, Postbox, Evolution, Outlook, Mail.app, and a whole lot more (though only Opera is really cross-platform).
Seriously, there are a lot of open source mail servers/client, some of which might be good enough for you, e.g. http://cworth.org/sup/a-mail-client-for-geeks/
The main problem with running a mail server is properly keeping it up, secure, spam free, and trusted by peers to actually send email. These are not exclusively _software engineering_ problems with the existing (SMTP/RFC822-based) infrastructure. Rather, there's a large management component here, that is unlikely to be solved by software alone (open or closed source).
This will only change with a radical infrastructure change, along the lines of djb's Internet Mail 2000 proposal. Until then, if you want your mail server to remain reliable without working hard yourself, you'll have to delegate it (to google or hundreds of other available services).