I suspect that people who actually try Vim are perfectly likely to learn and like it. People who are scared away by comments like yours, on the other hand...
Vi is incredibly easy to learn, but it takes some learning before you can accomplish anything. You can sit down with Bill Joy's original paper, read it, try stuff out, and you will be using Vi at 50% of the competence of a master with years of practice.
Emacs on the other hand seems to be simpler to get started in, but it never gets simple like Vi does. After you get over the first hump, you just see the Himalayas rising behind it.
I use Emacs right now, but I'm interested in moving to something that plays nicer with Mac OS and the god-awful lack of respect for Unix standards in the Rails community (like failure to exclude ~ files from the asset pipeline).
If backup files are giving you an issue, you can change the behavior of emacs so that it saves backup files in a separate directory, instead of the alongside the file that is being edited.