You do, this is Ubuntu Mobile strategy at the moment AFAIK.
However, drivers in ARM platforms are very opaque. You rely upon the goodwill of the manufacturer to update them, and most of the time they don't. See how most devices get unsupported in Cyanogenmod after a while.
And in any case, all the privacy concerns remain. It's a hardware issue. Android would be fine.