You can already switch/upgrade graphics drivers without rebooting, thanks to dynamically loaded modules. Just stop X, rmmod the old driver, modprobe the new one, and start X again.