apt doesn't remove the settings in your home directory. So you need to nuke them and reconfigure the entire desktop and switching DEs definitely break stuff due to file type handling and default apps. With Xorg there were other things like styles that got permanently broken unless you hunt for every file that has been changed.