X doesn't need a hardware specific DDX driver either when running on hardware supported by Wayland+Xwayland. The GLAMOR backend works just as well for a standalone X server, and it can just use the same libraries and APIs as Wayland does for the rest of the stuff. (There's not really any kind of hardware abstraction layer in Wayland itself.) The main reason why people use the hardware specific X drivers still is because they're more mature and robust than the Wayland stuff.