Yeah, but the "normal" BSDs just ship an X server out of the box; no need to port anything. (It remains curious to me that Apple bothered writing Quartz from the ground up, actually; seems like it would've been easier to build on top of X.)
Building on X gets you very little other than compatibility with the X ecosystem - something that had little to no value to Apple - they certainly didn't want to promote OSX as some kind of Xserver in place of writing quality Mac apps.
Note that with current X, HiDPI and multi-monitor and tear-free experience still have problems and they are handled via a morass of legacy extensions of varying design quality (this is aside from the drivers issue)-- when OS X was started, none of these existed except maybe DBE and a nascent RANDR (there was no COMPOSITE, no RENDER, RANDR vs very primitive), I think there was the Xinerama crap that no one uses anymore because it is terrible.
So they would have had to develop that, for what benefit? What does X get you out of the box? A drawing model that was outdated even by 1995 standards. And network transparency? Everytime these discussions come up, somebody brings this up, and I wonder what drugs they are on. The love by some for the X network model has always baffled me, because it is terrible - it has virtually no practical usability for modern drawing models and compositing, and the worst is it isn't robust - lose your TCP connection, goodbye session! X hits #1 and #2 of the "Fallacies of distributed computing" pretty hard.
RDP, VNC, SPICE are what we use today, for very good reasons.
By writing Quartz from the ground up, Apple accomplished "every frame is perfect, by design" in the year 2000, which we in the free world are only getting now by switching to Wayland.
