Actually no, you're confusing application serving (a term used in non X environments) with display serving (which X uses).
X display servers (which we both agree are local) do not initiate clients to display on them, end users do this work - you can see this quite easily by initiating a client when you don't have an X display - the app will look for a $DISPLAY variable and attempt to connect to that display. The app is launched first by the user, and connects to the display server.
