I suspect at least some of the audiophiles, if not all, are highly sensitive persons [0]. I wonder if anyone bothered to study this possibility. Maybe some exaggerate in their search for perfection, but I'm inclined to think the search starts with their ability to actually perceive details that most people don't.
My wife is very sensitive to sounds, and its linked to her autism. Not all autists suffer from this (my father was an "audiophile", yet I don't suffer from high sound sensitivity).
[0]: https://en.wikipedia.org/wiki/Sensory_processing_sensitivity