Humans have physically adapted though. Skin and eye color, subcutaneous fat, underwater vision, high-altitude adaptations, resting body temperature, nose shape and nostril size, etc. We just haven't had populations separated long enough to be unable to breed or be considered separate species.