You could say the same about genes in tons of other areas, and you'd be completely wrong. That's now how evolution works. "Beneficial" genes don't just get to spread everywhere quickly, otherwise parapatric speciation could never occur. Read the wikipedia article for Parapatric Speciation. What does "incomplete geographical barriers" remind you of?

Evolutionarily our intelligence IS very recent.