I think he actually argued that men might have a different distribution for mathematical ability than women. Specifically, he suggested men might have a higher variance in their distribution, meaning more men in the tails at both sides. But the distributions would still have the same mean. So men are not 'better than math' than women -- there would be more dumb ones and more smart ones, and less in the middle, effectively.
This phenomenon, by the way, is commonly observed in other species for various genetic traits. From an evolutionary perspective, it makes sense for the males to "roll the dice" and have a higher variance for many traits. For example, one superstar male frog can have many more offspring than one average male frog. But one superstar female can really only have as many offspring as the average female. That's a crude over-simplified explanation, but the idea is sound.
he suggested men might have a higher variance in their distribution, meaning more men in the tails at both sides
That's my understanding as well.
A professor I respect a lot presented his analysis of Summers' argument in a lecture I went to. This professor said that Summers' argument was statistically invalid, since the number of outliers in the tails of the IQ distribution was insufficient to tell if the distribution was in fact different at the tails between men and women.
It's fair to say that this rebuttal was not the one Summers got when he presented his idea to the faculty. :)
This phenomenon, by the way, is commonly observed in other species for various genetic traits. From an evolutionary perspective, it makes sense for the males to "roll the dice" and have a higher variance for many traits. For example, one superstar male frog can have many more offspring than one average male frog. But one superstar female can really only have as many offspring as the average female. That's a crude over-simplified explanation, but the idea is sound.