they're not talking about the same TPU

The comment is a fun poke. Please don't take it literally! NVidia's new Volta GPU has Tensor cores that I think are very similar to what was in first generation TPU. I read the TPU papers and blogs published recently and I am seriously impressed by TPUs!

The Volta is 16-bit floating point, so closer to second-gen TPU. The first-gen TPU was integer only and used for prediction, not training.

Maybe they are designed and built by AI itself? /s

You're joking, but I wonder how far off this is?