The passport card is slightly larger than a credit card and thus doesn't fit properly in most wallets. Otherwise I wouldn't mind having it on my all the time.
I have never seen a wallet that won't take a passport card. Also, my passport card is precisely the same dimensions as every other card in my wallet. Maybe you got a weird one?
I went back to check because I was second guessing myself now and it turns out it fits just fine. I think I was being silly because it comes in this little piece of paper that says it is a "protective sleeve" so I was trying to keep it inside the paper.