I've found the opposite, IME; perhaps the issue is with your boards? We regularly have this happen at work and the permanent marker is always removed. It also works on plastic containers - I've successfully done this hundreds of times on both reusable plastic bins for my food at work, as well as items I buy (and resell) from Goodwills and Savers / Value Villages. I've not yet run across a plastic container that this hasn't worked on.

I've obviously not tried this with "hundreds" of whiteboards, but the one we have on the fridge at home works as well as the several size and types we have at work.