You can't legally selectively default on debt; see the ongoing Argentinian "pari passu" saga as documented by FT blogs. Doing it illegally is basically hanging out a "we're now a rogue state lol" banner in New York to welcome businesses.

Having said that, I'm in favour of politely but firmly squashing the tax haven states. There is some work in this area but it's very slow.