I think you are right to focus on company culture instead of hiring/becoming better nudgers.

The teams that I've been in that were self-managing had a few things in common. The first one was that the incentive-structure was performance-based (even better, you should link it to the company goals) Second was that there was a social cohesion, 'the team' would get annoyed when somebody wasn't doing what he should have been doing, it's effective because people are social animals and nobody wants to be excluded (if they can help it). and third would be that they had the autonomy to make design decisions as long as these didn't impact stuff the others are working on.

I think that building a team that you can trust to get things done is well worth the hassle of firing/rehiring and maybe firing some more.