The MetaForum messageboard software has been doing this for years. Everyone gets to vote, but people's voting power is determined in turn by how valuable to the community they are. It removes the need for moderators entirely, and completely eliminates the ability for spammers to thrive.

Their formula bases a large amount of worth simply on how their posts have been modded in the past, but then also include things like post count and a decaying value on length of time.