I think there would need to be a little more to associate the action with the punishment. Since the point deduction would be pretty delayed from the voting, voters might not recognize which votes were poorly placed, and wouldn't learn to place better votes.