In general, the way to avoid this problem is for everyone to be vigilant and work together to keep the process fair. Blow the whistle if you see anything shady going on. To make a big change to the results, you need a big conspiracy, and the bigger the conspiracy the easier it is to spot.

When the process is simple and low-tech, like pen and paper ballots, it's possible for all ordinary people to contribute to keeping the process fair.

With electronic voting, you just have to trust the manufacturers and the voting officials. Everybody else is out of the loop.