3 simple sshd config lines remove a gigantic amount of worries if you run open ssh servers:

   Port NN
   PasswordAuthentication no
   AllowUsers user1 user2 user3

change those, sleep at night.