I don't use a towel to dry them, as any grit at all will fog the lenses. Instead, I just bang the edge of the glasses against my abdomen to knock the drops off.

They'll eventually fog, but it takes a lot longer.