I'd be SSHd in and restart the rules, then the SSH session would hang. I was actively modifying rules and hey look I was a noobie sysadmin!
I made dumb mistakes back then. I believe that's when I made a catch all rule for my home IP on ssh in and out.
Regardless, thanks Tom!