I've done this and I'd recommend getting rDNS set up if you can, on top of your SPF and DKIM, though it sounds like maybe it's not entirely necessary.

I ran a linksys NSLU2 as my mail server for a few years, with a USB stick as its storage. With Debian linux with Dovecot, Postfix, spamassassin, and the Spamhaus DNSBLs set up I managed to keep the signal to noise ratio pretty damn high too.

It was fun, and remarkably not hard.