I think it is actually pretty obvious what /happens/ when it runs out of RAM.
You have to either buy more RAM, start using the VM option, or start using OS virtual memory. The latter two will slow it down a bit.
Another possibility is to figure out which sets of data are taking up the most space, and offload them to a separate disk-backed DB, and just cache the most frequently accessed subset of that data through Redis (this could be easily done through key expiration). I find this option kind of painful, mostly because I really like the way that Redis approaches data, and using another database is comparatively inconvenient.
Having worked with it a bit, I think Redis is a completely awesome piece of technology, and in practice it is perfectly good to use for something like this, at least in the short term. I don't want to bring down Antirez's hard work in any way.
Being the type of developer that hems and haws over corner cases and unlikely what-ifs, however, I felt the need to ask, as it has been in the back of my mind for a while.
I am encouraged that there will be a version that manipulates the data-structures on disk, as that was the best solution that I could think of as well. (Basically you could then run two instances of Redis, a RAM based one and a disk based one, echoing commands to both, but allowing data on the disk backed Redis to expire and taking responses from whichever returns an answer first).
My only hope is that clustering won't take to long and that it won't be abandoned in favor of a different castle in the sky if it doesn't turn out to be perfect.