So, the leader must renew their lease every (period * ι), where ι < 1. The problem is that this assumes a synchronous network, or reliable clocks - since the Paxos implementation isn't tied to the master -- this results in the problem that you've given up safety, example:
1) The process p1 on the machine holding the lock stalls (VM stall, let's say).
2) τ seconds have gone by on the Cassandra cluster, and the lock is removed
3) Another process, p2 takes the lease
4) The process comes back, only to resume processing, and not realize its lease has expired...
Without the leasing system, and the leaser being tied together, it's very difficult to guarantee safety, and liveness.
Agreed. They discuss this in the chubby paper as well.
There is a grace period where the master is in jeopardy of failing over vs failing over straight away. I left this out for simplicity of the state machine but I pointed to the chubby paper as a true reference.
They also mention the fact that client and master clocks must advance no faster than a known constant factor.
Like the post says, this seems to be similar to Google's Chubby system, and ZooKeeper for locks/leases. Are there modifications to this process for those systems to help with safety, or do they also have the same problem?
I love the simplicity of this, especially if you're already using Cassandra.
Is anybody out there using a lease mechanism like this (i.e., atop Cassandra) in production? Are there any pitfalls or edge cases in taking this approach?
There are numerous users of the paxos/lwt feature of Cassandra to selectively guarantee immediate/strong consistency for operations that need it (some at very large scale). But I'm not aware of anybody using this type of lease mechanism on top of Cassandra in production yet. I'm guessing this will be changing soon, as it has a lot of potential for infrastructure and application simplification. If you are interested, I'm happy to discuss further. Feel free to ping me on twitter, or email me at DataStax.
1) The process p1 on the machine holding the lock stalls (VM stall, let's say). 2) τ seconds have gone by on the Cassandra cluster, and the lock is removed 3) Another process, p2 takes the lease 4) The process comes back, only to resume processing, and not realize its lease has expired...
Without the leasing system, and the leaser being tied together, it's very difficult to guarantee safety, and liveness.