This is the case of user changing password setting and and realizing you can't use them with old backups after accidentally destroying one dataset. zfs is intented for servers and sysdmins so it is not as friendly as some may expect, but it did not lose anything that user did not destroy. Author had to use logic to deduct what he did and walk it back.
> changing password setting and and realizing you can't use them with old backups
That's unfair to the author. The backups were new, post-password change. And neither old nor new password worked on them. The thing that was old was an otherwise empty container dataset.
This is a case where both sides are completely understandable and no one did anything wrong. ZFS didn't lose its mind. It worked as designed and intended. The author didn't know a critical detail about the implementation. It's a series of unfortunate events. The only failure could be lack of better ZFS documentation.
What ZFS did is understandable but wrong. Sending an incremental snapshot needs to send updates to the encryption parameters, even if they're inherited from another dataset.
I'm not sure if anybody is wrong or right. But this should be officially documented, a specific error provided- not "permission denied", and a workflow to fix it that doesn't involve patching the driver.
Hi, author here! I'm not sure why you're getting downvoted because you're absolutely right.
Depending on if you're being optimistic or pessimistic, either 1) neither I nor ZFS did anything wrong or 2) both ZFS and I did some things wrong. Either way, neither one nor the other is particularly at fault.
While I said "ZFS lost its mind" for title length reasons, it really would be more accurate to say "ZFS appeared to loose its mind", since as I learned, everything makes sense when you consider the encryption root.
> This is great writeup, and I really appreciate you taking the time on it. With my OpenZFS dev hat on, it's often quite difficult to understand exactly how people are using the things we make, especially when they go wrong - what were they expecting, what conceptual errors were involved, and so on. I'm passing it around at the moment and will give it a much slower and more thoughtful read as soon as I can. Thanks!
> While it's fresh on your mind, what would be one simple change that we could make today that would have prevented this is or made it much less likely? Doc change, warning output, etc. I have some ideas, but I don't want to lead the witness :)
First off, thank you for taking the effort to try and understand this from the OpenZFS side. It's really easy to dismiss this kind of thing as user error (which is true) since OpenZFS did actually behave as designed (which is also true), rather than taking it as an opportunity to better understand and improve the user experience.
When I think of the factors that lead me to make the mistake of not sending a snapshot of the encryption root, I think it comes down to a difference in expectations vs reality. When I think of a snapshot, I conceptualize it as fully consistent version of a dataset at a point in time (which as far as I know is still true for unencrypted datasets). Native encryption violates that expectation by 1) storing the wrapping key parameters in the encryption root which may be a different dataset and therefore exists outside of the snapshot and 2) allowing the wrapping key and dataset master keys to get out of sync.
If I send a snapshot from one pool to another, I expect ZFS to send everything necessary to reproduce the same state on the destination pool. As an uneducated user, I'd find it very unintuitive that I also need to send a new snapshot of another empty, unchanged dataset which through some "spooky action at a distance" affects the decryptability other child encrypted datasets just because it's the parent dataset at the root of the encrypted tree. I expect datasets to be isolated from one another. Users shouldn't have to know about wrapping keys and master keys, let alone worry about keeping them in sync across multiple datasets.
While I do think the docs could be improved to emphasize the importance of the encryption root (especially in zfs-send -w, --raw which doesn't even mention it) which would've made debugging the issue a bit easier, I'm not sure how much that would've helped prevent the issue in the first place. The reality we must face is that people don't read the docs unprompted to challenge their fundamental expectations; they work with the mental model they have and only consult the docs when they have a question to answer.
What I do think would've really helped is if zfs-recv could check the wrapping key parameters on the encryption root when it sees a new encrypted master key in the send stream and abort if they don't match. This wouldn't prevent every scenario, (eg. if instead of forgetting to send a snapshot of the encryption root, you forgot to send snapshots of the child encrypted datasets), but it would have prevented this one and would be a step in the right direction.
In the long term, I'd really love for OpenZFS to treat keeping wrapping keys and master keys in sync as an invariant that is always maintained so users don't need to know about encryption roots, wrapping keys, or master keys and can ignore them like any other implementation details. I've only just begun thinking about some potential options in the solution space, but I have a feeling this will not be an easy problem to fully solve. I'd love to hear your ideas as an actual OpenZFS developer.
The user has their encrypted data and two encryption keys. They should be able to decrypt. They don’t care about internal ZFS password settings.
I also confirm that people snapshot their data, which is usually child datasets. If you don’t care about an empty folder, snapshotting and replicating it according to a careful schedule is not expected.
This is the case of user changing password setting and and realizing you can't use them with old backups after accidentally destroying one dataset. zfs is intented for servers and sysdmins so it is not as friendly as some may expect, but it did not lose anything that user did not destroy. Author had to use logic to deduct what he did and walk it back.