(Author here) Yes, it was an XFS/controller issue, but Ceph reported the failure. :) (IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database, but nowadays ceph-disk (which uses those XFS partitions) is gone, and instead ceph-volume uses a different approach via LVM.)
Regarding configuration management/firmware version: yes - especially, as you'd need to also rebuild disks in the dev/test environment with the identical configuration (firmware, disks,...), to ensure it's actually identical. And even if we neglect load/capacity/usage issues (problems might show up only under specific work loads), there are also further "invisible" layers/components like cables, NICs, switches,… and their firmware versions which are also relevant. Not exactly trivial. :)
> IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database
It has been shown, that you are right. But not because of bugs like the one you encountered. The problem could as well have happened with a regular xfs fs holding a maildir.
(Author here)
Because it was absolutely unclear at that time yet, that an older kernel version could mount the XFS partition but the newer kernel versions could not (this only came up during the post mortem/RCA later). Furthermore the clock skew and mon_host issues gave us a wrong picture of the situation. (Also as the hosts are running as hypervisor systems, the kernel version should ~match with the environment (Proxmox/KVM), so there might be other/unclear risks with running such a setup.)
The project is quite old already and could need some love. If someone wants to contribute, it's available on GitHub: https://github.com/grml/zsh-lovers
A more recent project similar to zsh-lovers is http://grml.org/zsh-pony/ - this are the notes from a talk I gave at DebConf 2011 and also covers some nice features of Zsh. Should be easier to go through that than through zsh-lovers if you're interested in an introduction of nifty features.
I guess the reason why no up2date list of "what Zsh can do but Bash can't" exists is that most people who explore Zsh never really look back to Bash once they went the Zsh route. Bash v4 got some features from Zsh but there are so many small things that Zsh still provides that once you get used to them you never want to go back. :)
Your approach is valid. We're doing something similar (e.g. auto-building Debian packages on each commit/push) and use git's post-receive and svn's post-commit hook to detect the modified branch and then trigger a parameterized Jenkins build which takes the branch as argument. You can even trigger different kinds of builds/tests/... depending on the branch name. Due to the nature of branching inside Git it's much more fun with Git, but it basically works for subversion as well.
Regarding configuration management/firmware version: yes - especially, as you'd need to also rebuild disks in the dev/test environment with the identical configuration (firmware, disks,...), to ensure it's actually identical. And even if we neglect load/capacity/usage issues (problems might show up only under specific work loads), there are also further "invisible" layers/components like cables, NICs, switches,… and their firmware versions which are also relevant. Not exactly trivial. :)