Andy Klein from Backblaze here. We have done some survival analysis (kaplan-meier curves). In our case, we need to have a reasonable number of failures over the observation period to get decent results. You can take a look at some of our work here: https://www.backblaze.com/blog/hard-drive-life-expectancy/ and see if that is what you were expecting.
Yes! Thank you. I've been thinking about scratching this itch myself but it turns out you did already.
In particular the shape of the survival curve interests me -- you hear so many things about exponential here, bathtub there, but ver little data. I will read once I have a spare moment.
Andy Klein from Backblaze here. Your point is a good one in that temperature fluctuation can be an important factor. We actually sample smart stats, which contain the temperature attribute, multiple times a day looking for such changes. The Drives Stats data is captured once a day, so it looks static, but behind the scenes the monitoring is more dynamic.
Do you have plans to look at whether higher fluctuations translate into higher failure rates? Not sure whether you have historical data on this, but I would be really interested in this aspect, even if you can only run the stats on a smaller number of drives or shorter time periods.
Maybe dividing the drives roughly in "higher than average variability" and "low variability" and then looking at the AFR for this subset can show some relation. Of course as the AFR for many drives is already quite low, the effect might be too small to distinguish from noise.
On the topic of temperatures: Have you run an analysis whether a drive increasing in temperature (or maybe even decreasing) compared to its base line and "neighbors" results in a higher chance of failure?
Andy from Backblaze here:
The WDC model (WUH721816ALE6L0) seems to only be available in Europe, not the US. It was 549,00 EUR last time I looked back in November. I believe the last digit is about data erasure with 0 being "Instant Secure Erase", 1 being Bulk Data Encryption (SATA), 4 being Secure Erase (overwrite only). We only have reliability data for the model listed.
Andy for Backblaze here: I looked at that 3-4 years ago. It looks like power cycling increased failure rates, but we don't power cycle our systems very often, maybe 1-2 a year, so not the best use case. This is on my list for a relook one of these days, if we find anything interesting we'll let folks know.
Andy for Backblaze here: A while back we did an analysis of drive failure over time, i.e. the bathtub curve. It is probably a good idea to update that, as I believe we are seeing lower failure rates upfront these days.
Andy at Backblaze here. We do look at drive model failure over time. We did a post on this topic several years ago, 2015? At the time, most drive followed the bathtub curve of failure, but I'm not sure that is still the case. I think its time to update that report.
Andy from Backblaze here. Larger drives do take longer to rebuild, but to date we haven't changed the encoding algorithms we built. There are other strategies like cloning which can reduce rebuild time. We can also prioritize rebuilds or drop a drive into read-only mode as needed. The system was built expecting drive failures.
Andy at Backblaze here. All the drives are in data centers with temps around the 75-78 degree mark. Vibrations are kept to a minimum via the chassis design. We publish the data, including the SMART stats for all of the drives and there are attributes for temperature (SMART 194) and vibration (multiple) see https://en.wikipedia.org/wiki/S.M.A.R.T. for more info in SMART attributes.
Thanks Andy! Without me digging through the SMART data, has there been any difference in data center temps over the years for Backblaze? I ask because I've personally seen disk life vary greatly between "warm" datacenters (closer to 79F) and "cool" datacenters (closer to 72F). I don't have a huge dataset, only anecdotal evidence, but it seems to me temperature plays a pretty big role in drive longevity. Have you guys found the same, or is this a variable not controllable by Backblaze?
Andy from Backblaze here. Actually you can download a spreadsheet with all the data from the tables. There's a link at the end of the post. Better than parsing HTML for the data.
The spreadsheets are identical to the screenshots you took of them. The images aren't responsive, aren't available to assistive technologies, provide no text alternative, do not respect a user's text settings, cannot be translated etc. Why is that better than HTML?
Andy from Backblaze here. Nice thinking about the bulk ordering and considerations for RAID. All things we have considered. We use our own Reed-Solomon encoding with a 17/3 set-up across 20 drives across 20 different systems, we call that a Tome. Then we have a specific protocol we follow as drives fail in a Tome to protect the data at all costs. We have the luxury for example to stop writing to a given Tome as we have plenty of others available. This takes a lot of the stress off of the system.
Your thoughts on bulk buys and bad drive batches/models is solid. We test drives in small batches first, and we follow drive failures so we don't get to the point of hitting the wall. It would be great to mix and match drives, but you end up with a system that maxes out at the least performant drive. So not optimal.
Do you power cycle the disks during regular operations?
There is a failure mode in disks which can be modelled as something fails on the drive, but it continues to work fine for maybe months or years until the next power cycle, upon which it then won't work.
Obviously that's a problem for redundancy schemes because you think you have plenty of redundancy till there is a power outage and suddenly loads of drives fail at once.
I have never seen any of your reports measuring or reporting on these 'fail after power cycle' events, which is surprising.
I recall reading about your triple redundancy with R-S codes, but it's good to restate it for each audience.
From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents, and it feels like a waste. But it's possible that my comfort levels for available disk space are a lot more conservative than other people's, and so the difference is less pronounced in a 'normal' storage situation.
For instance, an array that's at 80% capacity takes 25% longer to rebuild than I wish it would, whereas an array that's at 66% capacity takes 50% longer.
> it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents
This is only true for drive-level raid rather than filesystem level raid, or a non-raid solution like ceph's replication.
ZFS's filesystem raid can repair a raid in time proportional to the amount of data stored in it.
mdadm and raid controllers aren't aware of which parts of the block device are in use or not, and thus have to repair the whole drive.
It's exceedingly likely that backblaze's solution does not require repairing entire block devices, but rather is likely to be closer to ceph, where only the in-use portion of a failed drive must be considered / must find a new home.
I think raid and distributed storage systems (like backblaze or ceph) are more different than they are alike.
> From the behavior of my RAID (which also uses Reed Solomon, doesn't it?)
Maybe. mdadm raid5 doesn't, nor does mdadm raid1 or raid10. I think mdadm's raid6 does.
You could then for example have 18/2, and then group together 400 drives in a 2nd layer of 19/1. Hey, I reckon you could do 19/1,39/1, reducing your storage costs 7.5%...
Sure, the worst case rebuild cost is much worse, but overall data loss probability is far lower, and a 2nd layer rebuild is a very rare event, and in that case, your customers totally prefer a few extra seconds latency over an email reporting their data is lost...
I assume you mostly do streaming rather than random writes, so the overhead is evenly spread amongst the disks, and is the same 15% as your current scheme.
