## RAID and hard disk read errors

27.02.2021 19:07

I have been preoccupied with data storage issues lately. What I though would be a simple installation of a solid state drive into my desktop PC turned into a month-long project. I found out I need to design and make new drive rails, decided to do some overdue restructuring of the RAID array and then had to replace two failing drives. In any case, a thing that caught my eye recently was a warning about RAID 5 I saw repeated on a few places on the web:

Note: RAID 5 is a common choice due to its combination of speed and data redundancy. The caveat is that if one drive were to fail and another drive failed before that drive was replaced, all data will be lost. Furthermore, with modern disk sizes and expected unrecoverable read error (URE) rates on consumer disks, the rebuild of a 4TiB array is expected (i.e. higher than 50% chance) to have at least one URE. Because of this, RAID 5 is no longer advised by the storage industry.

This is how I understand the second part of the warning: it talks about a RAID 5 array with total usable capacity of 4 TiB. Such an array would typically consist of three 2-terabyte disks. In the described scenario one of the disks has failed and, after adding in a replacement drive, the array is restored by reading the contents of the remaining two disks. This means we need to read out 2 times 2 terabytes of data without errors to successfully restore the array.

I was surprised by the stated higher-than-50% chance of a read error during this rebuild procedure. It seemed too high given my experience. Hence I've looked up the reliability section of the datasheet for the new P300-series, 2 TB Toshiba desktop-class hard drive I just bought:

I'm a bit suspicious of the way probability is specified here. Strictly reading the exponential notation, 10E14 means that the probability of an unrecoverable error (URE) is one error per 10⋅1014 bits. Expressed as probability of an error when reading a single bit:

P_{URE} = \frac{1}{10\cdot10^{14}} = 10^{-15}

In another datasheet for a different series of drives (however this time for data center instead of consumer use) the error rate is given as 10 errors per 1016 bits. This again gives the same error probability of 10-15.

Consider this probability for a second. It's such a fantastically low number. I don't remember ever encountering an actual technical specification that would involve a probability that has a one preceded by fifteen zeros - or in other words - fifteen nines of reliability.

The number is just on the edge of what you can represent with the common 64-bit double-precision floating-point format. If using a tool like numpy that only uses double-precision, any calculations with such values need to be done extra carefully to ensure that loss of numerical precision doesn't lead to nonsensical results.

Hard drives tend to use SI prefixes instead of binary, so I'll do the calculation for 4 terabytes instead of 4 tebibytes like it says in the quote:

n = 4 \cdot 10^{12} \cdot 8 \mathrm{bits}

For this calculation it doesn't matter whether we're reading this number of bites from one or two drives since the URE probabilities are assumed independent. The probability of getting at least one error during the rebuild is:

P_{rebuild-error} = 1 - (1 - P_{URE})^n \approx 3.1\%

Note that if I read 10E14 in the original reliability specification as 1014, the probability of a rebuild error goes up to 27%.

This comes out a bit more optimistic than the higher-than-50% figure given in the warning, at least for this specific series of hard drives. I guess whether 3.1% is still too high depends on how much you value your data. However consider that in the original scenario this is the probability of an error given that another hard drive in the array has already catastrophically failed. So the actual probability of data loss is this multiplied with the (unknown) probability of a complete drive failure.

Then again, consider that this is a desktop drive. It is not meant to be put into a RAID array and is typically used without any redundancy. Some people will even scream at you if you use desktop drives in a RAID due to timeout issues. Without any redundancy this probability directly becomes the probability of data loss. And that seems exceedingly high - especially considering that drives up to 8 TB seem to be sold with this same error rate specification. Even with that amazing reliability of reading a single bit, modern drives are simply so large that the vanishingly tiny error probabilities add up.

Posted by | Categories: Life