Re: the safety of backing up a RAID array with a failed drive first, versus swapping the failed drive and rebuilding the array first.

I would think the main advantage of the “backup first” strategy is that it does not require any new disk writes, only reads.

Seems to me that there would likely be far more potential failures resulting from re-writing all the data/parity during a rebuild than simply reading what’s there onto a backup.

Get 1,000,000 hard drives in a room. Run them all and see when they fail. Let’s say that in the first year you had one hard drive fail every hour. That would be 8760 drives that would fail. During the second year you might also have 8760 drives fail. During the third year you might have 8760 drives fail. During the 4th year you might have 50,000 drives fail. During the 5th year all the remaining drives might fail.

What is the MTBF? You would clearly decide that the useful life of a hard drive is 3 years, because you start getting a lot of failures in the 4th year, and all of them failed during the 5th year. So you look at you average failure rate for the first three years. Well for the first three years, you had 1,000,000 drives running, and one failed every hour. But every hour you have 1,000,000 hard-drive hours accumulated. So you have one failure per 1,000,000 hours of operation. Thus your MTBF is 1,000,000 hours. MTBF means mean time between failures. You have a one failure for every 1,000,000 hours of operation, thus a 1 million hour MTBF.

Notice the fact that all the drives failed by the fifth year. The MTBF has nothing to do with the life expectancy.

On an unrelated note, I have not read any of the referenced papers, but it seems to me that the statistic showing clustered failures is totally bogus. It turns out that when you find a drive has failed and you go to rebuild the raid, it’s not that another drive fails during the rebuild, but rather that the other drive has in fact failed before the rebuild (failure meaning having unreadable data), but the failure is not discovered until the rebuild.

The industry is moving towards using AFR (Annual Failure Rate). The reason is that MTBF is really confusing, and AFR gives the consumer a better idea of what the number is. an AFR of 0.87% is equivalent to MTBF of 1,000,000. the equation is AFR = 1-exp(-8760/MTBF)

Both of these measures are POPULATION statistics. One would expect from a large population that a small fraction might be faulty or break earlier than expected. Most people can intuitively understand that about 1% of disks might fail in a single year, or there is a 1% chance of a disk failing in a year. They also do not link this failure rate with the disks lifetime. As such AFR is much more sensible metric for this type of information. and AFR=0.87% is exactly the same as MTBF of 1,000,000 hours.

This statistic also in no way defines how long a disk will last. That is the useful life value (say 30,000 POH (power on hours)). This will be linked to the warranty period, wear-out etc.

On a slightly different note…. The paper did not measure disk failures, rather, “disk replacements”. There is a difference between the two, namely mis-diagnosis. This may also help explain why she got a autocorrelation. If I incorrectly replace a disk that is faulty, I still leave the root cause of the problem, and am likely to repeat the same mistake a week or so latter…. hence the autocorrelation result.

My hypothesis is that the autocorrelation seen is caused by mis-diagnosis. Unfortunately I do not have the data to prove/disprove that hypothesis.

And for those who have already figured out that their 1.5M MTBF drives don’t last 150 years, but are not sure what that MTBF thing is… here’s a quickie:

Why your hard drive doesn’t last 150 years.

(There are about 8700 hours in a year, but to make this example simple, let’s call it 10,000.)

Here’s how MTBF works: it’s an aggregate of many units based on expected life of a single unit.

Let’s say you have a hard drive that is warranted to last 3 years, or 30,000 hours.

You put it in a server, and behold, it lasts 3 years. You take it out and put in a new one, and that also lasts 3 years. So you replace it with a new one, and that too…. well, you get it.

Let’s say you keep doing that and finally, on the 50th unit, only two years into it’s life, it breaks.

You now have 3 years or 30,000 hours per unit, times 50 units = 1,500,000.

And that’s your MTBF.

So anyone who says “Wow! MTBF of 1.5 million hours! that mean this thing will last (1.5M / 10000) 150 years!” -clearly- doesn’t know what they’re talking about.

(MTBF is more complex than my example, including “infant mortality” and “wear out” phases; “theoretical” vs “operational” MTBF and so on, but the gist of what’s here is correct.)

Cordially,

Tracy Valleau

“Don’t believe everything you think.”

Table 2. — “Node outages that were attributed to hardware problems broken down by the responsible hardware component.”

Component (HPC1)
CPU 44%
Memory 29%
Hard drive 16%
PCI motherboard 9%
Power supply 2%

Fully 82% of the failures were related to “solid state” components.

This in spite of the fact that the system population included 3,406 disks and 784 servers. DRAM was almost twice as likely to cause a failure and the CPUs were three times more likely to cause an outage. Moreover, 784 motherboards produced 9% of failures while 3,400 disks produced only 16%.

And this is a very high-end system, presumably “top-shelf” DRAM, CPU and motherboard components.

Also, from the text:

“…we have analyzed failure data covering any type of node outage, including those caused by hardware, software, network problems, environmental problems, or operator mistakes. The data was collected over a period of 9 years on more than 20 HPC clusters and contains detailed root cause information. We found that, for most HPC systems in this data,
more than 50% of all outages are attributed to hardware problems… Consistent with the data in Table 2, the two most common hardware components to cause a node outage are memory and CPU.”

So much for the myth of “solid state” reliability.

For some perspective, while CPU makers stopped publishing MTBF many years ago, and DRAM manufacturers have to my knowledge never published them, most motherboard manufacturers do publish — typically in the 100,000 hour range. So…if 784 motherboards produced 9% of failures, and 3,400 disks only produced 16%, then it seems that perhaps the numbers published by the disk drive makers are, in relative terms, not so wildly off the mark. It would appear (from a system/sub-system perspective) that disks are relatively much more reliable than the “solid state” components.

I wonder how people would react if they actually knew the MTBF numbers on stuff like DRAM and CPUs? Perhaps we should all remember that silicon DOES “wear out” (in a manner of speaking).

All this makes me wonder why everyone assumes that Flash SSD is going to be so much more reliable than other silicon. Are we to believe the ridiculous MTBF claims of the SSD makers (Intel sez 2,000,000 hrs), given the numbers on DRAM?

It will be interesting to see the results on the first large-scale deployments of flash-SSD. Unfortunately it will probably be five or more years that the “free ride” for SSD continues before folks begin to realize that solid-state in not necessarily more reliable than mechanical disks…and very frequently (in the case of DRAM and CPUs) less reliable!

A bit of a late comment here, but I think what’s even more interesting than bogus MTBFs for drives is the interesting difference in bit error rate for SCSI/FC vs. SATA drives. I just wrote an article on this, Are Fibre Channel and SCSI Drives More Reliable? It turns out that they are, at least for RAID, and not for the reason you might suspect! I think there’s a false market segmentation going on here…

Jered Floyd
CTO, Permabit Technology Corp.