The industry is moving towards using AFR (Annual Failure Rate). The reason is that MTBF is really confusing, and AFR gives the consumer a better idea of what the number is. an AFR of 0.87% is equivalent to MTBF of 1,000,000. the equation is AFR = 1-exp(-8760/MTBF)

Both of these measures are POPULATION statistics. One would expect from a large population that a small fraction might be faulty or break earlier than expected. Most people can intuitively understand that about 1% of disks might fail in a single year, or there is a 1% chance of a disk failing in a year. They also do not link this failure rate with the disks lifetime. As such AFR is much more sensible metric for this type of information. and AFR=0.87% is exactly the same as MTBF of 1,000,000 hours.

This statistic also in no way defines how long a disk will last. That is the useful life value (say 30,000 POH (power on hours)). This will be linked to the warranty period, wear-out etc.

On a slightly different note…. The paper did not measure disk failures, rather, “disk replacements”. There is a difference between the two, namely mis-diagnosis. This may also help explain why she got a autocorrelation. If I incorrectly replace a disk that is faulty, I still leave the root cause of the problem, and am likely to repeat the same mistake a week or so latter…. hence the autocorrelation result.

My hypothesis is that the autocorrelation seen is caused by mis-diagnosis. Unfortunately I do not have the data to prove/disprove that hypothesis.

And for those who have already figured out that their 1.5M MTBF drives don’t last 150 years, but are not sure what that MTBF thing is… here’s a quickie:

Why your hard drive doesn’t last 150 years.

(There are about 8700 hours in a year, but to make this example simple, let’s call it 10,000.)

Here’s how MTBF works: it’s an aggregate of many units based on expected life of a single unit.

Let’s say you have a hard drive that is warranted to last 3 years, or 30,000 hours.

You put it in a server, and behold, it lasts 3 years. You take it out and put in a new one, and that also lasts 3 years. So you replace it with a new one, and that too…. well, you get it.

Let’s say you keep doing that and finally, on the 50th unit, only two years into it’s life, it breaks.

You now have 3 years or 30,000 hours per unit, times 50 units = 1,500,000.

And that’s your MTBF.

So anyone who says “Wow! MTBF of 1.5 million hours! that mean this thing will last (1.5M / 10000) 150 years!” -clearly- doesn’t know what they’re talking about.

(MTBF is more complex than my example, including “infant mortality” and “wear out” phases; “theoretical” vs “operational” MTBF and so on, but the gist of what’s here is correct.)

Cordially,

Tracy Valleau

“Don’t believe everything you think.”

Table 2. — “Node outages that were attributed to hardware problems broken down by the responsible hardware component.”

Component (HPC1)
CPU 44%
Memory 29%
Hard drive 16%
PCI motherboard 9%
Power supply 2%

Fully 82% of the failures were related to “solid state” components.

This in spite of the fact that the system population included 3,406 disks and 784 servers. DRAM was almost twice as likely to cause a failure and the CPUs were three times more likely to cause an outage. Moreover, 784 motherboards produced 9% of failures while 3,400 disks produced only 16%.

And this is a very high-end system, presumably “top-shelf” DRAM, CPU and motherboard components.

Also, from the text:

“…we have analyzed failure data covering any type of node outage, including those caused by hardware, software, network problems, environmental problems, or operator mistakes. The data was collected over a period of 9 years on more than 20 HPC clusters and contains detailed root cause information. We found that, for most HPC systems in this data,
more than 50% of all outages are attributed to hardware problems… Consistent with the data in Table 2, the two most common hardware components to cause a node outage are memory and CPU.”

So much for the myth of “solid state” reliability.

For some perspective, while CPU makers stopped publishing MTBF many years ago, and DRAM manufacturers have to my knowledge never published them, most motherboard manufacturers do publish — typically in the 100,000 hour range. So…if 784 motherboards produced 9% of failures, and 3,400 disks only produced 16%, then it seems that perhaps the numbers published by the disk drive makers are, in relative terms, not so wildly off the mark. It would appear (from a system/sub-system perspective) that disks are relatively much more reliable than the “solid state” components.

I wonder how people would react if they actually knew the MTBF numbers on stuff like DRAM and CPUs? Perhaps we should all remember that silicon DOES “wear out” (in a manner of speaking).

All this makes me wonder why everyone assumes that Flash SSD is going to be so much more reliable than other silicon. Are we to believe the ridiculous MTBF claims of the SSD makers (Intel sez 2,000,000 hrs), given the numbers on DRAM?

It will be interesting to see the results on the first large-scale deployments of flash-SSD. Unfortunately it will probably be five or more years that the “free ride” for SSD continues before folks begin to realize that solid-state in not necessarily more reliable than mechanical disks…and very frequently (in the case of DRAM and CPUs) less reliable!

A bit of a late comment here, but I think what’s even more interesting than bogus MTBFs for drives is the interesting difference in bit error rate for SCSI/FC vs. SATA drives. I just wrote an article on this, Are Fibre Channel and SCSI Drives More Reliable? It turns out that they are, at least for RAID, and not for the reason you might suspect! I think there’s a false market segmentation going on here…

Jered Floyd
CTO, Permabit Technology Corp.

Lately, there has been a lot of buzz in the enterprise storage arena about whether so-called “enterprise drives” are really any better than plain-Jane hard drives in Enterprise applications. This came to a head with the controversial findi…

A few assumptions:
-The target storage system needs to be able to contain 10 TB worth of data
-The storage system needs to be scalable
-The storage system needs optimal data security vs. costs. (of course data triplication is nice, but most of us, libraries including, don’t have that much money)
-The storage system needs to be web-accessible
-The storage system needs to be disaster-proof

If you are searching for something that should fit these needs as well, this is probably your best solution:

Two seperate servers stored at seperate locations (cheapest way of avoiding data-loss through distasters). Configure the first server for RAID5EE (hot spare integration) and the second for RAID60 (SAN). Use 500GB enterprise drives for your first server and 500GB nearline drives for the SAN. Make the first server backup daily to the SAN. Perform nightly checkdisks so you can determine when new spare drives should be ordered. And last, but not least, make sure you have the money to buy a whole new server environment within 7 years.

That isn’t anywhere near cheap, but it’s most cost-effective almost 100% guarantee for preserving your data. This configuration doesn’t necessarily have to be optimal for the next generation of hardware you will buy.

Perhaps noone is helped with this, but I’ll be happy if it just helps Someone. Just some (nearly offtopic) sidepoints, for cheap home RAID’s, check the Intel Matrix RAID solution. For future archiving, pay attention to holographic storage development. I’ll save you the other random findings of my study

Of course not. Radically different technologies, as you know.

Packing twice the blocks on the same physical spindle as onother drive built with the SAME TECHNOLOGY will and does result in twice the number of bad blocks for the same physical damage to, or inperfection in the platter.

There is no free lunch, and you do indeed get what you pay for. It doesn’t show up in this study, because this study doesn’t take into account the primary advantage of enterprise diks, which is twice the phyical media allocated to each block using the same platter technology as their consumer grade cousions.

Even if FC, SAS and SATA all do inded have similar rates of failure for their mechansisms, which I wouldn’t doubt, if you’re willing to pay for RAID redundancy, why not media redundancy teh blocks on your platter?

Apart from the advantages on the contoller board of FC or SAS, what your paying for is twice the saftey of the data contained on those blocks. If you don’t care about what lives on those blocks, I guarantee you someone will when they go missing.

Just my two cents.

-ted

Of course I’m talking about data corruption due to bad blocks, and the fact that only drive-wide hardware failures were taken into account in this study is the basis of my point.

Robin tried to dismiss my point as being architectural and not real world, yet my whole point is that this study misses some critical aspects of real world experience, which is that when you go to fetch data, and you can’t get it because the blocks are bad, or you can’t rebuild a portion of the data after a failure because the block are bad, then whoever needed that data is going to consider it to be a failture, regardless of whether the RAID controller labels the disk as failed or not.

Data corruption = failure. Anyone who tells you diffrent is trying to sell you something.

-ted

On a current “big iron” application, we made the chang from SATA drives to Fibre drives before implementation this past year. Storage costs increased exactly 100% for the same amount of storage, not 400 to 600% as has been suggested. So, if you’re thinking of doubling or “tripling” up on SATA, look at the costs also.

Facility costs on “big iron” projects are huge. The costs to double, or triple, up the space to stand up SATA and the added costs for cooling these drives over a period of years can be staggering.

Now if your just looking at a simple “one for one” replacment Fibre with SATA, with the same size of storage in the end, then it’s worth looking into because storage costs could be reduced by half.

As an example our costs could be reduced from $4 million to $2. I’ll be taking a look, and will have to make a complicated business decision.