In a FAST ’08 paper to be presented this week – Are Disks the Dominant Contributor for Storage Failures? A Comprehensive Study of Storage Subsystem Failure Characteristics – authors Weihang Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky analyze logs from 39,000 systems over 44 months to get answers.
1.8 million disks in 155,000 shelves
NetApp provided data from a variety of systems, including near-line, low-end, mid-range and high-end arrays. The team analyzed the log reports to understand what components led to failures.
The 15 page paper offers some interesting findings
- Physical interconnect failures are a significant contributor – anywhere from 27-68% – of storage subsystem failures.
- Subsystem failure rates that use the same disk models show similar disk failure rates – but the subsystem failure rates vary significantly.
- Enclosures have a strong impact on subsystem failures. Some enclosures work better with some drives than others.
- Dual-redundant FC shelf interconnects reduce annual failure rates 30-40%.
- Interconnect and protocol failure rates are much more bursty than disk failures. Some 48% of overall subsystem failure arrive at the same shelf within 10,000 seconds (~ 3 hours) of the previous failure.
- As interconnect failures are so bursty, resilience mechanisms beyond RAID are required to achieve subsystem availability.
They also found that enterprise drives had an AFR consistent with manufacturer specs – less than 1% AFR. This result derives from looking at the disks as the system does rather than as users see them.
The StorageMojo take
Interconnects, especially connectors, have long been fingered as a significant cause of the equipment problems – and not just in storage. While the team seems to report that interconnects are a greater cause of subsystem failure than disks, there seems to be some room for disagreement about what the numbers are telling us.
For example, this result doesn’t fully explain the delta between what disk users have found and the “trouble not found” rates that manufacturers report. Even if you accept the common 50% TNF vendors report, drive failures are still higher than this research finds.
Perhaps we should conclude that NetApp’s engineering is higher quality than the general run of storage arrays. Or perhaps system log analysis is still a dark art whose results are more indicative than conclusive.
Comments welcome, as always. I’m at the FAST ’08 conference this week in the San Jose Fairmont hotel.