Have you ever noticed that it is difficult to get good information about how flash works? The vendors know but they’ve never been terribly forthcoming.

For example, how does flash wear out? When most things break you lose their contents. But once flash stops working your data is still there. Huh?

And the fact that flash is a wearing medium spooks many people. How should we think about flash? Can we live with a wearing medium?

Or write amplification? How does that work? What can be done to reduce it?

That’s why it was a pleasure to sit down with Rob Ober of LSI. Rob is an LSI Fellow and system architect with deep technical knowledge of flash and how it interacts with systems and applications.

Rob holds dozens of patents and is articulate and open. Plus he’s a very nice guy.

I distilled down what I learned and some of Rob’s key points into a StorageMojo video white paper that LSI commissioned. If you are curious about flash, how it works, how it fails and how it can be turned into an enterprise class storage medium, you’ll find the video informative.

At least I did my level best to make it so, including video from Wilson Canyon, one of my favorite local hikes. Here’s the video:

The StorageMojo take
As a thought experiment I sometimes wonder about how storage would be different if IBM had invented flash back in 1956 instead of the RAMAC disk drive. What it reads were fast and free while writes were expensive?

That’s essentially the problem we’re trying to solve today. Except today we have an installed base of a couple billion disk drives and decades of driver, OS and application development all predicated on disk performance.

We’re still in the early days of flash integration, even though forward-leaning architects have been working on it for 6 years or more. Thanks to flash – and cloud – storage has never been more vibrant or exciting.

Courteous comments welcome, of course. Feel free to ask about anything in the video that wasn’t clear or didn’t go deep enough. Your questions help me understand what you find valuable.

There are years where new ideas and concepts explode. And there are years of consolidation. 2013 will be the latter.

The storage industry has a lot to digest. Here are some of the issues.

ReRAM
Would-be vendors of the enterprise NAND flash replacement technology, Resistance RAM or ReRAM, will be hunting for launch customers. While their new product won’t be as cheap as NAND flash, it doesn’t need to be because enterprise customers will pay for greater speed, reliability and durability. The medium isn’t a big cost driver.

If ReRAM launches this year, who will lead the charge? I’d bet on either Intel, Violin Memory, Kaminario or Samsung. Intel and Samsung would incorporate ReRAM into their SSDs, while Kaminario and Violin would build it into their high-end solid-state arrays.

Disks
Disk drive vendors are trying to figure out how to survive the deflation of their largest market due to tablets and smart phones. They’re counting on hybrid drives that blunt the performance advantages of pure SSDs. But they’ll need a higher level of commitment to architecting the right products than I’ve seen so far.

We’ll also see if Hitachi’s helium drives can make it in the marketplace. Hitachi/WD must have a major data center launch customer – Google, Facebook or Amazon – since no major server vendor would offer them without a 2nd source.

Next-gen systems
Architects from next-gen storage companies such as Avere, Nimble Storage, Amplidata, Nimbus Data, Fusion-io, Starboard, Nutanix, Violin Memory, Tegile and many others will be tuning their systems and software – mostly software these days – based on real customer workloads instead of guesstimates.

HP’s converged storage is a special case. Will Donatelli be able to motivate HP’s enterprise sales force to move the product?

Cloud
Cloud storage and computing will continue to bend the arc of enterprise new technology adoption. CFOs will become tougher in their questioning of CIOs given the obvious CapEx advantages of cloud infrastructure.

Big Data
Multiple slogs for Big Data. Privacy concerns and regulation. The plethora of infrastructure options slows decision making and PO writing. Dawning awareness that the output is only as good as the questions you ask – technology is secondary.

But we’re only at the beginning of the Big Data revolution.

The StorageMojo take
The last decade has seen an explosion of new storage architectures and technologies. That’s as it should be, since storage is the hardest part of information infrastructure – and the most critical.

But while innovation won’t halt this year, it’s time for the market to sort things out. CIOs who value their jobs will be looking beyond their usual suppliers to better compete with cloud IaaS vendors.

It will be an interesting year.

Courteous comments welcome, of course. I’ve done work for several of the mentioned vendors, but for most I haven’t.

The flooding-induced disk drive drought is over, except for some channel filling, but the last year’s drive vendor profitability may be the last good news they see for years. Trends are conspiring against disk drive vendors.

1st, worldwide PC sales are slowing and are projected to fall for the 1st time in 11 years. PCs are the largest single market for disk drives.

It would be OK if PCs were being replaced by devices with hard drives, but they aren’t. Mobile phones and tablets are replacing PCs for casual users.

2nd, the SSD market, expected to be about 40 million units this year, is taking an additional several percentage points of growth away from the disk drive. Ultrabooks and Apple’s hot selling MacBook Air are eating into the profitable mobile drive sector.

3rd, the driver shortage may have affected consumer behavior for the long-term. People who never thought about their capacity use and need were suddenly forced to consider it. And many found they were buying much more capacity than they needed.

While speculative, this is not unlike the 1973 Arab oil embargo, where many drivers decided against cars getting 10 miles to the gallon.

Market dislocations always force reconsideration of habits and assumptions. The only issue is how long the changes last.

Bright spots
The trends are not all grim. Cloud providers are buying millions of disk drives. Windows 8 is poised to provide some pop for the PC market. The growing consumer embrace of video will continue to gobble up storage capacity.

Vendors have responded, finally, to the last by moving into the external drive market. By all accounts this is been a major success. Their margins need all the help they can get.

Finally, if Ultrabooks do take off, hybrid drives will deserve significant credit – and vendors will get the volume they need.

As disk drives go so goes storage
This isn’t just a question of overcapacity or margin pressure. The disk drive industry is facing major investments – patterned media and heat assisted magnetic recording – to take areal density to the next plateau.

The StorageMojo take
If the drive vendors cannot afford new technology we may see the end of glorious 40% annual capacity increases that disk vendors have gifted to the storage industry for decades. And that would force much greater dislocations than we’ve seen in the last year.

But slowly – glacially – the drive vendors are starting to realize that they have to go after niches they’d spurned before. Like toothpaste – now available in Extra Bright, Baking Soda, Sensitive, Gel, Mint and Original Flavor – drives need more variety.

While 7mm, hybrid, helium and external drives are a good start, there are still major market opportunities for Seagate, WD and Toshiba to address. Archive drives – shingled? – that replace LTO tapes are one option. Home media servers designed with the attention to detail that Apple specializes in are another. Cloud gateways amd Drobo-class arrays are even more.

These all share the key feature of being cheaper than the enterprise storage vendors can afford to sell, so they don’t compete with these territorial customers. But will drive vendors make the investments in software and marketing they’ll need to be successful?

If history is a reliable guide, no. But maybe desperation will lead them to overcome their inbred conservatism and surprise us all.

Courteous comments welcome, of course. I want archive-quality disk drives!

Over the years StorageMojo has seen several architectures that just seemed smart, but whose market potential was blighted by management and funding issues. Violin Memory was one. Coraid was another.

In both cases a new CEO – at Coraid, Kevin Brown – has made a world of difference. New funding, new customers, more focus. All good.

What does Coraid do?
Storage, of course. Inexpensive block-based, scale-out, network storage. Global namespace.

Define inexpensive
How does $0.40/GB sound?

How?
Let’s start at the server. The HBAs are Intel NICs that are re-programmed to present themselves as SCSI controllers. The server sees a SCSI controller, but it’s Ethernet out the other side.

Running over the network is a connectionless datagram protocol that accesses the storage on commodity x86 servers. The protocol is fast and lightweight, being based on the simple and well-understood ATA command set.

There’s a router for connecting sites, but most users use it in a single site.

Who?
People who want a lot of inexpensive block storage. Average order is ≈175 TB. Average sales cycle is 30 days.

Coraid is a horizontal play, but they get interest from the usual big data suspects: media and entertainment; science; service providers; private clouds.

The StorageMojo take
Coraid takes the commodity game seriously: commodity servers; commodity HBAs; commodity open-source protocol; commodity network; and commodity SCSI drivers. Not much custom engineering required.

It’s prices reflect that. There’s a huge unmet need for fast, cheap storage that Coraid is tapping. Their biggest problem is that they do things differently than everyone else, and it takes people a while to decide to work with those differences.

Which is a shame. The more competition for the block storage dollar the better it is for all consumers.

Courteous comments welcome, of course. I saw Coraid as part of a Tech Field Day program, where they, among other vendors, helped pay for transportation, food and lodging.

Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

If SSDs are so great, shouldn’t we see the results in TPC-C benchmarks? They are, and we do.

But there are some surprises.

Cost
Looking at the TPC-C top 10 performance results showed the dramatic impact SSDs have had on the cost per thousand transactions (tpmC).

• There are no top-10 disk-only results after 2009.
• The most expensive top-10 SSD result is some 15% cheaper than the least expensive disk-based result – and the other SSD results are much less.
• No top-10 results posted during 2009 – the depth of the great recession.

Capacity
The conventional wisdom has it that disks must be way over-configured to get enough IOPS. You’d expect to see disk solutions have a lot more capacity than SSD solutions in top-10 results.

But we don’t:

The highest capacity – 1760 TB – is for an Oracle SSD-based solution. Yet the lowest capacity solution – 83 TB – is also SSD-based and is also the cheapest per tpmC.

Are we seeing issues with the rest of the infrastructure?

The StorageMojo take
I’ll be taking a deeper dive into the data, but perceptions may be at odds with what this limited set of performance focused benchmarks is showing us.

Readers: what do you think?

Courteous comments welcome, of course. Events beyond my control have reduced StorageMojo’s usual posting frequency. Hope to get things back to normal over the next several weeks.

Deduplication has been accepted as an enterprise-class compression technology. Is it time for data compression to be a standard feature of primary storage?

I’ve been doing some work for Nimble Storage a cool Valley startup. Talking to co-founder Varun Mehta, he mentioned that Nimble’s storage/backup/archive appliance does data compression on all data, all the time.

That’s right, primary storage on Nimble’s box is always compressed. Not only that, all their performance numbers are quoted with compressed data.

They aren’t kidding.

Venerable compression
Data compression is one of the oldest computer storage technologies around. Bell Labs mathematician Claude Shannon published A Mathematical Theory of Communication in 1948 which, among other things, laid out the math behind compression.

The ratio of the entropy of a source to the maximum value it could have while still restricted to the same symbols will be called its relative entropy. This is the maximum compression possible when we encode into the same alphabet. One minus the relative entropy is the redundancy. The redundancy of ordinary English, not considering statistical structure over greater distances than about eight letters, is roughly 50%.

In line compression has been part of every enterprise tape drive for decades. The algorithms – Lempel-Ziv was big 20 years ago – have been tuned to a fare-thee-well.

Compression is as thoroughly wrung out as any technology in the data center.

So why don’t we use it everywhere, like Nimble?

Not about capacity
The doubling of capacity from compression is not the big win. The larger benefit is that it more than doubles the internal bandwidth of the array – because bandwidth is more expensive than capacity.

And bandwidth is more important than capacity. As John von Neumann noted in his First Draft of a Report on the EDVAC (pdf):

This result deserves to be noted. It shows in a most striking way where the real difficulty, the main bottleneck, of an automatic very high speed computing device lies: At the memory.

Varun reports that Nimble’s comdec operates at wire speed on a multicore CPU, no ASIC or FPGA required. It must increase latency, but given Nimble’s focus on full stripe writes the increase in bandwidth must more than make up for it.

The StorageMojo take
Since it is possible to perform wire-speed compression/decompression with a commodity CPU, why not everywhere?

Will RAID controllers stumble reconstructing compressed data? Is compressed data more prone to corruption? Is bandwidth so cheap that we don’t need more?

I don’t think so, but I’m open to dissenting opinions. With disk capacity growth slowing comdec everywhere is a good way to increase performance, reduce $/GB and have something new to show customers.

Courteous comments welcome, of course. StorageMojo dove into this 5 years ago in 25x data compression made simple.

A hybrid of SSD and hard drive that gives you the best of both worlds. That’s the theory anyway. But I won’t keep you in suspense: I think Seagate has hit a home run with their new hybrid XT architecture.

Specs
Take a standard issue 7200 rpm, 500 GB notebook SATA drive. Add 4 GB of fast and reliable single-level cell (SLC) flash. Make it look like a standard drive to the OS by keeping all the magic internal.

Give it smarts to learn about high-frequency small-block transfers. Put those blocks in the flash and voilà: super-fast small block access; leaving big sequential I/Os to the disk. The algorithm

The SSD is 2 SLC chips, each with about 75 MB/sec bandwidth. If the flash should fail – more likely than wearing out IMHO – you still have a perfectly good 7200 rpm disk. The algorithm looks to provide the most benefit on the most used apps.

Theory
Disk drives offer cheap capacity and good large read/write (R/W) bandwidth. The R/W bandwidth improves over time as drive capacities grow due to higher bit density.

Flash SSDs offer sub-millisecond access times at a high price: $2-$3/GB. SSD access times are about 150x faster than a notebook hard drive, while R/W bandwidth is only about 2x faster.

Maximum bang for the buck? SSD to store many small files – like DLLs in Windows – to minimize accesses, and let the disk handle the large R/W traffic. Why pay 40x for a 2x performance boost?

Flash capacity? 4 GB can store 4 million 1k files. That saves a lot of access time.

Test
An early 2009 17″ unibody MacBook Pro with 4 GB RAM and 2.66 GHz Core Duo 2. Swapped the current 7200 rpm, 500 GB Hitachi for a 7200 rpm, 500 GB Momentus XT with 4 GB SLC flash.

For the first test I formatted it with HFS+ using the Mac’s Disk Utility. Then I used the excellent Carbon Copy Cloner to clone the system disk to the XT.

Results
Boot performance sucked compared to the old drive’s ~45 second boot:

• First boot took 2 minutes 4 seconds
• Second took 1 minute 19 seconds
• Third took 1 minute 6 seconds

After consulting with Seagate I did a clean install of OS X. Reformatted the XT, installed Mac OS 10.6 Snow Leopard, migrated accounts and data. Took 3 hours, but the process worked perfectly.

New boot times went to ~45 seconds again. Not what I had hoped – the SSD on my MacBook Air booted in 15-20 seconds – but competitive. Other testers have had better results so Seagate is sending me another drive. I’ll update the results later this week.

Application results were much more impressive. Because the drive learns, the first time you bring up an app it happens at disk speed. But the 2nd time!

Mail startup went from 5 seconds to 1.5. Microsoft Word startup went from about 12 seconds to less than 3. See for yourself in this 15 second clip:

And the bigger the app, the bigger the potential speed up. Look at – or not – this 1 minute clip of the 1st and 2nd time opening Final Cut Pro, the Mac pro-level video editing app:

From 45 seconds to 10 17? Way faster than my Mac Pro’s 10k system disk.

Note that rarely opened apps won’t do as well as the algorithm favors apps that get used more often. Pure SSDs don’t have that variability.

Pricing
Announcement pricing:

• 500 GB/$156
• 320 GB/$122
• 250 GB/$113

These prices are about double the street prices for the non-hybrid drives. Once Seagate gets some competition they should drop.

The StorageMojo take
Looks like a home run for Seagate. The architecture is clean, the performance advantages are real, and pricing is not too bad – especially compared to SSDs.

SSD adoption has stalled because flash prices have firmed up. Few are willing to pay $200-$300 for a smallish SSD in a $700 notebook. But $100? OK.

Seagate could use cheaper MLC flash in 8-16 GB caches without wearing it out. There is every reason to put this in 3.5″ drives as well. After all, if the flash does fail all your data is on the hard drive – no loss there.

By blurring the performance difference between disk and SSD, these drives will ensure that hard drives dominate for at least another decade. And they’ll put pressure on SSD prices.

But don’t count on these showing up in RAID arrays soon. Arrays already have a lot of moving parts and hybrid drives add some subtle wrinkles. Like how do you know what a small block transfer is? Or ensure you don’t have stale data? Not sure cached array controllers need this.

Courteous comments welcome, of course. No money changed hands but I’m not excited about returning the review unit. Also I sent a note off to CCC’s developer to alert him to what I saw – maybe he can figure it out. Update:As a number of commenters pointed out – thanks! – I screwed up: the original video showed a reboot from system cache. When I got the time I reshot the FCP startup after a reboot. It was still 40% of a regular drive and still much faster than a 10k drive on my Mac Pro. Sorry about that! End update.

Shouting at a disk drive will cause it to stop. But what about the constant nagging they get in busy data centers? That’s a bigger problem.

Bad, bad, bad vibrations
The use of consumer-grade SATA drives in the enterprise raises the concern. A 2005 study, Performance Impact of External Vibration on Consumer-grade and Enterprise-class Disk Drives, by Thomas Ruwart and Yingping Lu found that consumer

. . . disk drives are more sensitive to the vibration from physically coupled adjacent disk drives. . . . [G]reater care needs to be taken in enclosure design, particularly for the 3.5-inch form factor disk drives due to their higher-energy seek operations when compared to seek operations on a 2.5-inch form factor disk drive.

Yet another reason to end-of-life 3.5″ drives.

Mechanical engineers know that drive vibration is a serious problem. How serious is the question. We do know that steel does a good job of transmitting vibrations as this graph shows:
Courtesy Rockwell Automation
An internal study of a vibration-damping rack by a major storage and server vendor obtained by StorageMojo – and not available on the web – found a

20% decrease in IO throughput performance (read) and 25% increase in total time to perform a task of typical 2U servers resulting in 25% increase in energy consumption . . . when vibration level went up from 0.1 GRMS to 1 GRMS. This is a conservative result, as newer HDDs are more sensitive to vibration, “write operation” is 20% more sensitive than “read operation” to vibration and effect of vibration on rotating components , HDDs & fans, are not accounted for.

[GRMS - properly G_rms - is the root mean square of acceleration measured in g's. Check out The Calculation of GRMS (pdf) for a technical discussion.]

Drives use accelerometers to compensate for shock and linear and rotational vibration but the info isn’t available externally so it’s difficult to directly quantify vibration impact. But data centers are noisy places quivering with the vibration of fans, air conditioners, disk drives and 60 cycle hum, all supported by steel racks.

A limited study
In a USENIX paper presented at the SustainIT ’10 conference Julian Turner reported on limited tests of a prototype anti-vibration rack. The AVR-1000 is an engineered carbon fiber composite rack designed to dissipate vibration across a wide frequency range.

His observations include:

Performance improvements for random reads ranged from 56% to 246% while improvements for random writes ranged from 34% to 88% for a defined set of industry benchmarks. Streaming sequential reads and writes had a much smaller performance improvement. . . .

The internal study found similar results:

IO throughput performance (write) increased by 285% and total time and energy to perform a task of a 4U storage server with 48 HDDs decreased by 64% when vibration level went down from 1 GRMS to 0.1 GRMS.

Evidently the combination of random head movements and vibration has a substantial effect on disk read performance.

For the enterprise and Internet-scale data centers the implications are substantial:

• Energy savings. Anti-vibration racks can save power by improving disk performance and reducing run times.
• SSD value. Flash SSDs have fast random read access. But disks can improve their performance by 50% through vibration damping, changing the SSD value proposition.
• Array sizing. Enterprise arrays are over-configured to improve performance. If disks were suddenly 50% faster that could be reduced or, alternatively, the utilization could be increased.

The StorageMojo take
The research is limited, but everything we know about disks and vibration today suggests this is real. Assuming further research finds similar results we could see an explosion of products using engineered materials to improve disk performance.

This is already the norm in chip fabs, where nanometer precision requires extensive vibration damping. Disk feature sizes are even smaller, so why not?

Drive vendors should make accelerometer data available to researchers. Vibration-damping racks could make disks much more competitive with SSDs – and mean billions to drive vendors – but only if we have the data.

For home users of multi-drive towers it may be that damping carbon fiber towers or disk mounts could improve performance. It may not be economic, but it sure would look awesome.

The impact for Internet-scale infrastructures is more nuanced. Google, for one, doesn’t use high-density storage – maybe 120 drives per rack – so the performance gains may not be economic for them. But users of many drive servers with, say, 300+ drives in a rack would find it easier to justify.

Courteous comments welcome, of course.

StorageMojo’s best paper of FAST ’10 is Understanding Latent Sector Errors and How to Protect Against Them (pdf) by Bianca Schroeder, Sotirios Damouras, and Phillipa Gill, University of Toronto.

The paper builds on research and a dataset that StorageMojo reviewed 2 years ago in Latent sector errors in disk drives. That research analyzed the error logs of 50,000 NetApp arrays with 1.53 million enterprise and consumer drives disks.

Understanding
Understanding LSEs does a statistical deep dive on the disk LSE dataset and then evaluates scrubbing and intra-disk redundancy strategies against the field data.

Latent sector errors are important for 3 reasons:

• 1 LSE can cause a RAID reconstruction failure in a single parity RAID system (RAID 5).
• Ever-tinier disk storage geometries make LSEs more likely.
• The insidious failure mode: no detection until access is attempted.

Schroeder et. al. used a subset of the LSE dataset that included only drives that had LSEs. This covered 29,615 nearline (presumably SATA) drives and 17,513 enterprise drives that had been in the field at least 12 months.

LSE metrics
Some of the papers conclusions:

• For most drives almost all LSEs are a single error. Multiple contiguous logical block errors are less than 2.5% of all LSEs.
• If there is a 2nd error, most are within 100 sectors of the 1st error.
• Depending on the model, between 20% and 50% of errors are in the first 10% of the drive’s logical sector space. Some drives have a higher concentration of errors at the end of the drive as well.
• LSEs are highly concentrated in a few short time intervals, not randomly spread out over a drive’s life.
• It appears that events that are close in space are also close in time.

The rest of the paper
The paper also goes into 2 interesting topics – intra-disk redundancy and scrubbing strategies – that deserve posts of their own. For the latter the research found that changing the order in which sectors are scrubbed can improve mean time to error detection by 40% – with no increase in overhead or scrub frequency.

Conclusions
Key quote:

We observe that many of the statistical aspects of LSEs are well modeled by power-laws, including the length of error bursts (i.e. a series of contiguous sectors affected by LSEs), the number of good sectors that separate error bursts, and the number of LSEs observed per time. We find that these properties are poorly modeled by the most commonly used distributions, geometric and Poisson. Instead we observe that a Pareto distribution fits the data very well and report the parameters that provide the best fit. . . . We find no significant difference in the statistical properties of LSEs in nearline drives versus enterprise class drives.

[bolding added -ed. However, nearline drives are about 4x more likely get an error.]

The StorageMojo take
Disk-based storage arrays are facing a real challenge from flash and possibly PCM technology. Disks win the $/GB race, but piling double and triple parity on arrays increases costs and firmware complexity.

Understanding the nature of the enemy – in this case latent sector errors – helps array designers develop more reliable and cost-effective arrays. Yet one has to wonder if the RAID paradigm is reaching the end of the line.

Parallel and object-based systems from Isilon and Panasas, for example, are very fast at disk rebuilds because they can draw data from many disk drives in parallel – without the performance-killing overhead that RAID rebuilds impose.

But those are larger systems. Putting these techniques together may give us reliable and economical RAID 5 systems for the SMB market for another decade or more.

Courteous comments welcome, of course. I’ve done work for Isilon – who also advertises on StorageMojo – and Panasas. The official best paper of FAST ’10 was quFiles which I blogged about last week.

If you spot a typo please let me know. Thanks!

Late last year Sun engineer, DTrace co-inventor, flash architect and ZFS developer Adam Leventhal, analyzed RAID 6 as a viable data protection strategy. He lays it out in the Association of Computing Machinery’s Queue magazine, in the article Triple-Parity RAID and Beyond, which I draw from for much of this post.

The good news: Mr. Leventhal found that RAID 6 protection levels will be as good as RAID 5 was until 2019.

The bad news: Mr. Leventhal focussed on enterprise drives whose unrecoverable read error (URE) spec has improved faster than the more common SATA drives. SATA RAID 6 will stop being reliable sooner unless drive vendors get their game on. More good news: one of them already has.

The crux of the problem
SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 200,000,000 sectors, the disk will not be able to read a sector.

2 hundred million sectors is about 12 terabytes. When a drive fails in a 7 drive, 2 TB SATA disk RAID 5, you’ll have 6 remaining 2 TB drives. As the RAID controller is reconstructing the data it is very likely it will see an URE. At that point the RAID reconstruction stops.

Here’s the math:
(1 – 1 /(2.4 x 10^10)) ^ (2.3 x 10^10) = 0.3835

You have a 62% chance of data loss due to an uncorrectable read error on a 7 drive (2 TB each) RAID 5 with one failed disk, assuming a 10^14 read error rate and ~23 billion sectors in 12 TB. Feeling lucky?

When 4 TB drives ship later this year only 3 drives will equal 12 TB. If they don’t up the spec, this will be a mess.

RAID 6
RAID 6 creates enough parity data to handle 2 failures. You can lose a disk and have a URE and still reconstruct your data.

NetApp noted several years ago that you can have dual parity without increasing the percentage of disk devoted to parity. Doubling the size of RAID 5 stripe gives you dual disk protection with the same capacity.

Instead of a 7 drive RAID 5 stripe with 1 parity disk, build a 14 drive stripe with 2 parity disks: no more capacity for parity and protection against 2 failures. Of course, every rebuild will require twice as many I/Os since each disk in the stripe must be read. Larger stripes aren’t cost free.

Grit in the gears
The chance that a single sector rebuild will encounter 2 read errors is tiny, so what is the problem?

Mr. Leventhal says a confluence of factors are leading to a time when even dual parity will not suffice to protect enterprise data.

These include:

• Long rebuild times. As disk capacity grows, so do rebuild times. 7200 RPM full drive writes average about 115 MB/sec – they slow down as they fill up – which means about 2.5 hours per TB minimum to rebuild a failed drive. Most arrays can’t afford the overhead of a top speed rebuild, so rebuild times are usually 2-5x that.
• More latent errors. Enterprise arrays employ background disk-scrubbing to find and correct disk errors before they bite. But as disk capacities increase scrubbing takes longer. In a large array a disk might go for months between scrubs, meaning more errors on rebuild.
• Disk failure correlation. RAID proponents assumed that disk failures are independent events, but long experience has shown this is not the case: 1 drive failure means another is much more likely.

On the last point: in a corridor conversation at FAST ’10 I was told that at a large HPC installation they found that with drives from the same manufacturing lot that 1 drive failure made a 2nd 10x more likely – while a 2nd made a 3rd 100x more likely. Not clear how manufacturing or environmental issues – or interaction between the 2 – led to the result. YMMV.

Simplifying: bigger drives = longer rebuilds + more latent errors -> greater chance of RAID 6 failure.

Mr. Leventhal graphs the outcome:

Courtesy ACM Queue

The StorageMojo take
For enterprise users this conclusion is a Big Deal. While triple parity will solve the protection problem, there are significant trade-offs.

21 drive stripes? Week long rebuilds that mean arrays are always operating in a degraded rebuild mode? Wholesale move to 2.5″ drives to reduce drive and stripe capacities? Functional obsolescence of billions of dollars worth of current arrays?

What is scarier is that Mr. Leventhal assumes disk drive error rates of 1 in 10^16. That is true of the small, fast and costly enterprise drives, but most SATA drives are 2 orders of magnitude less: 1 in 10^14.

With one exception: Western Digital’s Caviar Green, model WD20EADS, is spec’d at 10^15, unlike Seagate’s 2 TB ST32000542AS or Hitachi’s Deskstar 7K2000 (pdf).

Before entering full panic mode though it would be good to see more detailed modeling of RAID 6 data loss probabilities. Perhaps a reader would like to take a whack at it.

Comments welcome, of course. I worked at Sun years ago and admire what they’ve been doing with ZFS, flash, DTrace and the great marketing job the ZFS team did without any “help” from Sun marketing. An earlier version of this post appeared on Storage Bits. Looking for a scientific calculator program? PCalc – Mac & Windows – is the best I’ve found.

WD has started shipping drives that drop the ancient 512 byte disk sector for a 4096 byte – 4k – sector, and the rest of industry isn’t far behind. For several decades disk sectors have been almost always been 512 bytes (NetApp tried 520 bytes – and irritated their customers no end). Why 4k and why now?

Why?
Rising bit density means smaller magnetic areas and more noise. The underlying or raw disk media error rate is approaching 1 error in every thousand bits on average – while tiny media defects can lose hundreds of bytes in a row. The larger sectors enable more powerful ECC to fix those gaps.

Why now?
A 512 byte sector can’t support enough ECC to correct for higher raw error rates. Thus bigger sectors with stronger ECC capable of detecting and correcting much larger errors – up to 400 bytes on a 4k sector.

The 4k sector enables disk manufacturers to keep cramming more bits on a disk. Without them the annual 40% capacity increases we’ve come to expect would stop.

Note: the longer ECC doesn’t change the drive level unrecoverable read error rate. It remains at 1 in every 10¹⁴ bytes.

4k sectors have been cooking for over a decade. The late adopters are the cloning software vendors. More on that in a moment.

Will 4k sectors use capacity faster?
If you write 500 bytes and the minimum sector is 4k, will that write take up the full 4k, wasting 3.5 KB? No.

The initial WD drives – and I assume other vendors as well – will operate in a 512 byte emulation mode. Eventually new disks will operate in native 4k mode, and then you might have a concern. But many operating systems already do 4k IO. And at a couple of cents per future GB, who cares?

Gotchas?
If you are in either of these 2 groups:

1. Windows XP users
2. Windows users who clone disks with software like Norton Ghost

there are a couple of gotchas if you want to use a 4k drive. Since most drives aren’t 4k and won’t be for another year or more, this may not affect you either. Vista and W7 users are cool except for cloning.

1) Windows XP does not automatically align writes on 4k boundaries, which hurts performance. WD has software – the Advanced Format Align Utility for their drives. I assume other vendors will too when they start shipping.

XP users need to run this utility once to use a 4k drive with a clean install, cloning software or a do-it-yourself USB drive. It isn’t needed for WD-branded 4k USB drives.

2) Windows clone software vendors have yet to implement 4k support. If you clone an XP, Vista or W7 drive you should run the align utility. The cloning vendors need to get on board Real Soon Now. Vendors are welcome to comment on their plans.

What about Macs?
No worries: Mac OS just works with 4k drives – including cloning.

Summary
There’s been a lot of heavy lifting behind the scenes to make this a smooth transition. With Vista, W7, Mac OS and Linux support well in hand most users won’t notice any change.

Some XP users will get bit by performance issues. The easiest solution for XP users: avoid 4k drives. Factory installed XP will be fine.

The StorageMojo take
My question: why not a better read-error spec? Today’s large SATA drives shouldn’t be used in RAID 5 arrays due to the high likelihood of a read error after a drive failure, which will abort the RAID rebuild. A better error spec would fix this.

Oh, RAID 6 sells more drives? Never mind.

Finally, the drive industry doesn’t know how to talk to consumers about technology. It took me an hour of digging to understand how this benefits consumers rather than vendors.

Comments welcome, of course. WD’s dynamic Heather Skinner arranged a briefing for me. No sectors, old or new, changed hands.

Geoff Barrall founded BlueArc at the high end of NAS performance. He then founded Data Robotics, maker of the Drobo low end arrays. A group of bloggers visited DR last month and a lucky few – not including me – took brand new Drobo2 units home.

The idea
For those who haven’t been following Drobo, the idea was to build a simple-as-possible-but-no-simpler storage array for data intensive civilians. Folks like photographers, videographers, musicians, scientists and designers who munge a lot of data.

Drobo users can put any size drive in the box and the capacity will be added automagically. The usable capacity for protected data is roughly the sum of the 3 smallest drives in the box.

The key point though is that civilians don’t have to know about volume sizes, drive capacities or configuring RAID. Stick a couple of drives in the box and Drobo tells you what you’ve got.

Need more, add another drive. Once the slots are filled, pull the smallest drive out and add a larger drive. Don’t get too frisky though: with large drives the data movement takes many hours.

The instruction manual is printed on the inside of the faceplate that covers the drives. Big green and red lights give drive status.

Product line
Data access and performance are 2 sides of the same coin. If the performance is too slow for the application, the data is essentially not available – at least until you can move it to something faster.

The gen1 Drobo was USB only and crippled by anemic performance. Fine for photographs, but any decent-sized video file would choke it.

The second Drobo – now the low end model – added FireWire 800 to the mix. You could archive video on it, but not edit it in situ. About a year ago they introduced a 8 drive version with GigE and single-server iSCSI support and a dual-drive failure protection.

Last month Drobo added 2 new models: the Drobo S with 5 drives and eSATA; and the Drobo Elite, an 8 drive unit with dual GigE and multi-server iSCSI support. The latter is spec’d at 255 virtual LUNs, but ~100 is more realistic.

Performance
Now fellow blogger Devang Panchigar over at StorageNerve has published performance test results (pdf) of the current low-end model.

The net/net: USB tops out at about 32 MB/sec; while FW800 manages 52 MB/sec. Neither is fast enough for HD video, but FW800 will handle standard def video just fine.

I’m planning to buy a 5 slot Drobo S later this month with the help of a gift discount coupon from DR. After I’ve played with it I’ll let you know what I think. One problem already: getting a decent Mac eSATA driver for a PCIe card.

The StorageMojo take
DR is moving up market. They plan to stay with a self-imposed $15k price ceiling. With 3 TB drives right around the corner, a raw 24 TB iSCSI SAN array could come in at $8k or less.

$300/TB for a capacity large enough for many SMBs is disruptive – especially when the easy-enough-for-mom management is factored in. If they go public next year I suspect there will be a bidding war for them in ’11.

At $400 for an empty 4 slot box they aren’t competing on price either. They are showing the industry what can be done with a premium price – compared to the Buffalos and Iomegas – array that offers much greater ease of use. Their growth rate proves that is a popular message.

The bigger issue for old-line vendors is that the SMB market is about to get a lot tougher – as if it wasn’t tough enough. The enterprise ROBO market is also in play.

DR is the one to beat in the prosumer storage market.

Courteous comments welcome, of course. DR was a sponsor of the tech blogger excursion that flew me to Silicon Valley. And just for the record, Drobo doesn’t use ZFS.

TDK recently demo’d an impressive technical achievement: a 10 layer optical disk with 320 GB capacity – using standard Blu-ray (BD) drive technology. Each layer has better than 90% light transmission and writing required no more than 20 mW of the 30 mW Blu-ray spec.

Too bad it will never be a commercial success. Optical is at the end of the line.

When do formats die?
When their combination of reliability, capacity, performance, density and cost aren’t competitive. Which is where optical is now – even 320 GB optical.

Some of you may remember punched paper tape – hot in the 60s and early 70s – and popular on 16 bit minicomputers back when 4k of RAM was respectable and 64k unaffordable. It was limited to a few dozen KB of capacity and unreliable in long-term use, so when 240KB 8” floppies arrived in 1973 paper tape was toast.

Floppies had to improve to compete with removable disk pack drives – like DEC’s RK05 family – with their 2 MB capacity and a screaming 150 KB/sec transfer rate, and floppies did by increasing capacity – what TDK demonstrated – and decreasing size, from 8” to 5.25” to 3.5”, and cost from over a thousand dollars for a drive to less than $20.

But floppies couldn’t keep up with the growing size of applications and data sets. The 100 MB Zip drive was insanely popular when introduced in 1994 – a woman offered me a $100 premium on the spot to buy mine at a Palo Alto sushi bar – but by 1999 the format was on the way out thanks to cheaper and more capacious CD-R drives.

Despite heroic efforts to increase removable magnetic disk capacities – culminating in 2001 with the 5.7 GB Orb drive – removable magnetic disk media is dead, killed by cheaper optical and more convenient flash media.

Removable: backup and transfer
Removable media has 2 major use cases: data backup and data transfer. Tape dominates removable media backup today with capacities rivaling the largest disks.

Thumb drives long ago replaced floppies for smaller file transfers – “sneakernet” – with external hard drives handling large capacities. With 1 TB 2.5” hard drives, even a writeable 50 GB Blu-ray (BD-R) can’t compete with a small hard drive in transfer speed or capacity.

TDK’s problem
Which gets us to the 10x Blu-ray problem: even if it were commercialized there would be no market. Why?

• Capacity. Successful optical media capacities have been competitive with current disks – CD-ROM in the early 90s; DVD-R in the early 2000s. Multi-layer Blu-ray will never be more than a small fraction of hard drive capacities.
• Performance. 24x Blu-ray transfer rates are half that of today’s disks. And as capacities increase, disks get faster. Not so with Blu-ray.
• Reliability. Early adopters report that BD burner disks often don’t play on many commercial players. That will get fixed someday, but multi-layer DB-R will have to solve it again.
• Density. Managing a single piece of media is much simpler than managing 6 or 10. External hard drive density makes them much more convenient.
• Cost. BD-playing DVD drives haven’t been popular on PCs, and BD burners are way more expensive, as is the media. A FireWire or USB 2 or 3 hard drive can be had for less than $100, has much faster access times, higher capacity and faster data transfer. With volume BD-R costs will come down – but where will the volume come from?

Multi-layer BD-R has advantages, especially if current BD players can be updated to use it. But there is no commercial justification for distributing content on 320 GB optical disks and there isn’t likely to be one.

Hollywood has a real chance to make 3D work this time, but 3D HD movies will fit fine on BD. Put a 3D “Band of Brothers” on a single disk? OK, but really, getting up every 50 minutes to change disks isn’t so hard, is it?

The Storage Bits take
New optical formats will get introduced – like 750 MB Zip drives and 5.7 GB Orb drives – but they’ll stumble around the fringes of consumer acceptance before a quiet death off stage. Many of the same forces that are killing BD – downloading, upconverting, cost – are closing in on optical media in general.

DVDs will be around for years – even as CDs still are – but the focus is shifting to online storage and local disks. The industry still hasn’t cracked the code on massive home disk storage, but that day is coming.

You’ll buy HD 3D content online, download it, store it in your digital library, and watch it when and where you want. If your house burns down your content suppliers will let you download again. Who needs the hassle to burn disks?

The one remaining piece is for hard drive vendors to get serious about building archive-quality hard disks. I love their technology, but they aren’t the most forward looking group.

Courteous comments welcome, of course. Anyone interested in buying a vintage USB Zip drive?