StorageMojo

The tension was real
The disk guys acknowledged that flash has killed the sub-1.8″ drive market. Will 1.8″ drives be next?

The flash guys acknowledged that they have a huge economic problem: heavy seasonality of demand. Consumer demand drives a Q4 peak in flash sales. But what to do with a $5 billion flash factory the rest of the year? Look for other markets, that’s what.

The flash guys announced a goal of taking 25-28% of disk revenue for themselves.

Flash drives aren’t living up to the hype
Which is no big surprise. The divergence is simply getting more visible (see my ZDnet articles Hybrid drives: not so fast, Flash drives: your mileage WILL vary, and Power, notebooks and solid state disk).

WD strikes back
The most outspoken disk defender is Rich Rutledge, SVP at Western Digital, the 2nd largest disk manufacturer. His argument, which is overlong for consumers, is that flash will *never* deliver on the hype, because it can’t. Rich’s point is that flash’s advantages are largely illusory:

• Power: disks are pretty efficient, and today’s notebooks have a lot of power-hogging systems (Wi-fi, Bluetooth, display, dual-core processors, GB of RAM and graphics co-processors) so the additional battery life that flash can deliver is less than 6%.
• Boot times: cold boot not all that much faster - 8-10% - and the fastest boot times in Vista come from Vista’s Sleep mode, which uses the battery to keep your data live in RAM, the fastest mass storage you own.
• Size & weight: important in handheld devices, but notebook size and weight are dominated by keyboard and screen requirements, not the extra grams of a disk drive. Flash in small ultra-light notebooks? Sure. In a 17″ behemoth? Much less likely.
• Performance: flash, is an unlimited dragster - fast in the quarter mile, but no good on curves. That is, the flash drive’s massive advantage in small random read speed is lost in the real world where small random writes and large sequential reads and writes drown out the flash drive’s one big performance advantage.

Rick had a larger point as well. The disk guys have an incredible story to tell and they aren’t telling it. I’ll be getting into some of that later this week.

Taking down the hype
About 20 years ago I took down a hype-fueled machine - Manufacturing Automation Protocol - that had large customer interest, seemingly unassailable technical advantages and major vendor support. And I had a blast doing it.

If the disk folks don’t tell their story they can’t expect anyone to do it for them.

The Storage Bits take
Rich makes some good points, but I’m not ready to write flash off. Let’s give the flash engineers a chance to work some magic. But I’m far from convinced that the flash vendors have the Mojo to pull off what they’ve promised. Enough hype, let’s see some results.

That said, I do believe that flash will own the low-end of the storage market such as handheld devices and ultra-light PCs where size, weight and power really are critical and capacity/performance much less so. 1.8″ drives are in the free-fire zone. They’ll hang in there, but only by driving capacity hard.

Hybrid drives are doomed. Relying on Microsoft to make a peripheral hardware product economically viable is weak. Plus spending a few bucks on DRAM which speeds everything up just makes more sense than spending that money on sometimes speeding up a disk.

Update: Barry Whyte, an IBM’er in the UK, sent me a link to his blog
where he discusses his test of the high-end STEC flash-based SSD behind an IBM controller. The money quote:

. . . the host was seeing close to 50,000 read ops and just shy of 19,000 write ops! Over 100MB/s and closer to 200MB/s with SVC’s sequential pre-fetch.

STEC was at DISKCON, and I tried to talk to one of their engineers, but it was not to be. Essentially they have a many parallel pipes to a lot of flash chips so they can do that many writes. I don’t share Barry’s confidence that STEC will be able to drive the price down as fast as NAND is going. That kind of engineering isn’t cheap and STEC will want their deserved 50%+ gross margins for their controller Mojo. End update.

Comments welcome, as always. Maybe someone out there understands the hybrid drive equation better than I do.

I was surprised at how many ZDnet readers reacted with disbelief to my recent Storage Bits series on data corruption (see How data gets lost, 50 ways to lose your data and How Microsoft puts your data at risk), claiming it had never happened to them.

Then I thought about it
What does data corruption look like to users? Does a window pop up with big red letters blaring “DATA CORRUPTION!!!” Nope, we get these “File not found” and other notices that could be - but who knows? - related to data corruption. Something goes badly wrong and you have to reinstall an application or the OS. But really, how prevalent is data corruption?

CERN does some research
That’s why I was delighted to see a new paper from CERN. Now, finally, some statistics are in, reported in a recent paper titled Data Integrity by Bernd Panzer-Steindel of the CERN IT group.

Petabytes of on-disk data analyzed
At CERN, the world’s largest particle physics lab, several researchers have analyzed the creation and propagation of silent data corruption. CERN’s huge collider - built beneath Switzerland and France - will generate 15 thousand terabytes of data next year.

The experiments at CERN - high energy “shots” that create many terabytes of data in a few seconds - then require months of careful statistical analysis to find traces of rare and short-lived particles. Errors in the data could invalidate the results, so CERN scientists and engineers did a systematic analysis to find silent data corruption events.

The program
The analysis looked at data corruption at 3 levels:

• Disk errors.The wrote a special 2 GB file to more than 3,000 nodes every 2 hours and read it back checking for errors for 5 weeks. They found 500 errors on 100 nodes.
• Single bit errors. 10% of disk errors.
• Sector (512 bytes) sized errors. 10% of disk errors.
• 64 KB regions. 80% of disk errors. This one turned out to be a bug in WD disk firmware interacting with 3Ware controller cards which CERN fixed by updating the firmware in 3,000 drives.
• RAID errors. They ran the verify command on 492 RAID systems each week for 4 weeks. The disks are spec’d at a Bit Error Rate of 10^14 read/written. The good news is that the observed BER was only about a 3rd of the spec’d rate. The bad news is that in reading/writing 2.4 petabytes of data there were some 300 errors.
• Memory errors. Good news: only 3 double-bit errors in 3 months on 1300 nodes. Bad news: according to the spec there shouldn’t have been any. Only double bit errors can’t be corrected.

All of these errors will corrupt user data. When they checked 8.7 TB of user data for corruption - 33,700 files - they found 22 corrupted files, or 1 in every 1500 files.

The bottom line
CERN found an overall byte error rate of 3 * 10^7, a rate considerably higher than numbers like 10^14 or 10^12 spec’d for components would suggest. This isn’t sinister.

It’s the BER of each link in the chain from CPU to disk and back again plus the fact that for some traffic, such as transferring a byte from the network to a disk, requires 6 memory r/w operations. That really pumps up the data volume and with it the likelihood of encountering an error.

The cost of accuracy
Accuracy isn’t free. The CERN paper concludes that taking measures to improve accuracy

. . . will lead to a doubling of the original required IO performance on the disk servers and . . . an increase of the available CPU capacity on the disk servers (50% ?!). This will of course have an influence on the costing and sizing of the CERN computing facility.

The Storage Bits take
My system has 1 TB of data on it, so if the CERN numbers hold true for me I have 3 corrupt files. Not a big deal for most people today. But if the industry doesn’t fix silent data corruption the problem will get worse. In “Rules of thumb in data engineering” the late Jim Gray posited that everything on disk today will be in main memory in 10 years.

If that empirical relationship holds, my PC in 2017 will have a 1 TB main memory and a 200 TB disk store. And about 500 corrupt files. At that point everyone will see data corruption and the vendors will have to do something.

So why not start fixing the problem now?

Comments welcome, of course.
Update: Peter Kelemen, one of the CERN researchers, kindly wrote in and pointed out that the it is the disks that are rated at 10^14, not the RAID card. There are no specs for the RAID cards. I’ve corrected it above.

I’m attending the IDEMA show this week at the Santa Clara Convention Center this Wednesday and Thursday.

Flash news
Flash technology and its integration into disks and application devices are a big topic this year. I’ve noted here and on Storage Bits that flash isn’t living up to all the hype. I’m hoping I’ll learn more about why and what the smart engineers are doing to fix it.

If you are in the neighborhood
I always enjoy meeting readers. Come on by and say hello.

Up-to-the-minute posting
That would be nice, eh? I’ll see what I can do.

Moderation may be slower than usual, especially Tuesday, as I make the epic trek from backwoods Arizona to the glittering shores of south San Francisco Bay, and on my heavily booked Friday. But moderate I shall!

Comments welcome. Include contact info if you’ll be at the conference or just look for me. Cheers.

Not so far
Flash drives have been rocketing up the hype cycle for the last year, helped along by credulous gadget sites and misleading performance numbers. But the free ride is coming to an end.

Over on Storage Bits (see Flash gets a fight from disks and Hybrid drives: not so fast) I’ve been delving into the differences between flash and disks as well as the flash hype and what flash delivers.

The flash guys deserve some slack
Flash technology is young and rapidly improving with impressive Moore’s Law cost gains. And they’ve been working on the translation layer that mediates between the the weird world of flash and disk drivers. The engineers are figuring out the wrinkles and I have no doubt they will continue to improve.

But we also have to talk about where we are today.

20/20 vision: 20x the price and 20% better performance
Over on AnandTech they posted review last week that tested 3 drives on a 4 GB laptop running Vista Home Premium. The drives were:

• A 32 GB Mtron SSD MSD-SATA6025
• A Seagate Momentus 7200.2 160GB
• A Samsung MH80 FlashON 160GB

They tested maximum data rates - the Mtron smoking at 90 MB/s vs 49 and 38 for Seagate and Samsung - and Vista standby and hibernate modes. You’ll save a few seconds with the flash drives assuming Vista memory leaks don’t make the sleep modes as unreliable as they’ve been on XP and W2k.

But the real surprise were the pathetic application numbers. Here’s a chart I derived from AnandTech’s numbers:

I’d have thought that with double the bandwidth and a fraction of the access time that the flash drive would be a lot better than a 7200 RPM drive. Your average user couldn’t tell the difference between flash and disk based on these numbers.

Update: I’ve added the black line at 0% to indicate Seagate’s performance. The performance percentages are normalized against the Seagate drive. A positive percent means faster; a negative percent means slower. End update.

The StorageMojo take
One set of benchmarks on one product don’t decide the issue, especially this early in the game. But it gives one pause when a high-performance flash drive can’t do any better.

Maybe the translation layer isn’t the problem. Perhaps our OS’s do too many small random writes that hose flash performance. Whatever the case it is too early to assume that flash drives will even take a majority of the notebook market without better performance given their price and capacity limitations.

Comments welcome, of course. Maybe you can explain it.

Tired of your slow notebook burner?
Want to rip a lot of CDs - as I have in the last week?
Or maybe you’ve decided that, really and truly, you are going to get serious about backing up important data.

Adding an external burner - especially for notebooks - is a good option. And if you build one it is economical too. You can build a burner for $50-60 shopping online. Expect to pay a lot more at the local mall for a drive that may not be as fast or that supports FireWire in addition to USB.

Show and tell
This video is aimed at the lady across the street who - despite owning 2 computers, several cameras and a wide format inkjet to support her photography business - wasn’t too clear on the difference between disk “memory” and system “memory”.

This is for small business folks.

If you’ve installed a PCI card you’ll find this pretty boring. But a lot of folks haven’t and this video is aimed at them.

Caveat emptor
The video assumes an ATA/IDE drive, not a SATA drive. And case to match, natch. Most cases are IDE or ATA, but it appears that most of the new HD drives are SATA. They don’t mix and match.

If you want to play movies this probably won’t work. I’m not sure why, but decoding movies from DVD is more than simply being able to read or write the disk.

Without further adieu
Here’s the video:

Update: where does the audio plug go? It goes in about half an inch to the left of the big, fat ATA connector. Directly to the left of the ATA connector on my Samsung drive are a group of 6 pins with a plastic connector or jumper connecting two of the pins. Do not remove the jumper or attempt to plug the flat audio connector into the other pins. Unlike the other connectors the audio connector is not keyed. You can see the narrow horizontal slot - on the Samsung - it goes in.

I’ve never used the audio before and the questions got me wondering why anyone would. These are computer optical drives: if you don’t hook them up to a computer you have no controls. If they are hooked up to a computer the FireWire or USB cable carries the audio signals, NOT the case’s audio connector that goes to a headphone jack.

To check it out I just opened the case and tried to get an audio signal out of the case through the headphone jack. Couldn’t do it. So unless someone has a good reason for hooking up the audio cable to the headphone jack on the case I’d say it isn’t worth the bother.

Comments welcome, as always. Pretty slick opening credits, eh?

And I mention LightScribe, which is actually a pretty handy way to label a few DVDs and CDs without too much bother.

Ever since the HAL 9000
was decommissioned by pulling out its clear plastic storage modules in Kubrick’s 2001: A Space Odyssey every storage geek has been jonesing for 3D storage. Holographic, multi-layer, whatever. I want capacity and I want it now!

“I’m sorry Dave, I’m afraid I can’t do that.”
Well, we’ll have to wait a little longer. But if they can commercialize it, the Israeli company Mempile has taken a baby step in the right direction:

Mempile’s patented non-linear two-photon technology allows for 3D recording of transparent virtual layers on the entire volume of the disc. Mempile’s recent demonstration proved that more than 100 layers could be recorded and read showing storage capabilities of slightly less than 300GB over a thickness of 0.6 mm of active material. By increasing this active material to the thickness of a DVD, 1.2 mm, Mempile will be able to demonstrate the recording and reading of at least 500GB of data. Future optimization will allow the recording of 200 layers and of up to 5GB of data per layer.

Mempile has hooked up with a French chemical company, Arkema, and a Japanese media manufacturer, Memory Tech, and they are

. . . focusing on the development of an inexpensive volumetric optical disc and a high-density disc drive able to be made available at consumer prices. With true WORM capabilities, bit-by-bit recording and addressing, and longevity greater than 50 years, Mempile’s TeraDisc technology will empower both consumers and enterprise sectors.

“I think you know what the problem is just as well as I do.”
It sounds too good to be true, so it probably is.

• This is little more than a lab demo. I’d guess they are at least 5 years away from a product.
• The 200 GB Blu-ray disk will be getting ready for its next capacity boost by then.
• 50 year life? Disks, maybe. Players, no.

“Dave, my mind is going. I can feel it. I can feel it. My mind is going. There is no question about it.”
Very few of the promised optical technologies have achieved success. It was just 25 years ago that the first commercial CD rolled off the production line - The Visitors by the Swedish pop group ABBA. CDs are still selling but lots of other optical technologies either never made it in the market or their time came and went.

• 12″ laser disks enjoyed some success until the cheaper and more convenient DVD killed it off.
• Heavily-backed optical startup Dataplay went bankrupt a few years ago - collateral damage on the iPod’s march to global music domination. Note that iPods have yet to use optical and probably never will.
• The latest: Universal Media Disks - UMD - for the Playstation. Instead folks are using Handbrake and ripping their DVDs for playback on iPods.

“Look Dave, I can see you’re really upset about this. I honestly think you ought to sit down calmly, take a stress pill, and think things over.”
Good advice, HAL. And after I disconnect you maybe I will.

The StorageMojo take
Most ventures don’t achieve the dreams of their founders. But that doesn’t mean they weren’t worth doing. I applaud Mempile for coming up with a new technology that, despite the odds, offers something better than what Blu-ray is planning in three years.

Optical storage, in some form, has a permanent place in the world of storage. As long as people keep pushing the envelope we’ll be figuring out how to use it.

Comments welcome, as always. Can you believe that CD’s are 25 years old?

Over on Storage Bits I’ve ignited quite a bit of controversy with the post Why RAID 5 stops working in 2009.

My point in that post is that as SATA disk drive capacity continues to increase, and the unrecoverable read error (URE) rate remains constant, the time will come - 2009? - when every RAID 5 disk failure will be likely to encounter a URE during rebuild.

The arithmetic goes like this. Take a 7 drive RAID 5 stripe. Each drive is 2 TB (in a couple of years). One drive fails, leaving 12 TB of capacity to read to recreate the lost data. With a SATA URE of 10^14, which is about 12 TB - OK, a little more - you are highly likely to encounter a URE. At that point an honest RAID controller will inform you that it can’t complete the rebuild.

I *think* different controllers have different responses to this scenario, but I will bow to the more knowledgeable among my readers who might care to elucidate.

The real question is URE
SATA drives that I’ve looked at have a URE of 10^14 while enterprise drives are spec’d at 10^15. My question is: why aren’t the drives spec’d at 10^16 or more?

Essentially, drive reads are a statistical process, as the unfortunate hyping of PRML (partial response, maximum likelihood) a few years ago made all too clear. (It’s highly probable that the data we read is the data you wrote, and we have the statistics to prove it!)

If the drive vendors devoted more space to ECC it seems that they could build drives with much lower URE rates. That is what they already do with enterprise drives.

Obligatory conspiracy theory
Maybe the drive vendors don’t do so because they know that with the advent of RAID 6 they’ll be selling that many more drives. And the array vendors will be as well.

As I noted in the Storage Bits post, the net effect of drive failure + URE is to render RAID 6 the new RAID 5. That doesn’t address the problem of dual drive failures, which we already know are more common than standard theory expects. So you’ll be paying RAID 6 prices for what is, in effect, RAID 5 protection. W00t!

I don’t think there is any conspiracy. I feel for disk folks because they are in such a competitive, cut-throat industry with 6-12 month product cycles and brutal pricing. It is hard for them to do much more than react as fast as they can.

The StorageMojo take
I’ve noted before that disk folks seem to have a hard time with strategy, a thought that first occurred to me when Seagate bought Xiotech: “let’s get into a business we know nothing about AND compete with our best customers! It’s a twofer!” It would have been much smarter to buy EuroLogic or Xyratex and move up the value chain with something of value for existing customers.

Endlessly pushing capacity as the only metric only guarantees an ever faster treadmill. Vendors should look at how they can subtly alter volume products, as WD has done with the 10k Raptors, to create new niches. Lots of people would like to have more reliable disk drives, so reducing capacity in favor of lower URE rates to create RAID 5-friendly SATA drives could be lucrative.

I believe consumers are educable if the value can be simply and vividly articulated. Drive vendors need to take a fresh look at their marketing to break out of the high-volume, low-margin box they are trapped in now.

Comments welcome, as always.