StorageMojo

Fresh off the HD-DVD fiasco, Toshiba execs are stepping up to pursue another expensive flop: notebook SSDs. Memo to Toshiba: people won’t pay huge SSD premiums for nothing. And almost nothing is what flash SSDs provide today - and for the foreseeable future.

Please sir, may I have another!
Given the multi-billion dollar cost of semiconductor fabs, getting the notebook SSD market wrong would make Toshiba’s $250 million HD-DVD loss look cheap. The president of Toshiba semi, Shozo Saito, recently opined that flash drives will be in 25% of notebooks by beginning 2011.

He is so-o-o wrong.

Hand me the back of the envelope, please
Guessing 200M notebook sales in 2011, 50 million flash drives of, say 250 GB, for total sales of 12.5 million TB of flash. Assuming a cost reduction curve of 50% annually from today’s spot market MLC $2500/TB to ~$320/TB in 2011 . . . hmm-m . . . $4 billion in chip sales.

Give or take. Yummy!

If Toshiba projects winning 20% of the market, $800 million in sales would justify over $1 billion in flash factory capacity. And if the market doesn’t appear, a billion dollar write off.

Same power, same performance and way more costly - I’m sold!
If flash drives delivered what proponents claim there would be no problem. But they don’t and they won’t.

Power: no SSD notebook has gained more than 10 minutes battery life over disks. Since flash is already power-efficient that won’t change. Disks have multiple opportunities to improve power use - and with over a $1 billion a year in R&D behind them - they will.

Performance: tested application performance hardly changes either - even with a $3,800 flash drive. Notebook I/O doesn’t favor flash drives - and the engineering contortions needed to fix flash aren’t cheap.

The one big win for flash performance: boot and app load times. It makes the system feel a lot snappier - if you often reboot. Sleep mode makes that much less important.

Reliability/durability: flash vendors tout 2 million hour MTBFs and superior shock & vibe specs. Yet Dell reports that their SSD infant failure rates are about the same as disks. And the return rates are higher.

So where, exactly, is the flash advantage? Plus, it is only conjecture that flash drives will prove to be more reliable in actual notebook use. Only time will tell.

And what about the 4-bit MLC that Toshiba is counting on to drive costs down at 40-50% per year? This will less durable than current SLC. No hard numbers from the vendors - depends on how good their signal processing algorithms are - but it could easily be 5,000 writes - down from 10,000 today.

How do you explain that to consumers?

Data integrity: the unasked question Of all the questions about flash drives, this is the biggest. I have yet to see an SSD read error spec.

Flash has read errors - that’s why vendors implement error detection.

But flash has a problem disks don’t: flash drives move your data around a lot more often than disks do. Every time a flash drive writes a page, it has to erase the entire block that page is in.

So what happens to the data in the block? It gets read - almost always correctly - and rewritten along with the new page. The new location must be tracked by the drive.

The map that keeps track of where your data is rapidly gets very complex - and itself is regularly read and rewritten. How well protected is this critical data structure? If it isn’t bulletproof you can kiss your data good bye.

If FTL’s are like every other storage product, catastrophic failure modes are hiding in the statistical weeds. Enterprise IT is rightly suspicious of storage that “auto-magically” moves data around. Consumers have no idea. SSD vendors better have their act together or the class action suits could be as big a problem as the empty fabs.

The StorageMojo take
The further I wade into flash issues, the worse it gets. My sense is that the flash industry close to creating a multi-billion dollar fiasco. Why?

• Over-promising on performance, reliability, battery life and data integrity. Take a systems level perspective, folks. Consumers do.
• Over-broad positioning of flash drives as a general replacement for notebook hard drives - when pricing clearly says they aren’t.
• Relying on system OEMs like Dell to market SSDs to consumers is a freeway to failure. They don’t have the bandwidth. The flash vendors need to market flash SSDs directly to consumers. Not sell them - market them.

The flash guys are caught in a vise: big expensive fabs that need to run all year; and seasonal demand that whipsaws their pricing all year.

Notebook flash drives can help even out demand - but only if consumers accept them for the right reasons. Otherwise Toshiba’s new fabs will build chips for a non-existent market.

Update: Flash has a place in one notebook niche: below the $40-$50 minimum cost of a disk. As we’re already seeing with the Asus Eee, replacing $50 of disk with $10 of flash makes a big price difference. But those units won’t solve the seasonality problem and may even make it worse. End update.

Comments welcome, of course.

How flash is really going to affect the storage industry is becoming clear. The short take: not as big a deal as flash vendors hoped. The longer take: There won’t be much of a mid-range flash market; instead we’ll see either costly fast flash or cheap slow flash.

There are lots of theories about how flash will alter the mass storage landscape. This is mine.

The flash write problem
The fundamental flash problem is the slow writes. There are 3 elements to the slow write problem.

• Flash has to be erased before it can be written. Every write operation is really 2 write operations.
• The writes are large. Typical block sizes are 128KB to 256KB. Writing a single page requires writing - after erasing it first - the entire block.
• The write bandwidth to a single block is less than a slow disk. High bandwidth writes requires parallel paths to multiple blocks.

These problems can all be engineered around.

• Garbage collection-like algorithms can be extended to enable a supply of erased blocks
• RAM backed by a small battery or capacitor can buffer writes for later re-writing to flash
• Controller chips can be built in high volume with multiple data paths

But at what cost? The first two require well-engineered software and some sort of CPU to run it. Since it is software it will have bugs. Can it be any more reliable than current drive firmware?

The dilemma
For enterprise use, flash-based SSDs need to be rock-solid, which implies a lot of careful and costly engineering. For consumer use, they need to be very high volume, which means low-cost.

It is a similar problem to RAID controllers: very low-end RAID controllers aren’t reliable enough for enterprise use. They also aren’t cheap enough - or easy enough - for consumers to buy in volume. RAID controllers have engineering problems similar to flash translation layers.

Making flash drives look like disks makes them easy to integrate, but if you really need performance it also makes them costly - like the $10k for the flash drive EMC is using in the Sym.

Flash in the disk controller?
As I’m writing this a NetApp exec says that flash will be disruptive because by placing flash in a disk controller they will reduce the need for the costly and highly profitable fibre channel disks. That could be correct. It sounds smarter than sticking flash on a disk.

The StorageMojo take
Despite the miracles of cost-reduction and integration the industry regularly performs, some things, like power provisioning, don’t get cheaper. High-quality software engineering doesn’t either. That is what high-performance flash drives require.

The high-performance consumer flash drive seems to be a mirage. I’d like to be proven wrong, but today’s notebook SSDs don’t offer superior application performance and cost 10x as much. Hardly a recipe for success.

Update: Intel is planning to offer “high-performance” flash drives with partner Micron. I saw an impressive demo - is there any other kind? - at the Storage Visions conference. But with the early marketing missteps of Samsung, it looks like the consumer flash drive may fall off the hype cycle into a deep ditch. Flash drive marketers: now is the time for precision marketing if you ever hope to establish a mass market. Consumers remember unkept promises. Until you are cheaper. End update.

Comments welcome, as always. Also check out BPLRU: A Buffer Management Scheme for Improving Random Writes in Flash Storage by two Samsung researchers, Hyojun Kim and Seongjun Ahn for a nice intro to flash issues.

I ran into Woody Hutsell, EVP at Texas Memory Systems, last week. He graciously agreed to a talk on camera about their experience with flash and DRAM-based solid state storage.

TMS sells both: a DRAM-based SSD with multiple FC and Infiniband ports; and a 2 TB flash box with 128 GB of DRAM cache. Woody offered some interesting insights. For example, workloads with a large number of writes - even if they are a small percentage of the total workload - may not be suitable for flash-based storage.

Here’s the video:

Blame me for the shaky camera work.

Disclosure: I taped and edited this gratis.

Comments welcome, as always. BTW, Google now accepts files up to 1 GB. Seagate and WD should be happy.

Flash may be getting all the attention, but the boffins are working hard to ensure we have options to flash. We need those options because flash has some serious limitations, like random write performance and density, that we may not be able to overcome.

On the other hand it is easy to underestimate the power of sustained investment in R&D. Disk drives have successfully fended off numerous would-be usurpers thanks to their incredible areal density growth.

Now flash is facing a challenger
Intel and STMicroelectronics’ new phase-change memory threatens flash, but not any time soon. Unveiled at ISSCC, the thermal phase-change device can store 2 bits per cell, like MLC flash.

I asked Jim Handy, of Objective Analysis, a semi-conductor market research firm, for his take on the Intel-STM announcement. He responded:

The big question is “When does Phase Change stand a chance?” I doubt that it could be competitive today because a wafer with a new material is bound to be much more costly than a pure silicon wafer until volumes get up, and the volume won’t get up until the price is competitive with flash. The number of chips per wafer is about the same for PCM as for NOR flash, so there’s no cost advantage from die size.

Once flash reaches its scaling limit brick wall (which was expected to be 2006 at 65nm, then at 25nm in 2012, now looking like 10nm around 2016) then PCM will zip right past it. Trouble is - that brick wall has legs and keeps zipping ahead of us.

Until then PCM should have trouble competing on cost, and cost is everything in the semiconductor memory markets.

One advantage PCM has is that it has a fast write, so in many cases a PCM chip can replace a flash and a RAM. This means that the cost target is something higher than simply matching a NOR price.

The significance of an MLC PCM is that it puts PCM on the same footing as MLC NOR. Had that not happened, then there would have been a longer delay before PCM replaced NOR, since a PCM die size would have been twice as large as an MLC NOR of the same density on the same process.

As for PCM replacing NAND, wellllll…… that’ll take longer since NAND’s about 1/3 the cost of NOR. The same brick wall impacts both technologies, though, so it will happen!

Thanks, Jim. I hadn’t realized that PCM could replace a NOR and a RAM chip.

The StorageMojo take
Jim’s take is better informed than mine. My take away is that the growth of PCM will depend on how broad a market niche it can build for itself over time. That won’t be easy.

Comments welcome, of course.

Update on mobile flash performance
Mikko Pitkanen over at the mobile development blog Delay ToleraNt posted some more tests on Nokia N800’s flash performance. He’s a doctoral candidate at the Helsinki Institute of Physics at CERN in Switzerland with a strong interest in storage.

The money quotes:

. . . the first observation is that we achieve write performance close to 1Mbit/s for small (less than 1 MB) files.

. . . the read performance is much better than for writing and is certainly enough to play movies. The write performance instead, is poor and would not allow the user to receive large files with the full bandwidth achievable by the device’s WLAN.

Mikko’s got a new Nokia N810 that he’s loving, so that will be it for the N800 data. Good data point. Mikko, if and when you get some N810 performance data please send it along. Thanks!

More flash high performance
Intel and Micron announced a very fast flash chip - 200 MB/s read and 100 MB/s write - but the press release included this big caveat:

“Micron looks forward to unlocking the possibilities with high speed NAND,” said Frankie Roohparvar, Micron vice president of NAND development. “We are working with an ecosystem of key enablers and partners to build and optimize corresponding system technologies that take advantage of its improved performance capabilities.

Translating from marketing speak: “nobody has the technology, like the translation layer, to take advantage of this chip.”

The StorageMojo take
Realizing flash’s potential will be a multi-year, multi-company effort. No one has a clear idea of what the ultimate limits will be. In the meantime the disk folks will be working to limit the damage by raising reliability, density and shock resistance. Both technologies have a place. The fight is about boundaries. And all of us consumers benefit.

Comments welcome, as always.

About time
I’m in Silicon Valley for a few days. So I’ll keep this brief.

EMC is pulling out the stops. First Hulk/Maui clusters and now putting flash SSDs in the Symm. They are positioning it as technology leadership, which it isn’t, but it is marketing leadership. I’m impressed.

SSDs have been around for decades. Symms have been around for over 15 years, so why now?

I suspect the rising chorus of customers complaining about 30% capacity utilization rates coupled with Wall Street’s economic woes - I wouldn’t want to be EMC’s Citibank account manager - helped them make the decision. Plus the rise of cluster block storage - XIV the latest case in point - means that if you want to own the high-performance array crown it is time to stake out the territory.

Plus the margins are great!
I haven’t seen any pricing yet, but knowing EMCs general strategy I suspect they are charging their usual 6x markup over cost for the SSDs. Despite that it should be an easy business case for a CFO to approve.

But if you are going to spend big bucks on an SSD, is putting it inside a single storage array the right way to go? The wide-awake folks at Texas Memory Systems think not. They provided me with this table comparing their SSD to the STEC ZeusIOPS drive EMC is using.

Make an entire SAN go faster instead of a single array? Sounds good to me.

The StorageMojo take
Will SSDs finally get some data center love? EMC’s endorsement of SSDs should provide an opening for the long-suffering SSD companies to get more attention from the enterprise. If it’s good enough for EMC . . . .

Comments welcome, as always. Moderation may be slightly more intermittent than usual, but moderate I shall. When I’m not enjoying the convertible I rented.

I wrote about my testing of notebook disk drive power usage on ZDnet yesterday (see How much does a flash disk increase battery life?). I pulled the 160 GB WD Scorpio out of my MacBook and ran it on wall power through a Kill-a-watt meter to better understand power usage. I learned - or relearned - a few things.

What surprised me most was the fact that as I measured power usage I saw that I/O, CPU and network usage were all intertwined. I’m surprised I was surprised since they are systems and the pieces work together.

I/O and CPU
I ran a defrag program that exercised the disk while driving CPU usage on a Core Duo to 90%. Since the CPU uses almost 3x the power of the disk it is the CPU and not the disk that is the power hog.

In fact, the biggest power hog is the base system: 13 watts sitting there doing nothing, LCD, Wi-Fi and Bluetooth all turned off, with no CPU load.

Hey, Taiwan, want to build long-battery life notebooks: figure out how to turn more pieces of the system off when not in use.

28 watts max
With the LCD turned on full, Wi-Fi and Bluetooth on, and the CPU at 90% the diskless notebook pulled 28 watts.

Turning on Wi-Fi was good for 2 watts, almost as much as a busy disk, partly due to the CPU load of running the TCP/IP stack.

Optical drives are power hogs too. Maybe not the drive itself, but the associated graphics or CPU processing.

Here are the numbers

The StorageMojo take
The significance of the intertwined nature of I/O, CPU and network usage is this. Flash drives sip power, but in a busy system it is all the other subsystems that chew up the battery.

The power saving advantages of a flash drive are best in a lightly loaded system with a long battery life, i.e. your cell phone, PDA or ultra-light notebook. In a 2-3 hour battery life 15-17 inch notebook the 2-3 watts a disk uses is almost noise level.

I see some evidence that the flash drive makers are adjusting their marketing to these facts. That is all to the good. Given the price/capacity differential you want flash disk customers buying for the right reasons.

Comments welcome, of course.

I came across this simple - too simple? - flash performance benchmark on an anonymous blog named DelayToleraNt.

The benchmark is

a Java program that creates random data (in chunks of kilobyte) and writes the data to a file. For comparison I ran the same program with the cheapest Intel based MacBook available using Java 1.4. I used same source level and same Java binary that I used on N800 running Java CDC.

So the comparison does not measure only storage performance, but also Java. Anyways, storage performance counts often only when some application is using the storage, and this application might as well be written on Java. In the tests, each size of file was created consecutively 10 times and average was calculated. For file size of 0 kB we measured only time to create an empty.

The flash didn’t do so well:

The author states he wants to do more tests combining download speeds with file creation times as well as testing an iPhone. That will be interesting.

The StorageMojo take
One benchmark is not enough to generalize from, so I won’t. If this is a consistent issue I have to wonder at the impact on mobile device user experience.

Comments welcome. Can someone try to replicate the results?

Addition: Another interesting performance topic is explored in A look at MySQL on ZFS by John David Duncan, a consulting engineer with MySQL Inc., that compares the performance and management of MySQL on UFS and ZFS.

The money quote:

ZFS introduces remarkable ease and flexibility of administration, without any real cost in performance. At its worst, in these tests, ZFS performed almost as well as UFS with Direct I/O. With InnoDB, the ZFS performance curve suggests a new strategy of “set the buffer pool size low, and let ZFS handle the data buffering.” I did not test Falcon, since it was not yet in Beta when I ran the benchmarks, but a similar strategy for Falcon on ZFS might be to concentrate on the row cache but minimize the page cache. And although double-buffering problems are clearly visible in this ZFS performance curve, even with those problems at their worst, ZFS still outperformed UFS. The real reason for the good performance on this benchmark is not clear — indeed, every workload will be different — but the ZFS I/O scheduler, the Sun engineers paying attention to database performance, and the ZFS bug fixes contributed in recent (late 2007) releases of Open Solaris seem to be adding up to something good.

Several readers have asked
Boy, do the Fusion io guys give a good demo! Now if only they can ship a product.

I like startups.
People putting their heart and soul into an idea that rarely turns out as well as they hoped. As is said of 2nd marriages, the triumph of hope over experience.

I expect near-delusional optimism and careful misdirection. All part of the fun.

Fusion io
Yet the the Fusion io media blitz leaves me skeptical. The info is sparse and perhaps inconsistent. And the claims seem too good to be true - which usually means they are. Here are some concerns.

• The economics are dicey. $30/GB means a $1200 40 GB drive - assuming the $/GB number comes from the smallest configuration and not the more likely 640 GB version. Some gamers and blade buyers will shell out that kind of money, but why not just max out your RAM at $35/GB?
• No word about their architecture - evidently some kind of parallel channel like the STEC Zeus drive - on their site. That just seems odd.
• Finally, their numbers don’t pass the smell test. 600 MB/s of 4KB writes equals 150,000 IOPS. The STEC Zeus drive, whose experience in flash I find credible, claim 50,000 IOPS for their FC drive. What does Fusion io know that STEC doesn’t?

The StorageMojo take
I’d love to be proven wrong, but I’ll be waiting for independent confirmation of the product’s performance and price. Besides the price, the product has all the disadvantages of direct attached storage. Gamer heaven? Yes. Economic server enhancement? I’ll wait and see.

Comments welcome, of course. Has anyone played with one of these puppies?

The tension was real
The disk guys acknowledged that flash has killed the sub-1.8″ drive market. Will 1.8″ drives be next?

The flash guys acknowledged that they have a huge economic problem: heavy seasonality of demand. Consumer demand drives a Q4 peak in flash sales. But what to do with a $5 billion flash factory the rest of the year? Look for other markets, that’s what.

The flash guys announced a goal of taking 25-28% of disk revenue for themselves.

Flash drives aren’t living up to the hype
Which is no big surprise. The divergence is simply getting more visible (see my ZDnet articles Hybrid drives: not so fast, Flash drives: your mileage WILL vary, and Power, notebooks and solid state disk).

WD strikes back
The most outspoken disk defender is Rich Rutledge, SVP at Western Digital, the 2nd largest disk manufacturer. His argument, which is overlong for consumers, is that flash will *never* deliver on the hype, because it can’t. Rich’s point is that flash’s advantages are largely illusory:

• Power: disks are pretty efficient, and today’s notebooks have a lot of power-hogging systems (Wi-fi, Bluetooth, display, dual-core processors, GB of RAM and graphics co-processors) so the additional battery life that flash can deliver is less than 6%.
• Boot times: cold boot not all that much faster - 8-10% - and the fastest boot times in Vista come from Vista’s Sleep mode, which uses the battery to keep your data live in RAM, the fastest mass storage you own.
• Size & weight: important in handheld devices, but notebook size and weight are dominated by keyboard and screen requirements, not the extra grams of a disk drive. Flash in small ultra-light notebooks? Sure. In a 17″ behemoth? Much less likely.
• Performance: flash, is an unlimited dragster - fast in the quarter mile, but no good on curves. That is, the flash drive’s massive advantage in small random read speed is lost in the real world where small random writes and large sequential reads and writes drown out the flash drive’s one big performance advantage.

Rick had a larger point as well. The disk guys have an incredible story to tell and they aren’t telling it. I’ll be getting into some of that later this week.

Taking down the hype
About 20 years ago I took down a hype-fueled machine - Manufacturing Automation Protocol - that had large customer interest, seemingly unassailable technical advantages and major vendor support. And I had a blast doing it.

If the disk folks don’t tell their story they can’t expect anyone to do it for them.

The Storage Bits take
Rich makes some good points, but I’m not ready to write flash off. Let’s give the flash engineers a chance to work some magic. But I’m far from convinced that the flash vendors have the Mojo to pull off what they’ve promised. Enough hype, let’s see some results.

That said, I do believe that flash will own the low-end of the storage market such as handheld devices and ultra-light PCs where size, weight and power really are critical and capacity/performance much less so. 1.8″ drives are in the free-fire zone. They’ll hang in there, but only by driving capacity hard.

Hybrid drives are doomed. Relying on Microsoft to make a peripheral hardware product economically viable is weak. Plus spending a few bucks on DRAM which speeds everything up just makes more sense than spending that money on sometimes speeding up a disk.

Update: Barry Whyte, an IBM’er in the UK, sent me a link to his blog
where he discusses his test of the high-end STEC flash-based SSD behind an IBM controller. The money quote:

. . . the host was seeing close to 50,000 read ops and just shy of 19,000 write ops! Over 100MB/s and closer to 200MB/s with SVC’s sequential pre-fetch.

STEC was at DISKCON, and I tried to talk to one of their engineers, but it was not to be. Essentially they have a many parallel pipes to a lot of flash chips so they can do that many writes. I don’t share Barry’s confidence that STEC will be able to drive the price down as fast as NAND is going. That kind of engineering isn’t cheap and STEC will want their deserved 50%+ gross margins for their controller Mojo. End update.

Comments welcome, as always. Maybe someone out there understands the hybrid drive equation better than I do.

I’m attending the IDEMA show this week at the Santa Clara Convention Center this Wednesday and Thursday.

Flash news
Flash technology and its integration into disks and application devices are a big topic this year. I’ve noted here and on Storage Bits that flash isn’t living up to all the hype. I’m hoping I’ll learn more about why and what the smart engineers are doing to fix it.

If you are in the neighborhood
I always enjoy meeting readers. Come on by and say hello.

Up-to-the-minute posting
That would be nice, eh? I’ll see what I can do.

Moderation may be slower than usual, especially Tuesday, as I make the epic trek from backwoods Arizona to the glittering shores of south San Francisco Bay, and on my heavily booked Friday. But moderate I shall!

Comments welcome. Include contact info if you’ll be at the conference or just look for me. Cheers.

Not so far
Flash drives have been rocketing up the hype cycle for the last year, helped along by credulous gadget sites and misleading performance numbers. But the free ride is coming to an end.

Over on Storage Bits (see Flash gets a fight from disks and Hybrid drives: not so fast) I’ve been delving into the differences between flash and disks as well as the flash hype and what flash delivers.

The flash guys deserve some slack
Flash technology is young and rapidly improving with impressive Moore’s Law cost gains. And they’ve been working on the translation layer that mediates between the the weird world of flash and disk drivers. The engineers are figuring out the wrinkles and I have no doubt they will continue to improve.

But we also have to talk about where we are today.

20/20 vision: 20x the price and 20% better performance
Over on AnandTech they posted review last week that tested 3 drives on a 4 GB laptop running Vista Home Premium. The drives were:

• A 32 GB Mtron SSD MSD-SATA6025
• A Seagate Momentus 7200.2 160GB
• A Samsung MH80 FlashON 160GB

They tested maximum data rates - the Mtron smoking at 90 MB/s vs 49 and 38 for Seagate and Samsung - and Vista standby and hibernate modes. You’ll save a few seconds with the flash drives assuming Vista memory leaks don’t make the sleep modes as unreliable as they’ve been on XP and W2k.

But the real surprise were the pathetic application numbers. Here’s a chart I derived from AnandTech’s numbers:

I’d have thought that with double the bandwidth and a fraction of the access time that the flash drive would be a lot better than a 7200 RPM drive. Your average user couldn’t tell the difference between flash and disk based on these numbers.

Update: I’ve added the black line at 0% to indicate Seagate’s performance. The performance percentages are normalized against the Seagate drive. A positive percent means faster; a negative percent means slower. End update.

The StorageMojo take
One set of benchmarks on one product don’t decide the issue, especially this early in the game. But it gives one pause when a high-performance flash drive can’t do any better.

Maybe the translation layer isn’t the problem. Perhaps our OS’s do too many small random writes that hose flash performance. Whatever the case it is too early to assume that flash drives will even take a majority of the notebook market without better performance given their price and capacity limitations.

Comments welcome, of course. Maybe you can explain it.

Solid state disks have been around for decades and failed to make the IT mainstream. Why is that about to change?

I did a stint of SSD marketing at DEC in the early 90’s. Then, as now, SSDs offered low I/O latency and high IOPS capacity at an astronomical $/GB.

I’m exceptionally good at making abstract product features meaningful to customers. Everybody says “think outside the box” but few can actually to it.

That’s why so many product benefits sound the same: ROI! CapEx! OpEx! Faster! Eyes glaze over.

Every product has a story to tell and part of marketing’s job is to tell that story well.

But SSDs left me stumped.

I saw three major problems.

• Cost. People couldn’t get past the $/GB metric. The issues would vary, but the most common was “I can buy a heck of a lot of disks for what you’re asking and take care of performance and capacity at the same time.”
• Acceptance. As Geoffrey Moore points out, IT directors have a herd mentality. Since so few other people were using them, what was the chance that they should?
• Need. These folks didn’t know if they needed an SSD and they had no easy way to find out. The fabulous SSD specs couldn’t create a need from unseen problems.

Consumerization of SSDs
IT consumerization is the process by which low-cost and high volume products migrate into IT. We’ve seen it with CPUs, networks, OS’s, I/O busses, disk drives, data bases and applications. Volume rules.

SSDs are next.

How?
Most consumerization works because the volume “consumer” products are about as good as the proprietary products and cost a lot less. Not only to buy, but to engineer because a volume-based infrastructure of development tools, support chips and helpful add-ons develops. These are network effects.

SSDs are going to benefit through another mechanism: consumer education.

Thumb drives as baby SSDs.
The consumer SSD is the flash-based thumb drive. The thumb drive and its close relative, the flash-based notebook disk, knock down 2 of the 3 reasons people haven’t bought into SSDs while helping with the 3rd.

Cost. Notebook SSDs currently cost about $15/GB, while consumer SATA drives are available for ~$0.20/GB. That is a 75x difference, yet consumers are psyched to buy. Why? Because folks now understand that capacity isn’t the important metric when you buy storage.

Flash-based SSDs in high-end notebooks will accelerate the education of the senior execs who buy them. So when an IT director says “I want to buy SSDs for the datacenter” the C-level execs will understand. Not the details, but the concept.

Acceptance. All the high-end sub-notebooks will have SSD options. That means the technology is safe if Apple, Dell, HP and Lenovo are selling it and their friends are buying it.

Need. Consumer SSDs will help datacenter types because for the first time C-level execs will understand emotionally and intellectually that SSDs can make a difference in computer system performance and power use.

But a notebook SSD is imperfectly analogous to a datacenter SSD. For one, it doesn’t replace all existing storage, which means decisions have to be made on what to migrate. Further, the performance implications for servers aren’t widely understood, so the economic benefits are underestimated.

The SSD companies will still have their marketing work cut out for them.

The StorageMojo take
While consumer SSD’s - including USB thumb drives - will provide datacenter SSD vendors a door into the enterprise, it is up to the vendors to walk through it.

After years of playing on the margins of IT SSD vendors face a major cultural shift. They currently sell their products through specialized integrators who target the high performance applications where SSD benefits have been recognized for years by knowledgeable technical customers.

The new SSD markets are horizontal and require new integrators and a new story. Enterprise IT has fundamentally changed in the last 10 years with the advent of network-based services. This has created new opportunities for traditional SSD companies such as Texas Memory Systems and Solid Data while also opening up the market to new approaches such as Gear6.

Engineering-dominated tech firms have a spotty record of success when consumerization hits. Solid state disks will have a major growth spurt over the next five+ years. Will the old-line SSD companies be the ones to take advantage of it?

Comments welcome.

Flash is weirder than you know - so keep reading - and you’ll know
I think I already mentioned that the flash guys refer to “programming” flash when a storage guy like moi would say “write.”

But they have a point. Flash is really an EEPROM. More than a write-once medium, but not a true random access device, either.

Update: I asked Ted Wobber, one of the paper’s authors, to comment on this post. He generously did so, and I have incorporated his comments after Update: flags. I thank Ted for his comments and StorageMojo readers for their understanding and support.

Let me count the ways

1. Flash drives can only write zeros. Every write must be preceded by an erase because the only way to write a one is to erase first, which writes all ones. Every write means an erase followed by a write, which is slows performance.
2. To write a page you must first erase the entire block. NAND flash, the most common kind, is divided into blocks - typically 128 KB - and each block is divided into pages - typically 2 KB. To write a new page, the entire 128 KB block must be copied first - less pages due for rewriting - and the entire block rewritten. Writes have a lot of latency. And small random writes? Don’t ask.
3. There are no random writes in a block. Each block write starts with page 0 and proceeds in order to the 64th block. This is great for the blazing sequential write speeds that vendors happily quote, but it means that small random write performance is pretty awful.
4. Block size is a tradeoff, not a given. As flash chip capacities grow, keeping block size constant means more blocks to manage. For example, if flash drives were divided into 512 byte blocks, a 64 GB flash drive’s block table would require 128 million entries and about 384 MB of storage. With a 128 KB block, the table size is a more manageable 524,352 entries and less than 2 MB of storage. Vendors have the opportunity to improve flash drive performance through smaller block sizes and better block management techniques.
5. The most important piece of a flash drive is the translation layer. This software takes the underlying weirdness of flash and makes it look like a disk. The translation layer is unique to each vendor and none of them are public. Each makes assumptions that can throttle or help performance under certain workloads. What workloads? Hm-m-m-m.

What do you mean we, Kemo Sabe?
The last point deserves explication. Fortunately, some folks a lot smarter than me have done some work analyzing translation layer performance. Four amigos at Microsoft Research: Andrew Birrell, Michael Isard, Chuck Thacker, and Ted Wobber. In their paper A Design for High-Performance Flash Disks they spend some quality time analyzing a translation layer’s performance.

Short answer: random writes stink
What they found isn’t pretty. The flash translation layer, which takes NAND flash and makes it look like a block device to the USB mass storage protocol, creates performance potholes.

Random 4 KB writes incurred an average latency of 22 ms to completion. A blazing 190 KB per second. Hardly worse than a 1.8″ disk drive you say? There’s more.

The gap between writes affects write latency in a big way.

The paired-write problem
They found a couple of interesting things:

• Not all flash translation layers have the same performance. Smart engineers have figured out ways to mask some performance issues. The bad news: you don’t know which is which.
• Even costly Mil-spec flash drives, like the M-Systems FFD-25, suffer from slow random writes when multiple writes are distributed across the drive.
• If there is an easy way around the paired-write problem, the flash drive vendors haven’t found it yet.

The cost of paired writes
I really like this second figure from the Microsoft paper for demonstrating how the paired-write latencies add up:

Apparently this is the performance of the M-systems drive, but the paper doesn’t make that explicit. Update: this chart refers, like the first one, to a Lexar flash key. The M-Systems drive has very fast writes within a 16 MB range, while performance degrades over a wider range. The MR guys didn’t attempt a full forensic analysis on the M-Systems device. I regret my interpretation error.

Isn’t the paper “High-Performance Flash Disks”?
The bulk of the paper, which any serious flash-crazed storage geek should read, is their design for overcoming these issues. As they candidly admit at the end of this paragraph:

In this paper we describe data structures and algorithms that mitigate the problem of slow, non-sequential writes. One of the main areas of concern is that in a UFD, power can fail at any time (because the user unplugs the UFD). Furthermore, it is essential that device power-ups be fast since users are unlikely to tolerate a perceptible slowdown when inserting a flash device. Dealing with these two constraints led to a fairly complicated, but we believe still practical design described here.

Which suggests a couple of things to me. One, notebook flash drives with good write performance should be simpler to design since you have a battery and no sudden power failures. Almost ditto for the power-up problem.

These guys have been working on the PC-on-a-keychain problem too, so these simplifications probably weren’t interesting. But if I wanted to conquer the notebook flash drive market I’d be very interested. So guys, if you read this, please comment on how much simpler your design could be with the notebook assumptions.

Update: Ted Wobber, one of the authors, commented thus:

You are right that designing for laptops where there already is a battery and where the drive can’t easily be removed eliminates some constraints. However, many hardware designers might be hesitant to produce a storage component that depends on continuity of the external environment (e.g. power) for data stability.

My comment in response: I appreciate the concern about power assumptions. If I had to guess, I suspect the number of notebooks stolen exceeds the number of notebooks whose power suddenly ceases. I’m thinking a couple of seconds, max, while the flash drive stores its state - or whatever it needs to store, being a non-volatile storage device - that from a real-world, non-fastidious marketing guy perspective, the power-always assumption could be relaxed a bit without much danger to the very foundations of Western civilization. Which, depending on how you feel about Dick Cheney, may or may not be an acceptable risk.

Update: More comments from Ted:

The only thing I’d have a slight quibble about is the bit about vendors having a lot of flexibility with respect to block sizes. Although I’m not a hardware guy, I’d be about 99% sure of the following. NAND flash has a slow erase time. Memory is organized into large blocks in order to erase many pages at a time, and this is to achieve an economy of scale (parallelism). If you had to erase pages serially at 1+ ms per pop, digital cameras wouldn’t be happy since it would add .5 seconds per MB when storing an image (500 2k-pages x .001 s).

In fact, the current trend is to make blocks BIGGER as flash chips scale up. The latest SAMSUNG chips have 4K pages and 256K blocks.

As you know, our paper was about dealing with the physics of NAND flash (e.g. the big block sizes) without the random write penalty.

I have some thoughts about a couple of Ted’s points, which are competing with Ms. Mojo’s thoughts for my time. Sorry, Ted!

The StorageMojo take
The bottom line is that flash drive write performance will be all over the map as engineers try to optimize for a wide range of workloads. The software fix is to have file systems bunch writes, ala log-structured file systems. But that is a lot of work. It is easier to fix flash drive architecture and then build in volume.

Big sensor networks hoping to save on RAM-based SSDs by going to flash - keep hoping. I think it will be a while.

They guys at the hardware eval sites will have their work cut out for them figuring out flash drive performance. If a co-inventor of Ethernet and numerous other advances can’t figure it out, what hope can mere one-leg-at-a-time mortals have?

Comments welcome, as always. Moderation is a virtue, except in the defense of liberty.

All we need now is teh Steve to say it . . .
Thanks to alert reader Petieg, I’ve learned that according to Mac Rumors Sun CEO Jonathan Schwartz said today that

In fact, this week you’ll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the file system in Mac OS 10.

Jonathan is wrong, of course, but it was sweet of him to say it
Folks tell me that if ZFS is in Leopard it is pretty well hidden. I’ll stick to my prediction that Apple, as with HFS+, will put ZFS on OS X Server first before bringing it out later for the great unwashed.

For one thing it will fix a persistent problem Xserve RAID admins have: pulling out the wrong drive, or scrambling drives, and losing lots of bits. V cool.

Now I’m going to pat myself on the back
As I noted in Bring Me the Head of WinFS:

Can Apple Trump Vista With ZFS?
Apple now has a clear path to trump Vista’s aging data management with a port of ZFS. While not offering a relational database and the promise of a single cross-application data store, ZFS is a modern file/storage management system whose end-to-end data integrity and protection makes it a strong foundation for future innovation. NTFS and Apple’s HFS+ are no match for it. Let’s hope Apple says more at their World Wide Developer Conference in August.

Well, cough, cough, it looks like August 2006 is finally arriving next week.

The NEW news
I finally put two and two together and figured this out: ZFS will be great for flash disks. Unlike today’s Mac OS and Windows, ZFS bunches writes - kind of like NetApp’s WAFL - which is just what flash drives need since their random write performance is even worse than I’d realized.

In fact, it just occurs to me that it could be on the iPhone. Why? Because Bonwick, Moore, et. al. managed to write all this stuff in very little code.

More info coming on flash
I’ve been delving deep into flash disks. Can you say “weird”? My take now is that flash drives are to disk drives what quantum mechanics is to Newtonian physics. I’m planning to have something out next week.

The StorageMojo take
The real importance of ZFS on Mac is that it raises the bar for the entire industry. Journaled file systems are better than not, but as the consumer-driven IT market booms customers need better data protection and recovery tools. And flash drives need a compatible file system. ZFS goes a long way towards meeting both requirements.

Update II: No mention of ZFS in Steve’s keynote or on the Apple website. I doubt we’ll hear much about it until Apple includes it in a release of OS X Server. Maybe in October, maybe not.

Update: Want to know more about ZFS? I’ve been hot on it for over a year. See:

• ZFS: Threat or Menace? Pt I
• ZFS: Threat or Menace? Pt II
• Bring Me The Head Of WinFS
• ZFS On Leopard: How Cool Is That?
• Is Apple’s Time Machine Built On Sun’s ZFS?
• Means, Motive & Opportunity: Apple Kills the Media Center PC
• ZFS On Mac: Now All-But-Official

Comments welcome, of course.