If SSDs are so great, shouldn’t we see the results in TPC-C benchmarks? They are, and we do.

But there are some surprises.

Cost
Looking at the TPC-C top 10 performance results showed the dramatic impact SSDs have had on the cost per thousand transactions (tpmC).

• There are no top-10 disk-only results after 2009.
• The most expensive top-10 SSD result is some 15% cheaper than the least expensive disk-based result – and the other SSD results are much less.
• No top-10 results posted during 2009 – the depth of the great recession.

Capacity
The conventional wisdom has it that disks must be way over-configured to get enough IOPS. You’d expect to see disk solutions have a lot more capacity than SSD solutions in top-10 results.

But we don’t:

The highest capacity – 1760 TB – is for an Oracle SSD-based solution. Yet the lowest capacity solution – 83 TB – is also SSD-based and is also the cheapest per tpmC.

Are we seeing issues with the rest of the infrastructure?

The StorageMojo take
I’ll be taking a deeper dive into the data, but perceptions may be at odds with what this limited set of performance focused benchmarks is showing us.

Readers: what do you think?

Courteous comments welcome, of course. Events beyond my control have reduced StorageMojo’s usual posting frequency. Hope to get things back to normal over the next several weeks.

EMC’s Chuck Hollis blogged about The Vendor Beating a couple of months ago. The unspoken question in the post is “how do we understand what customers are telling us?”

He writes

As an employee of a large IT vendor, I’ve been at the receiving end of a reasonable number of vendor beatings.

Occasionally it’s richly deserved. But, sometimes, it’s masking a deeper set of issues that have very little to do any vendor whatsoever.

Unhappy customers, like unhappy families, are all unhappy in their own way. This customer appeared to be overstaffed, under-skilled and poorly managed.

Interpretation
Interpreting customer complaints and behavior is hard. When companies can’t decipher what customers want – which is usually what the company isn’t selling – it is easy and dangerous to tune them out.

Customers can tell you things about your company and products that you can’t directly discover for yourself, but what customers say may be different from what they think. And both are influenced by the customer’s context, which can include company politics, prior vendor experiences, knowledge deficits and employee level.

Diagnosis
Steve Jobs once said that customers don’t know what they want until you show it to them. Customers know what would improve the current product in the current use case, but they can’t imagine bringing multiple novel technologies to bear on a much broader problem.

Tablet computers flopped for years until the iPad crystalized the market. Everyone saw the tablet problems: thick; heavy; slow; clunky UI; poor battery life; and, thanks to low volumes, cost. Incremental improvements – faster processors, more RAM, larger disks – didn’t help.

Tablets required a deep rethinking and application of several novel technologies – flash, gestures, CNC case milling, an app store and an energy-efficient OS – to create a compelling user experience.

The iPad illustrates the problem of listening to customers: they described symptoms and suggest fixes, but couldn’t articulate the underlying problem: how the use case differs from desktop and notebook PCs. That requires an act of imagination, not transcription.

The StorageMojo take
In Chuck’s post an EMC presales engineer identified the root cause of the customer’s pain:

. . . the database environment had grown willy-nilly over the years — it wasn’t laid out well, the queries weren’t particularly well written, and so on.

Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue than just storage performance.

But the larger question is: with high-speed and high-capacity SSDs, why isn’t this customer moving to an infrastructure that doesn’t need this fancy tuning? EMC can’t manage the fight between DBAs and storage admins, but they could be making it less contentious.

From within the EMC ecosystem the solution is clear: more training, professional services and faster gear. But from the outside the question is: who is building “it just works” high performance storage?

Courteous comments welcome, of course. I admire Tucci’s innovative EMC business model: outbid everyone else for chasm-crossing companies; give them global distribution and support; and watch the bucks roll in. It may not be innovative technically but it is innovative.

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

Sometimes in the midst of the endless tweaking needed to maximize storage performance one just wants to say “screw it! Put everything in RAM!” And that’s just what RAMCloud does.

Disk is the new tape, flash the new disk, DRAM the new flash.
RAMCloud is a research paper (pdf) and an open software project. The goal is enterprise-class availability with every bit of active data stored in DRAM, not disk or flash, for maximum performance. It is a key-value object store today, though as pure software that could change.

It’s the brainchild of John Ousterhout, a Stanford prof who invented Tcl back in the 80s at Berkeley.

Isn’t DRAM volatile and costly?
Right on both counts, grasshopper, so RAMCloud isn’t a 1 for 1 disk-style architecture. No Google FS-style triple replication here, or RAID-style erasure coding.

Instead RAMCloud uses buffered logging:

. . . a single copy of each object is stored in DRAM of a primary server and copies are kept on the disks of two or more backup servers; each server acts as both primary and backup. However, the disk copies are not updated synchronously during write operations. Instead, the primary server updates its DRAM and forwards log entries to the backup servers, where they are stored temporarily in DRAM.

Instead of working around crashes – using multiple object copies as scale-out storage does – RAMCloud recovers lost data from the DRAM logs or disk drives to replicate the lost data at high speed. That’s possible because all the log data is in DRAM or spread across many disks.

In a recent paper (Fast Crash Recovery in RAMCloud) (pdf) Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John Ousterhout, and Mendel Rosenblum (co-founder of VMware) go into more detail on this critical feature.

The key elements are:

• Scale. Servers scatter their backup data across all other servers so thousands of disks can serve the recovery.
• Log-structure. Reduces complexity and offers high performance.
• Randomization. Many decisions need to be made in a large cluster. Rather than CPU, time and bandwidth consuming determinism, injecting randomization speeds decisions with less overhead.
• Dynamic tablets. The key-value store tracks resource usage within a single table and ensures that no single partition is too large for fast restores.

DRAM is volatile so the log replication data is spread to other servers on other racks for redundancy before being committed to disk. Still, total system write throughput is limited by the disk write speed, whose limits are a key reason people are moving from disks. Flash drives may help, but other techniques, such as log truncation and sharding make it possible to get good performance from several thousand SATA drives.

How good? The team reports that in a 60 node cluster they recover 35GB in 1.6 seconds. With more nodes larger partitions should be restored even faster. Scale is good.

Lights out!
Power failures wipe all the data in DRAM. The obvious defense is to avoid failures: combine battery backup with diesel generator sets. Power ride-through will handle interruptions into the hundreds of milliseconds.

But who is going to trust that? That’s why future commercial implementations will insist on logging to stable storage, such as the flash SSDs.

They’re getting cheaper fast – faster than DRAM – which will make this a common approach.

Cost
Professor Ousterhout kindly sent a short note about cost, correctly noting that

. . . if you measure cost/operation, DRAM is roughly 100x cheaper than disk, since a disk can only perform about 100-200 operations/second. This is why RAMCloud makes sense for data-intensive applications. . . .

While you and I might find that persuasive, too many enterprises don’t. The deep conservatism of the storage culture – both figuratively and literally – makes cost a good excuse to stay with the tried and true, and easy to explain to CFOs.

The good news for the company I hope he is starting is that the primacy of $/GB is slowly eroding as customers see the system level savings from fast storage. SSD vendors and companies like TMS and Kaminario are breaking trail for RAMCloud.

The StorageMojo take
Make no mistake: RAMCloud is a research project, not a commercial product, years and million$ away from commercial application. But the concept is promising.

Imagine a world where data layout doesn’t matter, where apps are optimized for sub-millisecond storage, where 100 byte I/Os are faster and just as efficient as 8KB I/Os. The architectural implications are huge and would take a decade or more to absorb.

RAMCloud raises the thorny issue of tiering: getting hot data on the hot storage and everything else off to disk. There are OK answers for tiering but nothing insanely great.

RAMCloud shows we’re far from the end of the line in what storage can do. Faster, better, arguably cheaper: 2 out of 3 ain’t bad.

Courteous comments welcome, of course. A shorter version of this post appeared on ZDNet.

VMworld is the best storage show I’ve seen in years. VMware’s severe storage problems leave users hungry for solutions – and your friendly neighborhood storage industry is happy to oblige.

It’s almost as if VMware were owned by a storage company.

Flash everywhere
Fusion-io, Nimble Storage, Nimbus Data, Avere, Pure and more were talking about how well flash supports VMware. Fixes VDI boot storms, deduped VMDKs, I/O bound servers and much more.

Pure Storage
Here is Pure’s Matt Kixmoeller giving a nifty demo in this 50 second video:

Not exactly sure what those thousand VMs were doing. Maybe Pure will comment.

Falconstor
I lost track of Falconstor due to their OEM focus and sprawling product line. New CEO Jim McNiel has refocused the company – with the help of former Cheyenne teammates – on backup, business continuity/DR, dedup and virtualization.

Their clustered Network Storage Server turns all of Fstor’s products into tin-wrapped software suitable for channel partners. Takeaway: forget what you knew about them; they are a new company.

Virsto
While the release of their storage hypervisor for VMware makes them seem like a new company, Virsto has been shipping product for over a year, but on Hyper-V, not VMware. Microsoft lost interest in server virtualization and Virsto moved on.

Their product is a virtual appliance that:

. . . runs in each host, creating a transparent virtual storage layer that is thin provisioned, fully cluster-aware, supports very rapid snapshot and clone creation, and scales to support tens of thousands of high performance snapshots and clones.

Virsto . . . decouple[s] application performance from any dependence on the rotational latencies and seek times of underlying disk associated with random writes. All random writes are sequentialized and written directly to a transparent logging device . . . and then asynchronously de-staged to primary storage. . . .

Net/net: high performance virtual storage regardless of underlying physical storage. Virsto offers a free trial – if you try it, let me know how it works.

But wait! There’s more!
Cloud-related products from StorSimple, AMAX and Raidundant continue to pick at the problem of how/when/where cloud integrates with the enterprise.

The StorageMojo take
Many cool products and ideas. The storage problems of many virtual machines are not unlike those of earlier time-shared virtual memory systems, but the scale is much greater.

And when the scale is greater the problem is fundamentally different. As virtualization grows we’ll need to see more creative answers beyond deduplication and flash.

Courteous comments welcome, of course. Message to SNIA: storage networking is passé. Time to retool for the world of virtual machines, noSQL databases, scale-out storage and flash-enabled architectures. New name would be a start.

Pure Storage, a well-funded ($55M) valley startup, came out of hiding last week with a startling claim: enterprise flash that is cheaper^1,2,3 than disk.

¹Cheaper after compressing and deduping the data.

²Cheaper after using almost all the flash capacity, which you can’t do with disks because performance suffers.

³Cheaper compared to the most expensive disk-based enterprise storage you can buy.

Your mileage will vary
4 years ago an EMC VP predicted that flash would replace high-end disks in 2010. That didn’t happen. Why?

After 5 years of hype, enterprises are still leery of flash. Endurance, reliability, data integrity, security, integration – all unanswered questions. At least by the other IT guys in town, even if vendors think they’ve nailed it.

So people buy the known quantity: high-end drives.

The StorageMojo take
Kudos to Pure’s marketing for making a bold, attention-grabbing statement. Too often marketing falls back on the trite-and-true “faster, better, cheaper.”

But IT wants to solve old problems while not introducing new ones. The 10x performance boost would be enough, if IT believed.

Pure’s challenge – as well as other companies with similar products – is to convince IT that not only is flash ready for primetime – but that compression and dedup are too.

And once they do that, why not use them with disks, as well? As Nimble Storage has found, inline compression is now easily handled in software on multi-core chips.

With raw SATA drives down to 3¢/GB, storage vendors have ample opportunity to squeeze costs out. The flash/disk competition will be good for all of us.

Courteous comments welcome, of course. The storage high-end is more active than its been in 15 years. Good!

A post last month in ACM’s Queue raised a disturbing point around block-level deduplication in flash SSDs: it could hose your file system.

De-dup is a Good Thing, right?
Researchers found that at least 1 Sandforce SSD controller – the SF1200 – does block-level deduplication by default. Many file systems write critical metadata to multiple blocks in case one copy gets corrupted. But what if, unbeknownst to you, your SSD de-duplicates that block, leaving your file system with only 1 copy?

Yup, corruption of 1 block could wipe out your entire file system. And since all the “copies” point to the same corrupted block, there’s no way to recover.

Most Unix superblock-based FSs and ZFS could be pooched by loss of a single block. NTFS also mirrors critical metafile info and could be vulnerable as well.

To be fair, AFAIK no one has reported this failure in the wild, so it is conjecture today. That said, it may have happened to people who didn’t realize what went wrong.

But in the world of storage, if something can happen it will, usually at the worst possible time. Have you seen a total data loss on an otherwise functioning SSD?

The StorageMojo take
I’ve made calls to a number of vendors to get their responses, including Sandforce, Intel, Texas Memory Systems and OCZ. With any luck we’ll soon have a 1st pass on who does what to your data.

Don’t panic: not all SSD controllers do this. Texas Memory Systems controllers don’t, partly because they don’t use MLC flash and partly because minimizing capacity use and maximizing data availability are conflicting goals, and they chose the availability over capacity.

Also note that the SF-1200 is offered as a consumer grade controller. Not clear what Sandforce does with the rest of their line, but their site does repeatedly reference their “DuraWrite” technology which appears to include block-level dedup.

Just last week StorageMojo recommended faster adoption of SSDs in the enterprise – and still does. But this once again underlines the need for mirroring. The sooner we find these issues, the sooner they’ll be fixed.

Watch the comments for vendor info, and I’ll update this post with more info if and when it develops.

Update:Here is the Sandforce response:

In the recent article by David Rosenthal he mentions a conversation with Kirk McKusik and the ZFS team at Sun Microsystems (Oracle). That conversation explains why it is critical that meta data not be lost or corrupted. He goes on to say that “If the stored metadata gets corrupted, the corruption will apply to all copies, so recovery is impossible.”

SandForce employs a feature called DuraWrite which enables flash memory to last longer through innovative patent pending techniques. Although SandForce has not disclosed the specific operation of DuraWrite and its 100% lossless write reduction techniques, the concept of deduplication, compression, and data differencing is certainly related. Through all the years of development and OEM testing with our SSD manufacturers and top tier storage users, there has not been a single reported failure of the DuraWrite engine. There is no more likelihood of DuraWrite loosing data than if it was not present.

We completely agree that any loss of metadata is likely to corrupt access to the underlying data. That is why SandForce created RAISE (Redundant Array of Independent Silicon Elements) and includes it on every SSD that uses a SandForce SSD Processor. All storage devices include ECC protection to minimize the potential that a bit can be lost and corrupt data. Not only do SandForce SSD Processors employ ECC protection enabling an UBER (Uncorrectable Bit Error Rate) of greater than 10^-17, if the ECC engine is unable to correct the bit error RAISE will step in to correct a complete failure of an entire sector, page, or block.

This combination of ECC and RAISE protection provides a resulting UBER of 10^-29 virtually eliminates the probabilities of data corruption. This combined protection is much higher than any other currently shipping SSD or HDD solution we know about. The fact that ZFS stores up to three copies of the metadata and optionally can replicate user data is not an issue. All data stored on a SandForce Driven SSD is viewed critical and protected with the highest level of certainty.

Readers: how does that sound to you?
End update.
Update 2: Oddly enough, the Sandforce web site specifies the SD-1200 controller at

ECC Recovery: Up to 24 bytes correctable per 512-byte sector
Unrecoverable Read Errors: Less than 1 sector per 1016 bits read

which is about where many enterprise disk drives spec’d – and quite a bit less than 10^-29. Hmm-m.
End update 2.

Update 3:
Spoke to James Myers of Intel. He said that no current Intel SSD uses any form of compression, including dedup. He also cautioned against making too much of the risk: after all, you’d have to have an unrecoverable read error AND it would have to be that 1 critical block. Perhaps, he suggested, file systems that do use multiple copies of critical FS metadata could slightly alter the copies to eliminate the possibility of deduplication.
End update 3.

Courteous comments welcome, of course. TMS has been advertising on StorageMojo for a couple of years.

IT pros are always skeptical about new technology. Is it surprising that flash SSD’s are getting the gimlet eye?

The big worry seems to be endurance. Nobody wants to buy an expensive SSD and have it fail after a year on the job.

But IT infrastructures are designed to manage endurance failures. LTO tape, for example, is specified for a few hundred head passes. Yet tape is the paragon of data persistence.

Hard drive failure rates aren’t low enough for any of us to consider storing important data on one without backup. So why are IT pros so skittish about flash SSD’s?

Experience
Or rather, lack of experience. Flash SSDs are evolving rapidly, with new generations arriving every 12 to 18 months.

It takes time for experience with new models to percolate. In the meantime, bad experiences with earlier generation drives continue to circulate.

Vendor secrecy about failure rates and modes doesn’t help. Until the Bianca Schroeder/Google/CMU disk drive studies were released 4 years ago, we had no independent large-scale reliability data.

I hope it won’t take 20 years before we get that information on SSD’s. How about it, vendors?

Reliability
SSD’s may turn out to be more reliable than hard drives but I won’t believe it until I see independent data. The lack of moving parts is a plus but about half the failures and this drives come from the electronics not the spinning bits. SSDs have most of the same electronics.

SSD equivalent of a head disk assembly
Plane failures are a major trouble spot. Each die consists of two planes. These planes are prone to sudden failure, wiping out half the data on a die.

Most chip carriers contain multiple stacked dies, so a plane failure will remove anywhere from a quarter to an eighth of the chip’s total storage. Most flash controllers lay out the data in ways similar to a RAID array to guard against data loss.

What to look for
Since Maxtor’s well-deserved demise we’ve had reasonable parity between disk drives and disk drive vendors. But that is not the case with the still maturing flash drive market.

Storage Newsletter recently published a list of 85 SSD vendors, most of whom none of us have heard of. Many are focused on the embedded systems market, but also because the SSD market barriers to entry are small: buy controller chip; buy flash on the spot market; gen up a PC board and voilà you are in the SSD market.

But flash that ends up on the spot market at rock-bottom prices is often marginal. The big buyers, like Apple, get first dibs on the best.

SSDs made with spot-market flash and a no-name – USB thumb drive? – controller will have a lot more problems. Which is to say that in today’s SSD market brandnames count.

Other things to look for are a guarantee of total write capacity. Another is a statement on the amount of over provisioning the drive has.

Even better: a five-year guarantee such as Seagate popularized with disks and that Intel just started offering on one of its SSD lines.

The StorageMojo take
I have been as skeptical as anyone on SSDs – read some of my earliest posts – but the time for skepticism has passed. Of course, perform careful evals on any new IT product. But the best flash SSD’s are ready for the enterprise today.

And here’s an even more radical conclusion: the best consumer SSD’s are ready for the enterprise as well. Using any SATA drives in your enterprise?

The key: how is the SSD architected into the system? If it is storage tier the data has to be protected just like a RAID array. If it is a cache you have more flexibility – as long as the data is also on disk.

Yes, it’s more difficult to separate the wheat from the chaff in the SSD market today. But there are quality products available today.

Courteous comments welcome, of course.
Started thinking about this is result of the research project I did a few months ago. Leading-edge storage managers with workloads that would benefit enormously by flash SSD’s weren’t seriously evaluating them today. Big surprise. What do you think?

I was surprised by the number of questions at last week’s webinar – many more than we could get to – so I’m answering a few here.

Performance
Q: Can Robin talk about performance and how does flash help solve I/O bottleneck?

NAND flash is very good at random reads, and a good SSD can handle thousands per second, compared to a disk drive’s 150-500 (for a high-end drive). That’s one reason arrays are popular: they provide higher random I/O performance because multiple heads are seeking. But that’s also why capacity utilization is so low: the disks come with more capacity than most applications use.

So not only are you buying multiples of the most expensive disks, but then you only use a fraction of their capacity. This is why flash SSDs are predicted to kill the high-end drives even though SSDs cost much more per GB: a single SSD can eliminate 6-10 hard drives. That is a major cost saving.

Q: So the low cost assumes that the cache is read only, otherwise it needs to be RAID? That comes off as misleading.

While flash SSDs are much better at random reads than they are random writes, they still beat several high-end disks at writes.

Since most workloads are 80%-95% reads, an SSD that can handle 1,000 writes per second can handle a lot of work. Disks are still the most cost-effective solution for large sequential workloads because their performance is close to SSDs and they are so much cheaper.

Reliability
Q: What are Robin’s thoughts on the reliability of SSDs? We have seen failure rates of over 10% on drives less than two months old.

Flash SSD reliability today is all over the map. As flash SSD technology matures, I’d expect to see drive reliability rates converge. 5 years ago disk reliability was fairly similar with the glaring exception of Maxtor.

That said, it’s useful to recognize that there’s a lot more design and sourcing variability in SSDs. If someone uses the cheapest parts – and there are plenty available – they can offer good specs but highly variable reliability.

If they leave out too much redundancy they’ll have a cost advantage but will be more vulnerable to chip and plane failures. The market will eventually settle on similar specs for each application, but we’re years away from that.

Q: You mentioned 10,000 writes and failure can begin, what is that in years?

Like so many storage specs, that 10k write spec for MLC flash is a statistical one that can be improved upon by more robust ECC, as this chart from SNIA shows:

But the most important way to improve upon it is by increasing the capacity of the SSD. Double the size of the SSD and you double the total write capacity.

As to what that is in years, the industry is still figuring out how to spec that. The best vendor spec I’ve seen so far has been from Intel – 5 years at 20 GB of writes per day.

Courteous comments welcome, of course. I enjoyed the webinar – a new experience for me – and not just because I got paid. The crew at Nimble was a pleasure to work with. Here’s a link to the webinar.

A new media adventure
StorageMojo is decamping to Silicon Valley tomorrow. On Thursday I’ll be doing my first webinar – ever – with Dan Leary of Nimble Storage on the topic Not Just a Flash in the Pan: Proven Strategies for Successful SSD Deployment in the Midsize Enterprise.

Didn’t realize how long that title was until I tried to put it on a slide.

The StorageMojo take
My focus is on addressing IT concerns about flash-based products. One part carrot – look at the massive benefits that flash SSDs bring to storage architectures – and one part medicine – IT folks attitudes towards flash tend to be gloomier than the product’s rapid progress justifies.

My own flash posts, nearly 5 years ago, were downbeat – too many unanswered questions – and as in any new area some companies made better decisions than others. The resulting noise made “go really slow” the IT default. In the recent survey on StorageMojo I found that IT remains more skeptical than I’d expected.

But we’re moving past the fear and loathing stage with 4th generation flash SSDs. The question now: how do new system architectures enabled by flash help me do more with less? Nimble has a pretty good answer to that question.

Courteous comments welcome, of course. I have some free time on Wednesday if anyone would like to chat. And I’ll be hitting the Ryowa ramen house in Mountain View at least once.

JIm Gray’s comment that disk is the new tape is truer today than it was 8 years ago. We’ve been adding caches, striping disks, modifying applications and performing other unnatural acts to both reduce and accommodate random reads and writes to disk.

Flash changes the calculus of 20 years of storage engineering. Flash gives us abundant random reads – something hard drives are poor at – and reasonable random writes to whatever hot data we choose.

In a feverish burst of design and investment we’ve tried flash everywhere in the storage stack: disks; PCI cards; motherboards; controllers; built-in tiering; and appliances. These products have been focused on enterprise datacenters or very targeted applications where the cost of flash was justifiable.

But clarity is emerging. It isn’t so much where you put the flash as what you ask the flash to do. There are three requirements:

• Valuable data. Flash is an order of magnitude more costly than disk.
• Often accessed. If not, leave it on disk.
• Enables new functionality and/or lowers cost. If it doesn’t, why bother?

The buyer’s burden
These requirements frame a basic point: optimizing for flash requires a systems level approach. Adding flash can make current architectures go faster, but that isn’t the big win.

Buyers looking for an economic edge must make a cognitive leap: the old ways are no longer best. Flash enables efficiencies and capabilities in smaller systems that only costly enterprise gear had a few years ago.

Tiering
Tiered flash solutions are the most common approach today. Tiering software has improved in recent years, making the movement of data between flash and disk safe, fast and granular.

We’ve started to at least see interest in the midsize enterprise, like the EqualLogic hybrid SAS/SSD array in VDI deployments.

Metadata and cache
The best fit for flash today is metadata and caching. These best meet the requirements for value, access and functionality.

Once metadata is freed from disk constraints we can combine it with caching to build high-performance systems on commodity hardware. The win for innovators is to design new metadata structures and caching algorithms for flash.

They can design the (write) data layouts to best take advantage of the physics of disk and flash, such as with Nimble Storage’s CASL architecture, which combines a large flash cache with full-stripe writes, is one example.

Flash is also an important enabler for low-cost de-duplication because it’s cheaper to keep block metadata – fingerprints or hash codes – in flash than it is in RAM. Some vendors are encouraging the use of de-duplicated storage for midrange primary storage, enabled by flash indexes or caches that make it feasible to reconstruct files on-the-fly.

The StorageMojo take
Shaking off the effects of 50 years of disk-based limitations isn’t easy. Our disk-based orthodoxy is ingrained in architectures and our thinking.

But buyers face a difficult job: evaluating architectures and algorithms to choose products for eval. A shortcut: look for architectures that collapse existing storage stovepipes to reduce cost, total data stored and operational complexity. The three are related and offer the big wins.

In the last 10 years raw disk capacity cost has dropped to less than a 10th of what they were, but the cost of traditional storage systems haven’t. The culprits: operating costs; storage network infrastructure costs; and capacity requirements that have risen faster than management productivity.

The flood of data continues to rise, but cost and complexity doesn’t have to rise with it. We can – and are – doing better.

Courteous comments welcome, of course. I’ve been working with Nimble Storage lately and like what they’ve done.

Forget the flame wars over moving window versus fixed block de-duplication. A recent paper, A Study of Practical Deduplication (pdf) from William J. Bolosky of Microsoft Research and Dutch T. Meyer of the University of British Columbia found that whole file deduplication achieves about 75% of the space savings of the most aggressive block level de-dup for live filesystems and 87% of the savings for backup images.

Presented at FAST 11 – and winner of a “Best Paper” award – the researchers looked at file systems from 857 Microsoft desktop computers over 4 weeks. Researchers asked permission to install rather invasive scanning software.

The scanner took a snapshot using Window’s volume shadow copy service and then recorded metadata about the file system itself. The scanner recorded each file’s metadata, retrieval and allocation pointers as well as the computer’s hardware and systems configuration. They excluded the pagefile, hibernation file, the scanner itself and the VSS snapshots the scanner created.


During scanning each file was broken into chunks using both fixed block or Rabin fingerprinting. They also identified whole file duplicates.

Rabin uses dynamically variable block sizes to maximize compression. Figuring out where to break the file adds to the overhead.

The resulting data set was 4.1 TB compressed – too large to import into a database – and was further groomed to lose unneeded data.

De-dup issues
De-duplication is expensive. You’re giving up direct access to the data to save capacity.

The expense is in I/Os and CPU cycles. Comparing each chunk’s fingerprint to all other chunks is nontrivial. De-duplication indirection adds to I/O latency. A file’s chunks are scattered around, requiring small and expensive random I/O’s to read.

Older techniques, such as sparse files and Single Instance Storage, are more economical even if their compression ratios aren’t as high. Fewer CPU cycles, less indirection and good compression.

The StorageMojo take
If capacity is expensive – read “enterprise” – and I/Os cheap – SSD or NVRAM in the mix – fancy dedup can make sense. It is at the margin of capacity cost and I/O availability that the value prop gets dicey.

Low duty cycle storage – SOHO – with plenty of excess CPU and light transactions could use deduped primary storage. But with a 10 TB of data to backup, most users would’t notice the difference between whole file and 8KB Rabin.

It’s the price tag and user reviews the SOHO/SMB crowd will be looking at.

Courteous comments welcome, of course. The paper also included some interesting historical data about Windows file system that I covered on ZDNet.

It sounds unlikely: technology that makes MLC flash as reliable and long-lived as more expensive SLC? Right!

Or even better: make 3 bit per cell flash equivalent to 2 bit MLC? What are these guys smoking?

Whatever it is, I’d like to try some myself. Because it looks like they have the goods.

Flash problems
Flash vendors are close-mouthed about flash problems, not wanting to kill the goose that’s laying billion$ in revenue. But the problem most people worry about – life span – isn’t much of a problem.

So what is? Errors. The good news: most NAND flash errors have predictable causes. And what can be predicted can be corrected.

That’s Avraham Meir’s story and he’s the CTO of Anobit. We had a chat at SNW.

According to their web site:

Mr. Avraham Meir is an internationally recognized authority in NAND Flash technology and products. Prior to joining Anobit, Mr. Meir was VP Corporate Engineering at SanDisk (NASDAQ: SNDK), and the CTO at M-Systems (NASDAQ: FLSH, acquired by SanDisk).

Flash errors are commonly due to cross coupling between adjacent cells, read disturbs and program disturbs, and reading or programing the wrong cell. Retention impairments and endurance impairments are more common as geometries shrink.

Data retention phenomena and endurance effects are not random: normally high levels go to lower levels. In adjacent cells, one high and the other low, voltage will leak across to reduce the higher voltage.

So the first stage is reducing errors is knowing how flash impairments behave to predict an error. Then working with flash vendors they can repair the impairment in the wild.

By reducing the errors first and then applying signal processing and error correction Anobit makes less-reliabile flash look like more reliable flash.

Extending endurance has another effect: with flash as the number of writes increases so does the error rate. And as the endurance increases the data retention time drops to as little as a few months. Write once and your data may last for years. But don’t trust that old thumb drive.

The StorageMojo take
Anobit has an impressive IP portfolio: go to the USPTO patent application search engine, type in anobit and you’ll see what I mean. And they have about a dozen granted already.

What is evident is that flash as a medium is still in its infancy. If it turns out, as some predict, that flash will hit a geometry wall in a generation or two, Anobit’s technology looks to be transferrable to whatever next-gen non-volatile device wins. In storage, reliability and endurance are problems that will never go away.

Courteous comments welcome, of course.

It isn’t about the capacity. It’s the performance.

I few weeks ago I asked StorageMojo readers to help out Jim Handy of Objective Analysis, a semiconductor research firm. They did, and Jim got some surprising results.

Now that I have finished interviewing 21 PCIe SSD users (both Fusion-io and Texas Memory Systems users, thanks to help from StorageMojo) I can step back and see the common threads. One surprises me a bit.

The almost universal reply from users is that, although they were able to reduce the use of other hardware by adding flash to their system, they were unhappy that the price per gigabyte for flash is so much higher than HDDs.

In one or two cases the SSD displaced very expensive hardware. One company was able to re-deploy a $500K high-speed SAN which it now uses to perform backups.

A couple of others slashed their DRAM sizes by as much as 2/3, leading to power savings. Answers.com decreased the server count in each of the company’s five data centers from four to one by eliminating the need to shard its databases.

There are side benefits to such moves.

• Any future system expansion that uses the new approach will be able to get by with less hardware, saving money along the way.
• A smaller system is usually easier to manage than a large one, saving in HR costs.
• In some cases fewer software licenses will be required, cutting operating costs appreciably.
• Smaller systems are likely to consume less power than their predecessors.

This last may seem to be a small point until its effect over time is considered. The last three all accumulate over time to become important sums.

Yes, this is a TCO argument, I agree, but it’s a GOOD TCO argument. Although the cost per gigabyte of an SSD may be an order of magnitude greater than that of hard drives, very many systems can save money by deploying these devices. Unfortunately it’s difficult to tell exactly how this will come about until after the SSD is deployed.

As I knuckle down to compile the survey data into a report I’ll be considering the impact of SSDs on the data center, and how the findings of these 21 companies can help others who have not yet adopted the technology.

This exercise has reinforced my position that flash will become widespread in the data center, but not as storage; it will rather be thought of as “something else” that brings cost and power efficiencies that can’t be attained through more established methods.

The StorageMojo take
Jim’s findings surprised me. But thinking a bit I’m surprised that I’m surprised.

Why? Back in ’06 I wrote about the Capacity Illusion, the storage industry equivalent of the economist’s “money illusion.”

The capacity illusion: most customers need IOPS, but they buy capacity. And then they moan about how much unused capacity they have or, in the SSD case, how much SSD capacity costs.

While faster and cheaper SSDs are rewiring data center architectures, something more powerful is needed to rewire customers. A concerted industry effort to classify and promote on IOPS, perhaps?

Courteous comments welcome, of course. Speaking of research, if you haven’t done so yet, please support StorageMojo by completing the Internal Storage Survey. More about it in the previous post. Thanks!