EMC’s Chuck Hollis blogged about The Vendor Beating a couple of months ago. The unspoken question in the post is “how do we understand what customers are telling us?”

He writes

As an employee of a large IT vendor, I’ve been at the receiving end of a reasonable number of vendor beatings.

Occasionally it’s richly deserved. But, sometimes, it’s masking a deeper set of issues that have very little to do any vendor whatsoever.

Unhappy customers, like unhappy families, are all unhappy in their own way. This customer appeared to be overstaffed, under-skilled and poorly managed.

Interpretation
Interpreting customer complaints and behavior is hard. When companies can’t decipher what customers want – which is usually what the company isn’t selling – it is easy and dangerous to tune them out.

Customers can tell you things about your company and products that you can’t directly discover for yourself, but what customers say may be different from what they think. And both are influenced by the customer’s context, which can include company politics, prior vendor experiences, knowledge deficits and employee level.

Diagnosis
Steve Jobs once said that customers don’t know what they want until you show it to them. Customers know what would improve the current product in the current use case, but they can’t imagine bringing multiple novel technologies to bear on a much broader problem.

Tablet computers flopped for years until the iPad crystalized the market. Everyone saw the tablet problems: thick; heavy; slow; clunky UI; poor battery life; and, thanks to low volumes, cost. Incremental improvements – faster processors, more RAM, larger disks – didn’t help.

Tablets required a deep rethinking and application of several novel technologies – flash, gestures, CNC case milling, an app store and an energy-efficient OS – to create a compelling user experience.

The iPad illustrates the problem of listening to customers: they described symptoms and suggest fixes, but couldn’t articulate the underlying problem: how the use case differs from desktop and notebook PCs. That requires an act of imagination, not transcription.

The StorageMojo take
In Chuck’s post an EMC presales engineer identified the root cause of the customer’s pain:

. . . the database environment had grown willy-nilly over the years — it wasn’t laid out well, the queries weren’t particularly well written, and so on.

Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue than just storage performance.

But the larger question is: with high-speed and high-capacity SSDs, why isn’t this customer moving to an infrastructure that doesn’t need this fancy tuning? EMC can’t manage the fight between DBAs and storage admins, but they could be making it less contentious.

From within the EMC ecosystem the solution is clear: more training, professional services and faster gear. But from the outside the question is: who is building “it just works” high performance storage?

Courteous comments welcome, of course. I admire Tucci’s innovative EMC business model: outbid everyone else for chasm-crossing companies; give them global distribution and support; and watch the bucks roll in. It may not be innovative technically but it is innovative.

Reader Kyle asks a good question:

SANs are advertised up the wazoo as having lots of internal redundancy such as redundant power, redundant controllers, etc. I’ve spent enough time with redundancy to know that having two pieces of hardware often doesn’t cut it. I was wondering what the real story is from someone who has spent a lot of time in the storage space. Do complete SAN failures really pretty much *never* happen or are they just on the rare side? If so what are the common points of failure? Perhaps people, the OS, non-redundant hardware parts?

Please, SAN folks, tell StorageMojo readers your experience. In the meantime, here’s

The StorageMojo take
Kyle asks 2 questions: how reliable and available are the individual devices that make up a SAN and how reliable and available is the system – the SAN as a whole.

Redundancy is aimed at ensuring availability. Because of the redundancy’s greater component count you also have more failures.

Failures of redundant components shouldn’t affect availability – assuming, that is, that failures are not correlated. That assumption turned out not to be true of RAID arrays, making them less available than advertised.

How much redundancy is enough? Customers generally prefer triple redundancy if they can afford it, partly for availability and partly for performance: losing ⅓rd of system performance is less painful than ½. But for the moonshots, NASA chose quintuple redundancy on critical systems.

Yet I’d guess that most are more concerned about SAN system availability – which includes not only what we consider SAN hardware, but also the server-side HBAs, drivers and management software. It is here that the nastiest bugs lurk: untestable interactions between applications, drivers, firmware and architecture that bite us hard – and crash entire SANs.

But what is your experience, gentle reader? Many of us would like to know.

Courteous comments welcome, of course. Update: Bayesian analysis is the best tool to evaluate system-level availability, as noted in this StorageMojo video. Sadly, the tool referred to is no longer online. Anyone want to take a whack at a new one?

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

Amazon Web Services architect James Hamilton has been posting on network issues for over a year and researching them much longer. As Ethernet becomes the de facto SAN technology, his views become more relevant to the larger storage market.

Critique
Part of Mr. Hamilton’s concern is the structure of the networking industry: the high margins; the dominance of a single player, Cisco; the closed technology; and the heavy vertical integration. All antithetical to the dynamics that have driven server costs down so successfully in the last 20 years.

These are issues the storage industry knows too well. But Mr. Hamilton is more concerned about the waste the current high-cost industry structure causes.

Waste?

Workload placement
The cost of network bandwidth leads to network over-subscription. Networks are configured as tree topologies: the further you move from end nodes the worse the over subscription.

As described in the 2009 Microsoft Research paper VL2: A Scalable and Flexible Data Center Network:

. . . the capacity between different branches of the tree is typically over- subscribed by factors of 1:5 or more, with paths through the highest levels of the tree oversubscribed by factors of 1:80 to 1:240. This limits communication between servers to the point that it fragments the server pool — congestion and computation hot-spots are prevalent even when spare capacity is available elsewhere.

This throttles data center performance by limiting server-to-server bandwidth, fragmenting resources and reducing network utilization. The latter reflects the redundant paths needed in case of switch failure: ≈50% or more of costly data center bandwidth goes unused.

As might be expected, big Internet data centers like Amazon’s have complex and unpredictable workloads. They need lots of bandwidth between all servers all the time.

A solution
The VL2 paper describes an experimental solution to these problems that includes location-specific and application-specific addressing, multi-path traffic load balancing and a novel directory design that efficiently handles lookups and updates to network mappings.

In an 75-node test cluster the design moved 2.75TB of data in 395 seconds – 94% of maximum network bandwidth – at a fraction of the cost of current enterprise networks. The paper calculates that a cloud-service scale network with no over-subscription could be built with commodity switches at 1/14th the cost of a traditional data center Ethernet.

Whoa!

The StorageMojo take
VC and engineering dollars follow high-growth markets. What Google, Amazon and Microsoft want, they get. With the rapid growth of public cloud services the network over-subscription problem will get solved.

Merchant silicon from Broadcom, Intel and Marvell is making a tried-and-true Moore’s Law attack on hardware cost. The protocol stack is tougher, but several open-source industry initiatives are under way with support from major companies. Progress will be slower than hoped, but within 3 years we’ll have a viable stack to build on.

Where does this leave the networking industry? That depends on where you sit.

Cisco will be the biggest loser, because they’ve been the biggest winner with the current model. They may need to pull an IBM and move big into services if they want to stick around. Ironically, Cisco’s UCS product line – which bakes in the tree-structured network – has further motivated broader industry action.

The rest of the industry can go after this emerging market with a lower-GM business model. Not all of them will, but it will be a critical success factor.

The big winner will be storage. Scale-out storage relies on spraying data across multiple racks for maximum availability, utilization and performance. Cheaper, faster, better scale-out networks will only drive storage demand.

For most of us this is an academic problem today. Lightly used systems – such as for backup and archiving – don’t see Amazon’s problems. But in 5 years this will be common even outside the public cloud providers.

Just as IT users have benefited from Google’s push on energy efficiency and much more, they will also benefit from much lower cost and more scalable networks.

Courteous comments welcome, of course. I can’t help but continue to marvel at how dumb Cisco’s UCS has turned out to be. It’s a gift that keeps on giving.

Sometimes in the midst of the endless tweaking needed to maximize storage performance one just wants to say “screw it! Put everything in RAM!” And that’s just what RAMCloud does.

Disk is the new tape, flash the new disk, DRAM the new flash.
RAMCloud is a research paper (pdf) and an open software project. The goal is enterprise-class availability with every bit of active data stored in DRAM, not disk or flash, for maximum performance. It is a key-value object store today, though as pure software that could change.

It’s the brainchild of John Ousterhout, a Stanford prof who invented Tcl back in the 80s at Berkeley.

Isn’t DRAM volatile and costly?
Right on both counts, grasshopper, so RAMCloud isn’t a 1 for 1 disk-style architecture. No Google FS-style triple replication here, or RAID-style erasure coding.

Instead RAMCloud uses buffered logging:

. . . a single copy of each object is stored in DRAM of a primary server and copies are kept on the disks of two or more backup servers; each server acts as both primary and backup. However, the disk copies are not updated synchronously during write operations. Instead, the primary server updates its DRAM and forwards log entries to the backup servers, where they are stored temporarily in DRAM.

Instead of working around crashes – using multiple object copies as scale-out storage does – RAMCloud recovers lost data from the DRAM logs or disk drives to replicate the lost data at high speed. That’s possible because all the log data is in DRAM or spread across many disks.

In a recent paper (Fast Crash Recovery in RAMCloud) (pdf) Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John Ousterhout, and Mendel Rosenblum (co-founder of VMware) go into more detail on this critical feature.

The key elements are:

• Scale. Servers scatter their backup data across all other servers so thousands of disks can serve the recovery.
• Log-structure. Reduces complexity and offers high performance.
• Randomization. Many decisions need to be made in a large cluster. Rather than CPU, time and bandwidth consuming determinism, injecting randomization speeds decisions with less overhead.
• Dynamic tablets. The key-value store tracks resource usage within a single table and ensures that no single partition is too large for fast restores.

DRAM is volatile so the log replication data is spread to other servers on other racks for redundancy before being committed to disk. Still, total system write throughput is limited by the disk write speed, whose limits are a key reason people are moving from disks. Flash drives may help, but other techniques, such as log truncation and sharding make it possible to get good performance from several thousand SATA drives.

How good? The team reports that in a 60 node cluster they recover 35GB in 1.6 seconds. With more nodes larger partitions should be restored even faster. Scale is good.

Lights out!
Power failures wipe all the data in DRAM. The obvious defense is to avoid failures: combine battery backup with diesel generator sets. Power ride-through will handle interruptions into the hundreds of milliseconds.

But who is going to trust that? That’s why future commercial implementations will insist on logging to stable storage, such as the flash SSDs.

They’re getting cheaper fast – faster than DRAM – which will make this a common approach.

Cost
Professor Ousterhout kindly sent a short note about cost, correctly noting that

. . . if you measure cost/operation, DRAM is roughly 100x cheaper than disk, since a disk can only perform about 100-200 operations/second. This is why RAMCloud makes sense for data-intensive applications. . . .

While you and I might find that persuasive, too many enterprises don’t. The deep conservatism of the storage culture – both figuratively and literally – makes cost a good excuse to stay with the tried and true, and easy to explain to CFOs.

The good news for the company I hope he is starting is that the primacy of $/GB is slowly eroding as customers see the system level savings from fast storage. SSD vendors and companies like TMS and Kaminario are breaking trail for RAMCloud.

The StorageMojo take
Make no mistake: RAMCloud is a research project, not a commercial product, years and million$ away from commercial application. But the concept is promising.

Imagine a world where data layout doesn’t matter, where apps are optimized for sub-millisecond storage, where 100 byte I/Os are faster and just as efficient as 8KB I/Os. The architectural implications are huge and would take a decade or more to absorb.

RAMCloud raises the thorny issue of tiering: getting hot data on the hot storage and everything else off to disk. There are OK answers for tiering but nothing insanely great.

RAMCloud shows we’re far from the end of the line in what storage can do. Faster, better, arguably cheaper: 2 out of 3 ain’t bad.

Courteous comments welcome, of course. A shorter version of this post appeared on ZDNet.

One more datapoint and we’ll have a trend: NoSQL databases managing metadata. It’s obvious in retrospect: use a scalable big data tool to handle scale-out metadata. Maybe not a requirement today, but surely will be with even bigger data tomorrow.

Metadata is a fraction of the user data set, but it gets hammered much more. As more metadata is found useful the hammering will get more insistent.

Nutanix
Nutanix, whose CTO and co-founder, Mohit Aron, was a developer of the Google File System, uses MapReduce. Nutanix achieves it scale due to its distributed metadata, masterless architecture – powered by MapReduce jobs that run in the background.

Druva
Druva, a backup company for mobile devices, also uses a NoSQL database to manage storage metadata. They say they’ve found that NoSQL scales over an order of magnitude better than relational in similar applications.

Somebody else
A company that shall remain nameless is porting Hadoop to their backend. The customer won’t be able to access Hadoop for their work – it is strictly for the system’s internal use.

It is a proof of concept so it isn’t a 3rd data point, but they see the potential advantages. Call it data point 2½.

The StorageMojo take
Small advances are the building blocks of disruption. RAID made it possible to build available storage using cheap disks. Consumer adoption of PCs made disks even cheaper. Moore’s Law made RAID controllers cheaper and faster, or faster and more capable.

A virtuous circle of disruption.

The basic architecture of scale-out storage systems – purpose-built software on clustered commodity hardware – has been stable. But this is the beginning of scale-out storage 2.0: taking scale-out technology developed for users and incorporating it into the storage infrastructure itself.

These ideas are bubbling up among the latest startups and among the establishment players. At some point the old RAID architectures will be well and truly broken, able to compete in smaller and smaller niches until the revenue can’t justify more investment.

Of course vendors have been making RAID controllers out of servers for years now, and those servers can run any software they want. But at some point the explicit and implicit assumptions in the old architecture crash into current realities – either in cost, development time, feature completeness or management overhead – and then we move on.

Courteous comments welcome, of course. I learned about Nutanix at the last Tech Field Day “The Independent IT Influencer Event” which paid for my travel expenses to Silicon Valley.

I sat down with Nimble Storage co-founder and VP of engineering Varun Mehta to discuss their architecture – and shoot some video. Varun has been part of several Valley success stories – NetApp, Sun, Data Domain – and has a first hand perspective on disruptive technologies.

Varun and co-founder Umesh Maheshwari – a brilliant architect and a very nice guy – designed the Nimble product that he discusses. Take 4 minutes to learn more about Innovations in Storage Architecture at Nimble Storage:

Or you can see it in HD on YouTube.

The StorageMojo take
The Nimble guys have great technology, but they’ve also put together a compelling value proposition: collapse 3 time-consuming and complex workflows – primary storage, backup and archiving – into 1 appliance. Include all the needed software, price it well, target under-served mid-sized companies and you have a recipe for another Valley success.

The tech trends they’re riding will only get better. But the business trends are in their favor as well. SMB’s today have many TB of data and little staff to manage it – or capital to invest. With Congress ensuring that America operates well below capacity for years to come, the times favor thrifty solutions like Nimble’s.

Courteous comments welcome, of course.
Nimble bought my time for this video, but I made all editorial decisions.

Quantum announced a new deduplication appliance series – the DXi 6701 and 6702 – that claims exceptional scalability. Why? Because it uses technology from Quantum’s StorNext cluster file system.

Scale out
Quantum says the units grow from 8 to 80TB of usable RAID 6 capacity with no subtractions for landing areas, hidden reserves or multiplication. And they say they’re fast: 5.8TB/hr using VTL or OST; 5TB/hr for NAS; all dedup at wire speed.

The only difference between the 2 models is that one has 1Gig Ethernet and the other 10Gig. All the software is included in the price: NAS, OST, VTL, tape support, replication and client side dedup option.

List prices start at $56k. Quantum sells through the channel, so that’s a maximum.

The StorageMojo take
The DXi 670x is a good example of the power and economy of scale-out vs scale-up. The cluster file system technology underlying it enables the 10x capacity expansion with high performance and low-costs.

With a scale-up approach hardware volumes would be lower with hardware and software qual and support costs higher. That Quantum owns the underlying technology makes their job that much easier.

Quantum’s market power is a fraction of EMC’s Data Domain. But the power of their architecture’s advantages in performance, flexibility and cost point to a larger trend.

The problem with Quantum’s marketing is that they only play the price/performance card. Important, no doubt, but by ignoring the fundamental advantages of their scale-out architecture, they let the competition sidestep their long-term problem: they don’t scale.

Winning in the development lab is only part of the battle. Helping customers appreciate – and making competitors react to – the differences, is needed to win in the market.

Courteous comments welcome, of course.

Commenters on the last post – Open source storage array – helped crystallize an idea that’s been lurking for years: comparing disk storage hardware on per-slot price. The Backblaze box, which costs about $50/slot, got a comment that said, in effect, “it doesn’t have the features of a $200/slot box.” Good!

But the comment raised an interesting point: since we all use the same disks from the same few – and soon to be fewer – manufacturers, isn’t the cost of the tin we wrap them in a key metric? Let’s call it PSC – Per Slot Cost.

Some advantages:

• Focus on value-add. We know how many disk slots there are in a storage system. We know how much disks cost. Therefore, the per-slot price tells us what the vendor’s value-add per disk is – or what we’re supposed to think it is.
• Increases pricing contrast. Disk costs are typically 10-15% of the price of a mid-to-high end array. The number of disk slots in those arrays vary, as do individual disk capacities. These variables obscure what the vendor is asking for their value add.
• Cleaner comparisons. As a corollary to the previous point, PSC makes it easier to compare architecturally similar systems – SAS vs SAS, hybrid SSD/SATA systems, RAID 6 systems – whose hardware cost structures should be similar.
• Focus on software value. Since most storage systems – even high-end systems – run on commodity hardware, the biggest price variable is in software. Isn’t that where we should focus?

The cloud storage angle
PSC should be useful for market segmentation. Instead of dumping arrays into entry-level price buckets – such as $75-$100k or $/GB – the PSC should track with the value of the stored data.

Expect to see segments range from Bulk (the Backblaze segment) to Heavy Transactional (traditional big iron) with yet-to-be-named segments between. But the most important use for PSC is in highly-scalable architectures in the public vs private cloud storage arena.

Cloud architectures are distinguished by the fact that the larger they scale, the lower their PSC. This is partly a function of economic necessity – who can afford 2 dozen PB of Symm? – and largely due to their use of software-based object replication instead of RAID.

When your storage is cheap, you can afford triple replication. And when you have massive numbers of boxes – and at least 2 data centers – you can have strong disaster tolerance. So large-scale cloud suppliers have motive and opportunity to reduce PSC.

The private cloud space is where the calculus gets interesting. Many observers dismiss the private cloud concept because they can’t possibly compete with Amazon, Microsoft and Google on scale or cost, including PSC.

The StorageMojo take
There is a private cloud market because there are other issues, such as network latency, and the commercialization of high-scale software such as Hadoop, that make it possible for any focused billion-dollar company to build a competitive cloud infrastructure. The hardware is already a commodity, and many of the improvements that Google 1st pushed, such as more efficient power supplies, are now widely available.

The bigger issue for competitive private clouds is the enterprise IT mindset that lacks the skills to specify and manage them. This is where PSC comes in: it allows CFOs to compare their costs to best-in-breed cloud providers in a simple way.

PSC is just a metric, not the metric. The big guys are optimizing things – like power distribution – that won’t move the needle for smaller players.

But if you use commodity hardware then you should focus on the software. And since every big player is already running on commodity hardware – a Good Thing, BTW – let’s focus on getting software that delivers business value. To the extent that PSC helps decision-makers do that, it will help the industry shift the focus from things like $/GB to a higher-level discussion.

Courteous comments welcome, of course. I just paid $250 per slot for an array with 1 controller, 1 fan and 1 Thunderbolt connection to my 1 desktop. Yes, I could have done better – if I didn’t want Thunderbolt. So PSC doesn’t trump all.

Deduplication has been accepted as an enterprise-class compression technology. Is it time for data compression to be a standard feature of primary storage?

I’ve been doing some work for Nimble Storage a cool Valley startup. Talking to co-founder Varun Mehta, he mentioned that Nimble’s storage/backup/archive appliance does data compression on all data, all the time.

That’s right, primary storage on Nimble’s box is always compressed. Not only that, all their performance numbers are quoted with compressed data.

They aren’t kidding.

Venerable compression
Data compression is one of the oldest computer storage technologies around. Bell Labs mathematician Claude Shannon published A Mathematical Theory of Communication in 1948 which, among other things, laid out the math behind compression.

The ratio of the entropy of a source to the maximum value it could have while still restricted to the same symbols will be called its relative entropy. This is the maximum compression possible when we encode into the same alphabet. One minus the relative entropy is the redundancy. The redundancy of ordinary English, not considering statistical structure over greater distances than about eight letters, is roughly 50%.

In line compression has been part of every enterprise tape drive for decades. The algorithms – Lempel-Ziv was big 20 years ago – have been tuned to a fare-thee-well.

Compression is as thoroughly wrung out as any technology in the data center.

So why don’t we use it everywhere, like Nimble?

Not about capacity
The doubling of capacity from compression is not the big win. The larger benefit is that it more than doubles the internal bandwidth of the array – because bandwidth is more expensive than capacity.

And bandwidth is more important than capacity. As John von Neumann noted in his First Draft of a Report on the EDVAC (pdf):

This result deserves to be noted. It shows in a most striking way where the real difficulty, the main bottleneck, of an automatic very high speed computing device lies: At the memory.

Varun reports that Nimble’s comdec operates at wire speed on a multicore CPU, no ASIC or FPGA required. It must increase latency, but given Nimble’s focus on full stripe writes the increase in bandwidth must more than make up for it.

The StorageMojo take
Since it is possible to perform wire-speed compression/decompression with a commodity CPU, why not everywhere?

Will RAID controllers stumble reconstructing compressed data? Is compressed data more prone to corruption? Is bandwidth so cheap that we don’t need more?

I don’t think so, but I’m open to dissenting opinions. With disk capacity growth slowing comdec everywhere is a good way to increase performance, reduce $/GB and have something new to show customers.

Courteous comments welcome, of course. StorageMojo dove into this 5 years ago in 25x data compression made simple.

A post last month in ACM’s Queue raised a disturbing point around block-level deduplication in flash SSDs: it could hose your file system.

De-dup is a Good Thing, right?
Researchers found that at least 1 Sandforce SSD controller – the SF1200 – does block-level deduplication by default. Many file systems write critical metadata to multiple blocks in case one copy gets corrupted. But what if, unbeknownst to you, your SSD de-duplicates that block, leaving your file system with only 1 copy?

Yup, corruption of 1 block could wipe out your entire file system. And since all the “copies” point to the same corrupted block, there’s no way to recover.

Most Unix superblock-based FSs and ZFS could be pooched by loss of a single block. NTFS also mirrors critical metafile info and could be vulnerable as well.

To be fair, AFAIK no one has reported this failure in the wild, so it is conjecture today. That said, it may have happened to people who didn’t realize what went wrong.

But in the world of storage, if something can happen it will, usually at the worst possible time. Have you seen a total data loss on an otherwise functioning SSD?

The StorageMojo take
I’ve made calls to a number of vendors to get their responses, including Sandforce, Intel, Texas Memory Systems and OCZ. With any luck we’ll soon have a 1st pass on who does what to your data.

Don’t panic: not all SSD controllers do this. Texas Memory Systems controllers don’t, partly because they don’t use MLC flash and partly because minimizing capacity use and maximizing data availability are conflicting goals, and they chose the availability over capacity.

Also note that the SF-1200 is offered as a consumer grade controller. Not clear what Sandforce does with the rest of their line, but their site does repeatedly reference their “DuraWrite” technology which appears to include block-level dedup.

Just last week StorageMojo recommended faster adoption of SSDs in the enterprise – and still does. But this once again underlines the need for mirroring. The sooner we find these issues, the sooner they’ll be fixed.

Watch the comments for vendor info, and I’ll update this post with more info if and when it develops.

Update:Here is the Sandforce response:

In the recent article by David Rosenthal he mentions a conversation with Kirk McKusik and the ZFS team at Sun Microsystems (Oracle). That conversation explains why it is critical that meta data not be lost or corrupted. He goes on to say that “If the stored metadata gets corrupted, the corruption will apply to all copies, so recovery is impossible.”

SandForce employs a feature called DuraWrite which enables flash memory to last longer through innovative patent pending techniques. Although SandForce has not disclosed the specific operation of DuraWrite and its 100% lossless write reduction techniques, the concept of deduplication, compression, and data differencing is certainly related. Through all the years of development and OEM testing with our SSD manufacturers and top tier storage users, there has not been a single reported failure of the DuraWrite engine. There is no more likelihood of DuraWrite loosing data than if it was not present.

We completely agree that any loss of metadata is likely to corrupt access to the underlying data. That is why SandForce created RAISE (Redundant Array of Independent Silicon Elements) and includes it on every SSD that uses a SandForce SSD Processor. All storage devices include ECC protection to minimize the potential that a bit can be lost and corrupt data. Not only do SandForce SSD Processors employ ECC protection enabling an UBER (Uncorrectable Bit Error Rate) of greater than 10^-17, if the ECC engine is unable to correct the bit error RAISE will step in to correct a complete failure of an entire sector, page, or block.

This combination of ECC and RAISE protection provides a resulting UBER of 10^-29 virtually eliminates the probabilities of data corruption. This combined protection is much higher than any other currently shipping SSD or HDD solution we know about. The fact that ZFS stores up to three copies of the metadata and optionally can replicate user data is not an issue. All data stored on a SandForce Driven SSD is viewed critical and protected with the highest level of certainty.

Readers: how does that sound to you?
End update.
Update 2: Oddly enough, the Sandforce web site specifies the SD-1200 controller at

ECC Recovery: Up to 24 bytes correctable per 512-byte sector
Unrecoverable Read Errors: Less than 1 sector per 1016 bits read

which is about where many enterprise disk drives spec’d – and quite a bit less than 10^-29. Hmm-m.
End update 2.

Update 3:
Spoke to James Myers of Intel. He said that no current Intel SSD uses any form of compression, including dedup. He also cautioned against making too much of the risk: after all, you’d have to have an unrecoverable read error AND it would have to be that 1 critical block. Perhaps, he suggested, file systems that do use multiple copies of critical FS metadata could slightly alter the copies to eliminate the possibility of deduplication.
End update 3.

Courteous comments welcome, of course. TMS has been advertising on StorageMojo for a couple of years.

IT pros are always skeptical about new technology. Is it surprising that flash SSD’s are getting the gimlet eye?

The big worry seems to be endurance. Nobody wants to buy an expensive SSD and have it fail after a year on the job.

But IT infrastructures are designed to manage endurance failures. LTO tape, for example, is specified for a few hundred head passes. Yet tape is the paragon of data persistence.

Hard drive failure rates aren’t low enough for any of us to consider storing important data on one without backup. So why are IT pros so skittish about flash SSD’s?

Experience
Or rather, lack of experience. Flash SSDs are evolving rapidly, with new generations arriving every 12 to 18 months.

It takes time for experience with new models to percolate. In the meantime, bad experiences with earlier generation drives continue to circulate.

Vendor secrecy about failure rates and modes doesn’t help. Until the Bianca Schroeder/Google/CMU disk drive studies were released 4 years ago, we had no independent large-scale reliability data.

I hope it won’t take 20 years before we get that information on SSD’s. How about it, vendors?

Reliability
SSD’s may turn out to be more reliable than hard drives but I won’t believe it until I see independent data. The lack of moving parts is a plus but about half the failures and this drives come from the electronics not the spinning bits. SSDs have most of the same electronics.

SSD equivalent of a head disk assembly
Plane failures are a major trouble spot. Each die consists of two planes. These planes are prone to sudden failure, wiping out half the data on a die.

Most chip carriers contain multiple stacked dies, so a plane failure will remove anywhere from a quarter to an eighth of the chip’s total storage. Most flash controllers lay out the data in ways similar to a RAID array to guard against data loss.

What to look for
Since Maxtor’s well-deserved demise we’ve had reasonable parity between disk drives and disk drive vendors. But that is not the case with the still maturing flash drive market.

Storage Newsletter recently published a list of 85 SSD vendors, most of whom none of us have heard of. Many are focused on the embedded systems market, but also because the SSD market barriers to entry are small: buy controller chip; buy flash on the spot market; gen up a PC board and voilà you are in the SSD market.

But flash that ends up on the spot market at rock-bottom prices is often marginal. The big buyers, like Apple, get first dibs on the best.

SSDs made with spot-market flash and a no-name – USB thumb drive? – controller will have a lot more problems. Which is to say that in today’s SSD market brandnames count.

Other things to look for are a guarantee of total write capacity. Another is a statement on the amount of over provisioning the drive has.

Even better: a five-year guarantee such as Seagate popularized with disks and that Intel just started offering on one of its SSD lines.

The StorageMojo take
I have been as skeptical as anyone on SSDs – read some of my earliest posts – but the time for skepticism has passed. Of course, perform careful evals on any new IT product. But the best flash SSD’s are ready for the enterprise today.

And here’s an even more radical conclusion: the best consumer SSD’s are ready for the enterprise as well. Using any SATA drives in your enterprise?

The key: how is the SSD architected into the system? If it is storage tier the data has to be protected just like a RAID array. If it is a cache you have more flexibility – as long as the data is also on disk.

Yes, it’s more difficult to separate the wheat from the chaff in the SSD market today. But there are quality products available today.

Courteous comments welcome, of course.
Started thinking about this is result of the research project I did a few months ago. Leading-edge storage managers with workloads that would benefit enormously by flash SSD’s weren’t seriously evaluating them today. Big surprise. What do you think?

I was surprised by the number of questions at last week’s webinar – many more than we could get to – so I’m answering a few here.

Performance
Q: Can Robin talk about performance and how does flash help solve I/O bottleneck?

NAND flash is very good at random reads, and a good SSD can handle thousands per second, compared to a disk drive’s 150-500 (for a high-end drive). That’s one reason arrays are popular: they provide higher random I/O performance because multiple heads are seeking. But that’s also why capacity utilization is so low: the disks come with more capacity than most applications use.

So not only are you buying multiples of the most expensive disks, but then you only use a fraction of their capacity. This is why flash SSDs are predicted to kill the high-end drives even though SSDs cost much more per GB: a single SSD can eliminate 6-10 hard drives. That is a major cost saving.

Q: So the low cost assumes that the cache is read only, otherwise it needs to be RAID? That comes off as misleading.

While flash SSDs are much better at random reads than they are random writes, they still beat several high-end disks at writes.

Since most workloads are 80%-95% reads, an SSD that can handle 1,000 writes per second can handle a lot of work. Disks are still the most cost-effective solution for large sequential workloads because their performance is close to SSDs and they are so much cheaper.

Reliability
Q: What are Robin’s thoughts on the reliability of SSDs? We have seen failure rates of over 10% on drives less than two months old.

Flash SSD reliability today is all over the map. As flash SSD technology matures, I’d expect to see drive reliability rates converge. 5 years ago disk reliability was fairly similar with the glaring exception of Maxtor.

That said, it’s useful to recognize that there’s a lot more design and sourcing variability in SSDs. If someone uses the cheapest parts – and there are plenty available – they can offer good specs but highly variable reliability.

If they leave out too much redundancy they’ll have a cost advantage but will be more vulnerable to chip and plane failures. The market will eventually settle on similar specs for each application, but we’re years away from that.

Q: You mentioned 10,000 writes and failure can begin, what is that in years?

Like so many storage specs, that 10k write spec for MLC flash is a statistical one that can be improved upon by more robust ECC, as this chart from SNIA shows:

But the most important way to improve upon it is by increasing the capacity of the SSD. Double the size of the SSD and you double the total write capacity.

As to what that is in years, the industry is still figuring out how to spec that. The best vendor spec I’ve seen so far has been from Intel – 5 years at 20 GB of writes per day.

Courteous comments welcome, of course. I enjoyed the webinar – a new experience for me – and not just because I got paid. The crew at Nimble was a pleasure to work with. Here’s a link to the webinar.

JIm Gray’s comment that disk is the new tape is truer today than it was 8 years ago. We’ve been adding caches, striping disks, modifying applications and performing other unnatural acts to both reduce and accommodate random reads and writes to disk.

Flash changes the calculus of 20 years of storage engineering. Flash gives us abundant random reads – something hard drives are poor at – and reasonable random writes to whatever hot data we choose.

In a feverish burst of design and investment we’ve tried flash everywhere in the storage stack: disks; PCI cards; motherboards; controllers; built-in tiering; and appliances. These products have been focused on enterprise datacenters or very targeted applications where the cost of flash was justifiable.

But clarity is emerging. It isn’t so much where you put the flash as what you ask the flash to do. There are three requirements:

• Valuable data. Flash is an order of magnitude more costly than disk.
• Often accessed. If not, leave it on disk.
• Enables new functionality and/or lowers cost. If it doesn’t, why bother?

The buyer’s burden
These requirements frame a basic point: optimizing for flash requires a systems level approach. Adding flash can make current architectures go faster, but that isn’t the big win.

Buyers looking for an economic edge must make a cognitive leap: the old ways are no longer best. Flash enables efficiencies and capabilities in smaller systems that only costly enterprise gear had a few years ago.

Tiering
Tiered flash solutions are the most common approach today. Tiering software has improved in recent years, making the movement of data between flash and disk safe, fast and granular.

We’ve started to at least see interest in the midsize enterprise, like the EqualLogic hybrid SAS/SSD array in VDI deployments.

Metadata and cache
The best fit for flash today is metadata and caching. These best meet the requirements for value, access and functionality.

Once metadata is freed from disk constraints we can combine it with caching to build high-performance systems on commodity hardware. The win for innovators is to design new metadata structures and caching algorithms for flash.

They can design the (write) data layouts to best take advantage of the physics of disk and flash, such as with Nimble Storage’s CASL architecture, which combines a large flash cache with full-stripe writes, is one example.

Flash is also an important enabler for low-cost de-duplication because it’s cheaper to keep block metadata – fingerprints or hash codes – in flash than it is in RAM. Some vendors are encouraging the use of de-duplicated storage for midrange primary storage, enabled by flash indexes or caches that make it feasible to reconstruct files on-the-fly.

The StorageMojo take
Shaking off the effects of 50 years of disk-based limitations isn’t easy. Our disk-based orthodoxy is ingrained in architectures and our thinking.

But buyers face a difficult job: evaluating architectures and algorithms to choose products for eval. A shortcut: look for architectures that collapse existing storage stovepipes to reduce cost, total data stored and operational complexity. The three are related and offer the big wins.

In the last 10 years raw disk capacity cost has dropped to less than a 10th of what they were, but the cost of traditional storage systems haven’t. The culprits: operating costs; storage network infrastructure costs; and capacity requirements that have risen faster than management productivity.

The flood of data continues to rise, but cost and complexity doesn’t have to rise with it. We can – and are – doing better.

Courteous comments welcome, of course. I’ve been working with Nimble Storage lately and like what they’ve done.