Maximizing value for time spent is a difficult calculus. Especially when it the goal is perspective rather than information.

Efficient use of your time will be why the Solid State Storage Symposium works. Starting at 10am at the Doubletree San Jose Airport and going until 4pm, you’ll get to hear about – and question – 9 different SSS-using vendors.

The crack StorageMojo research team will be there in force, delivering a keynote that looks at the next 10 years in storage technology and architecture. Not everyone will agree with every point – that would be no fun at all – but there is no doubt that the next 10 years will see more change than the last 30.

This is a tech-heavy symposium with many of the smartest people in leading edge architectures speaking, including Dave Wright of SolidFire, Suresh Vasudevan of Nimble Storage, Jonathan Goldick of Violin Memory and Jered Floyd of Permabit among others. The panels intend to elicit critical differences, not dry recitations of marketing hype.

It should be good.

The StorageMojo take
One of the reasons big companies dominate is that it is so hard for small companies to get their unique value across to users. This is an opportunity for the up-and-comers to explain what it is they do, and for tech-savvy users to judge for themselves the goodness these vendors are offering.

Hope to see you there!

Courteous comments welcome, of course. I’ve done work for Nimble and Violin.

Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

Actually, StorageMojo is wrong, wrong, wrong, wrong and wrong. Umesh Maheshwari of Nimble Storage wrote a detailed and thoughtful response to the StorageMojo post Are SSD-based arrays a bad idea?

The StorageMojo take
Umesh makes good points, but perhaps due to Nimble’s hybrid disk/SSD architecture some seem to miss the mark. What’s missing is ample consideration of what the alternative to an SSD might be, a problem Nimble didn’t have because SSDs work fine, technically and economically, for their hybrid system.

For example, on the issue of reliability Umesh says:

Reliability. There are good reasons to be able to replace failed flash devices similar to how hard disks can be hot swapped. The raw bit error rate (RBER) of flash is actually worse than that of hard disks, and it gets worse as blocks are rewritten. It is also getting worse as manufacturers are moving to increase density. (See this paper from FAST 2012: The Bleak Future of NAND Flash and a related blog post.)

This is correct, but based on the Google/Bianca Schroeder research, the StorageMojo point is that the disk electronics – apart from the head/media pieces – are a major – 40%-50% – source of HDD/SSD failures. The flash controller has to handle the RBER and declining flash performance, but why add the other HDD bits that account for a substantial percentage of drive failures?

I could niggle about Umesh’s other points, but what fun is that? StorageMojo readers are encouraged to check out Umesh’s post and make up their own minds.

Courteous comments welcome, of course. I recently did a nifty video white paper for Nimble, which is a great intro to their innovative architecture. Check it out.

More is coming on SSDs RSN, but in the meantime there is the following piece from Virsto’s Eric Burgener on HA considerations for SSDs. Virsto is a software company focused on making VIRtual STOrage for VMware and HyperV much more functional than the physical kind.

Thus Eric’s response has a very particular POV: what is needed to use SSDs in a virtual environment to ensure high availability – from a company that DOESN’T sell HA hardware. One key point: virtual server environments are much more write intensive than most enterprise apps, so using SSDs as a cache is a losing strategy.

If that is intriguing, read on!

In Thinking About SSD, You Can’t Leave HA Considerations Out

In the “host vs array-based SSD” discussion as it pertains to enterprise accounts, the need for HA must play a critical role. This is true whether you’re working with physical or virtual environments. Any committed data that is not sitting on a shared, non-volatile, external storage device and accessible by at least one other node cannot be recovered until that failed node (on which it resides locally) is brought back up.

There are technical ways to solve this using synchronous replication technologies, but that’s an extra credit project you do yourself for now – as of yet, that hasn’t been built into any host-based SSD products. The reality today is that using host-based SSD precludes the use of HA (but not necessarily things like vMotion, which is NOT HA).

This was touched on in some other posts, but I think it’s an increasingly critical issue in virtual computing environments that may have been a bit downplayed in other comments. If you’re either thinking about moving production server workloads to VMs or have already got them there, HA is critical for a high percentage of workloads.

I can’t imagine an enterprise customer spec’ing out a production virtual server environment without asking about HA. True, there are workloads that don’t require it, but most do.

And it’s not just virtual server environments. We’re running into an increasing number of VDI environments where they want to enable HA for at least a small percentage of the desktops – usually executive desktops. HA isn’t a deal breaker for VDI like it is for “VSI” (virtual server infrastructure), but there are clearly use cases in VDI where you want and/or need it.

Today SSD is pretty much only used as a cache, regardless of where its deployed. And to provide a given level of performance speedup, caches generally have been sized at somewhere around 2% – 4% of the primary data store (it varies by application and exactly what you’re trying to speed up).

In virtual environments, write performance is much more critical because it tends to comprise a much higher percentage of the read/write workload – in VSI environments its not uncommon to see 50% reads/50% writes, and in VDI environments we’ve seen 70% write environments. Unless you’re using a write back cache (with all the attendant additional expense associated with that), you’re not going to get any write performance speedup from the conventional cache architectures, just read.

But now think about what a log architecture, applied at the storage layer, could add. Circular logs that are continuously draining (asynchronously) as they are filling need very little storage capacity to speed up ALL writes for ALL VMs ALL the time. In our experience, you need a log of about 10GB in size for each heavily loaded physical host.

Think about what that could mean for a 16 host environment with 20TB. You could get away with 2-4 200GB enterprise flash drives instead of the 10-12 that you might otherwise deploy in a 20TB environment. If you have a “linked clone” type snapshot technology combined with storage tiering, you could take the extra SSD capacity and create a tier 0 for critical VMs that need very high read performance, like for example the golden masters in a VDI environment or common templates you use to create your server VMs.

This covers both needs – read and write performance – using a lot less storage. That means pretty much the same performance you’d get with the more expensive configuration with more SSDs for a lot less money. If you want to use SSD efficiently, a log architecture is a great idea. And if the logs are placed in shared, non-volatile, external storage (like the SSDs hosted in a SAN array or SAN-based SSD appliance), you can fully support HA.

Host-based SSD cards are closer to the physical host so theoretically they’ll provide more performance speedup, but given Amdahl’s law, how much of that can you really use? Array-based SSD will still get you past storage latencies as your critical bottleneck, and if they’re implemented using a log based architecture you’ll get HA and large write performance speedups as well using a lot less of it.

The StorageMojo take
Write-through caches avoid a lot of sticky update synchronization problems, but as Eric notes they aren’t the best choice in write-intensive environments. And HA adds to the requirements: the cache must be network accessible.

But his larger point bears repeating: SSDs are wonderful for handling metadata. And as we move to object storage and more metadata that capability will become even more valuable.

Courteous comments welcome, of course. I think Virsto’s architecture is smart and fixes some real problems with VMware and vMotion.

StorageMojo offered its soapbox to any vendors willing to weigh in on the question of whether enterprise arrays should be built from flash SSDs or not. Ed Lee, architect at Tintri, formerly of Data Domain and a Berkeley Ph.D, elected to respond. It is a long piece but rich in insight.

Tintri produces hybrid disk/flash SSD appliances optimized for virtual environments, not Symm-killers. They use SSDs in their products, as do other folks like Nimble Storage.

No money changed hands between Tintri and StorageMojo or related entities. My accountant is weeping in the next room.

Begin Tintri’s response:

Outside the SSD Box: More than Faster Disk
Robin Harris of Storage Mojo in his recent article, “Are SSD-based arrays a bad idea? and Matt Kixmoeller of Pure in his response, The SSD is Key to Economic Flash Arrays, present interesting perspectives on whether or not SSDs are the best technology for building flash-based arrays. Robin argues that by rethinking how flash can be packaged outside the SSD box, you can achieve better performance, reliability, cost and flexibility. And these observations are supported by the experience of existing flash-based storage vendors who have developed their own custom flash modules and packaging. Matt argues that SSDs provide an industry-standard product that requires less investment to leverage, better economies of scale, and rapid improvement in technology. These are also very valid points, especially for startups with limited time and capital.

Latency
Taking latency as a point for comparison, flash-based storage vendors using custom packaging often quote IO latencies in the tens of microseconds versus SSD latencies of low hundreds of microseconds. While this is a notable difference, software and interfaces can also add overhead and the final latency seen at the subsystem level may differ by only a factor of two to four. Server-side flash products can avoid more of the software and interface overhead and provide better latencies – but may require rewriting applications to capitalize on this advantage. Keep in mind that hard disk latencies can easily reach tens of milliseconds under even moderate load. ALL of these flash-based products have latencies that are hundreds of times faster than disk.

In short, most of the performance improvement comes from simply replacing hard disk with some form of flash. This immediately shifts the performance bottleneck from storage to some other component in your system. As a result, you won’t be able to take full advantage of flash performance without also optimizing the performance of the rest of your infrastructure, and ultimately rewriting your applications as well.

The above phenomenon explains why replacing your hard disk with flash often speeds up your applications by only a factor of two to three rather than ten or a hundred. Congratulations! You’ve just moved the bottleneck from storage to some other component of your system. By Amdahl’s Law, further improving only storage performance has diminishing returns. So while custom packaging does provide significant advantages in latency, most applications are unlikely to benefit until the rest of the computing ecosystem is optimized to take full advantage of flash.

To take a closer look at SSD latencies, I ran the following simple experiment:
1) Erase an MLC SSD so that no logical blocks were actually mapped to flash, and then issue small random reads.
2) Overwrite the entire SSD so that all logical blocks are mapped, and issue the same small random reads in step 1.

The idea here is to measure the software and protocol overheads of accessing flash packaged as SSD separately from accessing the data on the SSD. Reads with no blocks mapped had latencies of around 70us, while the reads with all blocks mapped had latencies of 250us. In this case only a fraction of the overall IO latency was due to SW and protocol overhead, indicating that SSDs may still have significant room for improving latency.

Form factor
Another important issue discussed by both Robin and Matt is the relative cost of flash packaged in SSD versus non-SSD form factors. Robin argues that an SSD costs significantly more $/GB than the underlying flash while Matt argues that non-SSD packaging is expensive to develop, and SSDs provide useful flash management functions as well as hot-swap capability. It’s certainly true that developing custom packaging has a high up front cost, although this is likely balanced by lower unit costs. But as Robin points out, there are also standard packaging options available for non-SSD form factor flash, which may make custom packaging for non-SSD flash unnecessary.

A very important point to keep in mind when thinking about commercially available SSD vs. non-SSD form factors is that SSDs are designed as a substitute for disk, while non-SSD form factors are often designed as substitutes for memory. This means that SSDs focus primarily on reducing $/GB (its greatest weakness vs. disk), while non-SSDs focus on reducing $/IOPS (its greatest weakness vs. DRAM). This explains why SSD is currently much cheaper on a $/GB basis than PCIe flash, while PCIe flash designed as memory expansion is cheaper on a $/IOPS basis than SSD. This is not to say that you can’t build a non-SSD form factor that has lower $/GB than SSD, just that the primary applications for these non-SSD form factors today is usually not as a replacement for disk.

Whether flash in SSD versus non-SSD form factors is better for use in storage subsystems in the long run primarily depends on the relative volumes of these products, and the feature and price sensitivity of the applications these products serve. At this point the ‘winning’ form-factor seems hard to predict. So as a flash subsystem vendor, it seems desirable to keep your options open and ensure that your technology will work well with a variety of packaging options.

More than just a faster disk
But flash is about more than just performance and packing. Flash enables much more than just a faster, denser replacement for disk. With flash, we can finally remove a key mechanical barrier to scaling not only storage systems, but computing systems in general. Going forward, CPU, network and storage can now all scale with improvements in semiconductor technology. When transistors replaced vacuum tubes, we got more than just compact radios; we got simpler, more powerful computing systems. Similarly, flash is a catalyst that will enable far greater levels of automation and functionality for storage and computing systems than is possible today.

I tend to think of the value of new technology as the product of its simplicity times the functionality it offers. It’s clear why functionality is important, but why is simplicity so important? Technology that is simple to use will be used more often, to solve more problems, in less time. As a result, simplicity has a compounding effect on value:

Value = Simplicity * Functionality

How does one measure simplicity? One way is to list the basic steps it takes to perform a task and how long each step takes. One to three is good, four to six is manageable, and anything resembling a twelve step program will likely require written directions and a significant amount of focus. Note that in assessing the simplicity and functionality of a technology, one must do it in the context of the job that needs to be done. For example, a chainsaw has great features for cutting down trees but not for giving haircuts.

A common problem with many general purpose storage products when applied to applications such as virtualization is that they require executing long lists of steps to get anything done – and most of the features are not directly applicable to virtualization. Paradoxically, many of the features that try to make these products better suited to the application end up making the products more complex – resulting in little improvement in overall value. Kind of like adding too many tools to a Swiss army knife until you have so many that the attachments start to stick and rub against each other.

Flash as a catalyst
Flash eliminates a key mechanical barrier to scaling computing systems and is 400 times faster than disk. To keep things in perspective, the speed of sound is “only” 250 times faster than walking! If I could get to work at supersonic speeds, I would no doubt save a lot of time each year. But would I do no more which such an ability? Similarly, is flash just a faster replacement for disk? Will it make no significant difference in the way storage is managed and used? We obviously don’t think so. Flash will greatly increase the value of storage by improving both the simplicity and functionality of enterprise storage products. But these gains will not come easily or without their own set of problems.

An obvious way flash promotes simplicity is by eliminating performance bottlenecks, but as flash enables more dense storage systems many of those gains will be converted to problems in quality-of-service. A more significant way flash promotes value is by providing a better building block for constructing storage systems: flash promotes simplicity by enabling higher levels of automation and allows the implementation of more powerful functionality.

Flash will fragment the enterprise storage market. The general purpose storage systems of today will be supplanted by new flash-based products that are far simpler and more powerful for the specific application areas that they target. This will amplify the simplicity and power that flash already makes possible, and further accelerate the fragmentation of the storage market. This is precisely what happened in the 1980’s when advances in networking technology caused a shift from centralized computing to networked computing – and in the process fragmented the direct attached storage market into ones based on networked storage technology. Over time, the networked storage markets consolidated into the current general purpose storage market dominated by a few major vendors. And so the cycle is repeating itself.

We are at the start of a new technological shift. A shift that is made possible by flash and one that will disrupt the existing enterprise storage market. Just as transistors enabled new products such as personal computers and smart phones, flash will enable simple, intelligent and fast enterprise storage systems. In turn, this will lead to much higher value for end users, but only if we think outside the storage box and treat flash as more than just a faster, denser disk.

The StorageMojo take
For the record the original post wasn’t looking at hybrid solutions, although it is obvious that SSDs can help legacy designs stay competitive without replacing all disks for a few years. For folks like Tintri and Nimble who want to speed up disk storage to stay affordable SSDs make sense. Why engineer a small part of your system when an off-the-shelf solution will suffice?

But for high end transactional SAN storage I still don’t see how SSDs are the right way to go. But I’m expecting more responses, so stay tuned.

Courteous comments welcome, of course. I’m working on a post that reflects directly on Ed’s comment about SSD latency. You’ll like it.

Pure’s Matt Kixmoeller saw the Are SSD-based arrays a bad idea post and, unsurprisingly, responded. The SSD is Key to Economic Flash Arrays is a good post and I urge interested readers to check it out.

Pure has a stellar team with deep experience. Their views are worth considering.

As Matt notes:

This post caught our eye for an obvious reason: Pure Storage did start “fresh” to build an all-flash enterprise storage array, and we did decide to use the SSD form factor, after quite exhaustive looks at all the other options. Quite simply, we found that SSDs are the most efficient and economic building blocks from which to build a flash array. Let’s explore why.

After dismissing disk arrays that add flash drives – as I do – Matt focuses on (1) all flash appliances built from raw NAND and (2) flash arrays using flash SSDs.

SSDs are most efficient
Matt argues that SSD-based arrays have 3 key advantages:

• Economics. SSDs are a commodity product that raw flash arrays will have a hard time out-engineering.
• Flash controller complexity. Matt notes, correctly, that the flash controller is at the heart of argument. Better to use a controller that goes into millions of SSDs or one purpose-built for a single vendor’s array? How will the single vendor be able to keep up?
• Servicability. Pure’s use of SSDs enables them to offer a familiar hot-swap experience that higher density designs may not offer. Futhermore, Pure’s data reduction features increase effective density to rival raw flash designs.

In conclusion, Matt makes a couple of more points. First, that SSD form factors will become much more compact, such as Apple’s DIMM-like mini-SATA SSD used in the MacBook Air. Second, that the proof is in the pudding: Pure, he says, has “. . . delivered with break-through performance, at a cost below traditional spinning disk.”

The StorageMojo take
How does Matt’s response stack up to the criteria in the original post? Not that there’s anything magic about them, but . . . .

• Latency. No response, which doesn’t mean they’re worse.
• SSD bandwidth. No response, but to be fair with enough SSDs you should be able saturate 16Gb Fibre Channel.
• Reliability. No direct response. Instead a focus on servicability. More on that below.
• Cost. Says Pure is cost-effective using their data reduction technology.
• Flexibility. This is the heart of Matt’s argument: due to the commodity volume of the flash controllers flash SSDs will evolve faster – in functionality and cost – than any proprietary solution could. Proprietary flash controllers, he says, will be boat anchors for flash array vendors and are likely to end up controlled by flash manufacturers.

Servicability is an interesting response to the question of reliability. After all, the reason hot swap is important for some components but not others is because they either a)fail often – individually or in aggregate – b)failure compromises the product or c)online expansion, upgrading or reconfiguation is desirable.

Power supplies are routinely hot swappable because they have the lowest MTBF of any major system component. Disks are hot swappable because they come in multiples that reduce their aggregate MTBF while their standardized design makes hot swap cheap. I/O cards are often hot swappable because they are critical and needs change.

SSDs should be hot swappable because their failure rates are at best about half that of disks. But DIMMs, another critical component, especially if you invest in high-capacity ones, aren’t, because they rarely fail.

While I’m not aware of any non-SSD enterprise array vendor whose arrays don’t include hot swap components – love to be educated – which is more important: a short mean time to repair (MTTR) or a long mean time between failures (MTBF)? Because that is the argument about servicability.

I’d like to publish responses from vendors who feel strongly about this issue. Not in the comments, but as a blog post. Any takers?

Courteous comments welcome, of course. I was so impressed with the Pure Storage team that I signed a rare NDA with them last spring to get briefed, the first of 2 visits to their Castro street HQ.

Another StorageMojo Best paper, The Bleak Future of NAND Flash Memory, presented at this year’s FAST ’12 conference, quantifies flash’s declining reliability, endurance, and performance as density increases.

Researchers Laura M. Grupp and Steven Swanson from the UCSD Non-volatile Systems Lab and John D. Davis of Microsoft Research collected data from 45 flash chips from 6 manufacturers. Using that empirical data they predict the performance and cost characteristics of future SSDs.

Faster better cheaper or slower worse cheaper?
While NAND flash is produced with semiconductor processes, smaller feature sizes don’t lead to faster performance or greater reliability. As NAND features shrink, so do the number of trapped electrons that store information.

Figures of merit
The research found that performance, program/erase endurance, energy efficiency, and data retention time all got worse with feature shrink.

Based on past performance, the team derived equations to describe how changes in feature size have affected key specs. They looked at SLC, MLC and TLC and feature sizes scaled from 72 nm to 6.5 nm (the consensus smallest feature size published in the International Technology Roadmap for Semiconductors (ITRS0), and assumed a fixed silicon budget for flash storage.

Key results

• Latency. MLC write latency will double over time. Triple-level cell writes will grow to over 2.5MS, noticably reducing its performance advantage over disk writes.
• Bandwidth. Small – 512B – read bandwidth and all writes decline by up to 50% over time. The impact is greatest on high-performance SLC flash.
• IOPS. MLC flash I/O rates will drop almost in half.

Flash may be the new disk in a few years.

The StorageMojo take
One important qualifier is that for the purposes of their modeling the team constrained the number of chips in the hypothetical future devices whose performance they predicted. While fine for isolating the impact of future chip shrinks, it ignores the potential of much greater parallelism for managing these changes.

Bandwidth drops by half? Double the number of chips.

But if something can’t go on forever, it won’t. NAND flash will soon enter an end-of-life crisis for computer applications that need performance. That’s why ReRAM (resistance RAM) looks to be a good bet for replacing computer flash – not mobile device flash – over the next decade.

Courteous comments welcome, of course. A version of this post was published on ZDNet last week.

FAST – File and Storage Technology – is a must-see conference for StorageMojo, and I’ll be reviewing several Best Papers from FAST ’12 . While most emerging technology is developed in private company labs, FAST is where much of the first publicly available research is published.

Case in point, a StorageMojo Best Paper of FAST ’12: Optimizing NAND Flash-Based SSDs via Retention Relaxation by Ren-Shuo Liu and Chia-Lin Yang of National Taiwan University, and Wei Wu of Intel. NAND engineers have known for years that it is possible to speed up writes by allowing for shorter retention, but this paper quantifies the process.

Data retention was a theme of several papers. Disk drives don’t care if an update needs to last a minute or a year, but flash does.

NAND retention
NAND flash writes are spec’d – by JEDEC – for one year of retention. But relaxing that retention requirement can be beneficial.

• Speed. Writes can be 1.8 to 5.7x faster, depending on how long the data is to be kept.
• SSD architecture. The need for overprovisioning and other choices is a direct result of incoming data rates and flash write speeds. Faster writes might also mean allow less aggressive garbage collection.
• ECC. As feature sizes shrink and NAND cells get flakier, the ECC overhead required to achieve a year’s retention grows. Single error correcting codes used to suffice. Now we need 24-error correcting codes and the arms race continues.

These advantages are meaningless if most writes need to be retained for more than, say, 2 weeks. The authors looked at a number workload traces and found that for all but one of them, at least 50% of the writes were retained for 1 week or less. For active enterprise workloads the percentage is likely to over 75%.

What happens when the time is up?
The authors propose that the Flash Translation Layer keep track of how long each block remains unchanged. When – and if – it reaches the threshold, a background process rewrites the data for the standard 1 year retention.

It is feasible to differentiate between host writes and background writes – garbage collection, for example – and to write them differently. Long-term writes would get improved ECC, while host writes would avoid the costly ECC encoding required.

Yes, there is overhead in managing the fast blocks and rewriting long-term data. But the added performance appears to make that a small price to pay.

The StorageMojo take
The paper presents a strong case for relaxing retention requirements to improve performance. As future generations of flash become less reliable and slower we’ll need this and other techniques to improve – or at least maintain – performance.

Many performance enhancement schemes require unrealistic levels of intelligence about application or system behavior to be effective. But this is within the realm of practical implementation.

The retention issue is a fair example of being handed a lemon and making lemonade. Or offering another degree of freedom to system architects.

In fact, some vendors are already exploring this possibility. If it extends the useful life of flash for a few years it will be well worth the engineering effort.

Courteous comments welcome, of course. A somewhat analogous process for disks is the concept of shingle writes, an area UCSC has been working in. Will disk vendors pick it up?

If SSDs are so great, shouldn’t we see the results in TPC-C benchmarks? They are, and we do.

But there are some surprises.

Cost
Looking at the TPC-C top 10 performance results showed the dramatic impact SSDs have had on the cost per thousand transactions (tpmC).

• There are no top-10 disk-only results after 2009.
• The most expensive top-10 SSD result is some 15% cheaper than the least expensive disk-based result – and the other SSD results are much less.
• No top-10 results posted during 2009 – the depth of the great recession.

Capacity
The conventional wisdom has it that disks must be way over-configured to get enough IOPS. You’d expect to see disk solutions have a lot more capacity than SSD solutions in top-10 results.

But we don’t:

The highest capacity – 1760 TB – is for an Oracle SSD-based solution. Yet the lowest capacity solution – 83 TB – is also SSD-based and is also the cheapest per tpmC.

Are we seeing issues with the rest of the infrastructure?

The StorageMojo take
I’ll be taking a deeper dive into the data, but perceptions may be at odds with what this limited set of performance focused benchmarks is showing us.

Readers: what do you think?

Courteous comments welcome, of course. Events beyond my control have reduced StorageMojo’s usual posting frequency. Hope to get things back to normal over the next several weeks.

EMC’s Chuck Hollis blogged about The Vendor Beating a couple of months ago. The unspoken question in the post is “how do we understand what customers are telling us?”

He writes

As an employee of a large IT vendor, I’ve been at the receiving end of a reasonable number of vendor beatings.

Occasionally it’s richly deserved. But, sometimes, it’s masking a deeper set of issues that have very little to do any vendor whatsoever.

Unhappy customers, like unhappy families, are all unhappy in their own way. This customer appeared to be overstaffed, under-skilled and poorly managed.

Interpretation
Interpreting customer complaints and behavior is hard. When companies can’t decipher what customers want – which is usually what the company isn’t selling – it is easy and dangerous to tune them out.

Customers can tell you things about your company and products that you can’t directly discover for yourself, but what customers say may be different from what they think. And both are influenced by the customer’s context, which can include company politics, prior vendor experiences, knowledge deficits and employee level.

Diagnosis
Steve Jobs once said that customers don’t know what they want until you show it to them. Customers know what would improve the current product in the current use case, but they can’t imagine bringing multiple novel technologies to bear on a much broader problem.

Tablet computers flopped for years until the iPad crystalized the market. Everyone saw the tablet problems: thick; heavy; slow; clunky UI; poor battery life; and, thanks to low volumes, cost. Incremental improvements – faster processors, more RAM, larger disks – didn’t help.

Tablets required a deep rethinking and application of several novel technologies – flash, gestures, CNC case milling, an app store and an energy-efficient OS – to create a compelling user experience.

The iPad illustrates the problem of listening to customers: they described symptoms and suggest fixes, but couldn’t articulate the underlying problem: how the use case differs from desktop and notebook PCs. That requires an act of imagination, not transcription.

The StorageMojo take
In Chuck’s post an EMC presales engineer identified the root cause of the customer’s pain:

. . . the database environment had grown willy-nilly over the years — it wasn’t laid out well, the queries weren’t particularly well written, and so on.

Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue than just storage performance.

But the larger question is: with high-speed and high-capacity SSDs, why isn’t this customer moving to an infrastructure that doesn’t need this fancy tuning? EMC can’t manage the fight between DBAs and storage admins, but they could be making it less contentious.

From within the EMC ecosystem the solution is clear: more training, professional services and faster gear. But from the outside the question is: who is building “it just works” high performance storage?

Courteous comments welcome, of course. I admire Tucci’s innovative EMC business model: outbid everyone else for chasm-crossing companies; give them global distribution and support; and watch the bucks roll in. It may not be innovative technically but it is innovative.

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

Sometimes in the midst of the endless tweaking needed to maximize storage performance one just wants to say “screw it! Put everything in RAM!” And that’s just what RAMCloud does.

Disk is the new tape, flash the new disk, DRAM the new flash.
RAMCloud is a research paper (pdf) and an open software project. The goal is enterprise-class availability with every bit of active data stored in DRAM, not disk or flash, for maximum performance. It is a key-value object store today, though as pure software that could change.

It’s the brainchild of John Ousterhout, a Stanford prof who invented Tcl back in the 80s at Berkeley.

Isn’t DRAM volatile and costly?
Right on both counts, grasshopper, so RAMCloud isn’t a 1 for 1 disk-style architecture. No Google FS-style triple replication here, or RAID-style erasure coding.

Instead RAMCloud uses buffered logging:

. . . a single copy of each object is stored in DRAM of a primary server and copies are kept on the disks of two or more backup servers; each server acts as both primary and backup. However, the disk copies are not updated synchronously during write operations. Instead, the primary server updates its DRAM and forwards log entries to the backup servers, where they are stored temporarily in DRAM.

Instead of working around crashes – using multiple object copies as scale-out storage does – RAMCloud recovers lost data from the DRAM logs or disk drives to replicate the lost data at high speed. That’s possible because all the log data is in DRAM or spread across many disks.

In a recent paper (Fast Crash Recovery in RAMCloud) (pdf) Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John Ousterhout, and Mendel Rosenblum (co-founder of VMware) go into more detail on this critical feature.

The key elements are:

• Scale. Servers scatter their backup data across all other servers so thousands of disks can serve the recovery.
• Log-structure. Reduces complexity and offers high performance.
• Randomization. Many decisions need to be made in a large cluster. Rather than CPU, time and bandwidth consuming determinism, injecting randomization speeds decisions with less overhead.
• Dynamic tablets. The key-value store tracks resource usage within a single table and ensures that no single partition is too large for fast restores.

DRAM is volatile so the log replication data is spread to other servers on other racks for redundancy before being committed to disk. Still, total system write throughput is limited by the disk write speed, whose limits are a key reason people are moving from disks. Flash drives may help, but other techniques, such as log truncation and sharding make it possible to get good performance from several thousand SATA drives.

How good? The team reports that in a 60 node cluster they recover 35GB in 1.6 seconds. With more nodes larger partitions should be restored even faster. Scale is good.

Lights out!
Power failures wipe all the data in DRAM. The obvious defense is to avoid failures: combine battery backup with diesel generator sets. Power ride-through will handle interruptions into the hundreds of milliseconds.

But who is going to trust that? That’s why future commercial implementations will insist on logging to stable storage, such as the flash SSDs.

They’re getting cheaper fast – faster than DRAM – which will make this a common approach.

Cost
Professor Ousterhout kindly sent a short note about cost, correctly noting that

. . . if you measure cost/operation, DRAM is roughly 100x cheaper than disk, since a disk can only perform about 100-200 operations/second. This is why RAMCloud makes sense for data-intensive applications. . . .

While you and I might find that persuasive, too many enterprises don’t. The deep conservatism of the storage culture – both figuratively and literally – makes cost a good excuse to stay with the tried and true, and easy to explain to CFOs.

The good news for the company I hope he is starting is that the primacy of $/GB is slowly eroding as customers see the system level savings from fast storage. SSD vendors and companies like TMS and Kaminario are breaking trail for RAMCloud.

The StorageMojo take
Make no mistake: RAMCloud is a research project, not a commercial product, years and million$ away from commercial application. But the concept is promising.

Imagine a world where data layout doesn’t matter, where apps are optimized for sub-millisecond storage, where 100 byte I/Os are faster and just as efficient as 8KB I/Os. The architectural implications are huge and would take a decade or more to absorb.

RAMCloud raises the thorny issue of tiering: getting hot data on the hot storage and everything else off to disk. There are OK answers for tiering but nothing insanely great.

RAMCloud shows we’re far from the end of the line in what storage can do. Faster, better, arguably cheaper: 2 out of 3 ain’t bad.

Courteous comments welcome, of course. A shorter version of this post appeared on ZDNet.

VMworld is the best storage show I’ve seen in years. VMware’s severe storage problems leave users hungry for solutions – and your friendly neighborhood storage industry is happy to oblige.

It’s almost as if VMware were owned by a storage company.

Flash everywhere
Fusion-io, Nimble Storage, Nimbus Data, Avere, Pure and more were talking about how well flash supports VMware. Fixes VDI boot storms, deduped VMDKs, I/O bound servers and much more.

Pure Storage
Here is Pure’s Matt Kixmoeller giving a nifty demo in this 50 second video:

Not exactly sure what those thousand VMs were doing. Maybe Pure will comment.

Falconstor
I lost track of Falconstor due to their OEM focus and sprawling product line. New CEO Jim McNiel has refocused the company – with the help of former Cheyenne teammates – on backup, business continuity/DR, dedup and virtualization.

Their clustered Network Storage Server turns all of Fstor’s products into tin-wrapped software suitable for channel partners. Takeaway: forget what you knew about them; they are a new company.

Virsto
While the release of their storage hypervisor for VMware makes them seem like a new company, Virsto has been shipping product for over a year, but on Hyper-V, not VMware. Microsoft lost interest in server virtualization and Virsto moved on.

Their product is a virtual appliance that:

. . . runs in each host, creating a transparent virtual storage layer that is thin provisioned, fully cluster-aware, supports very rapid snapshot and clone creation, and scales to support tens of thousands of high performance snapshots and clones.

Virsto . . . decouple[s] application performance from any dependence on the rotational latencies and seek times of underlying disk associated with random writes. All random writes are sequentialized and written directly to a transparent logging device . . . and then asynchronously de-staged to primary storage. . . .

Net/net: high performance virtual storage regardless of underlying physical storage. Virsto offers a free trial – if you try it, let me know how it works.

But wait! There’s more!
Cloud-related products from StorSimple, AMAX and Raidundant continue to pick at the problem of how/when/where cloud integrates with the enterprise.

The StorageMojo take
Many cool products and ideas. The storage problems of many virtual machines are not unlike those of earlier time-shared virtual memory systems, but the scale is much greater.

And when the scale is greater the problem is fundamentally different. As virtualization grows we’ll need to see more creative answers beyond deduplication and flash.

Courteous comments welcome, of course. Message to SNIA: storage networking is passé. Time to retool for the world of virtual machines, noSQL databases, scale-out storage and flash-enabled architectures. New name would be a start.

Pure Storage, a well-funded ($55M) valley startup, came out of hiding last week with a startling claim: enterprise flash that is cheaper^1,2,3 than disk.

¹Cheaper after compressing and deduping the data.

²Cheaper after using almost all the flash capacity, which you can’t do with disks because performance suffers.

³Cheaper compared to the most expensive disk-based enterprise storage you can buy.

Your mileage will vary
4 years ago an EMC VP predicted that flash would replace high-end disks in 2010. That didn’t happen. Why?

After 5 years of hype, enterprises are still leery of flash. Endurance, reliability, data integrity, security, integration – all unanswered questions. At least by the other IT guys in town, even if vendors think they’ve nailed it.

So people buy the known quantity: high-end drives.

The StorageMojo take
Kudos to Pure’s marketing for making a bold, attention-grabbing statement. Too often marketing falls back on the trite-and-true “faster, better, cheaper.”

But IT wants to solve old problems while not introducing new ones. The 10x performance boost would be enough, if IT believed.

Pure’s challenge – as well as other companies with similar products – is to convince IT that not only is flash ready for primetime – but that compression and dedup are too.

And once they do that, why not use them with disks, as well? As Nimble Storage has found, inline compression is now easily handled in software on multi-core chips.

With raw SATA drives down to 3¢/GB, storage vendors have ample opportunity to squeeze costs out. The flash/disk competition will be good for all of us.

Courteous comments welcome, of course. The storage high-end is more active than its been in 15 years. Good!