More virtual machines than physical machines were sold last year. What does that mean for storage?

As noted 4 years ago in The virtual machine I/O blender

Engineers have spent decades optimizing the OS, drivers, caching, controllers and disks for specific workloads.

Observed behavior such as locality of reference have informed many strategies. Like read-ahead.


But when you put 25 virtual machines on a single server, what happens to all this hard-won empiricism? It’s gone.

The new empiricism
Enter Tintri. Their mission: making storage for virtual machines more efficient.

Co-founder Kieran Harty was an engineering VP at VMware. The other co-founder, Mark Gritter, was part of Kealia, a server company founded by Andy Bechtolsheim and bought by Sun. Architect Ed Lee has deep storage experience dating from Berkeley’s RAID work in the late 80s.

Tintri brings several ideas to the issue of VM storage.

• A VM-aware file system
• Inline dedup, compression and a hybrid file system that manages flash and SATA drives as a single pool
• Managing VMs directly through interaction with vCenter
• Manage VMs, not LUNS or volumes
• Automatic VM alignment

The StorageMojo take
Much of this mirrors good ideas that other 21st century storage companies have adopted. The big win is the Tintri’s direct management of VMs.

What this means in practice is that an admin looks directly at the I/O activity of each VM, not on a LUN or volume basis, but at the VM level. Which means the array is doing the same thing, able to optimize performance for each VM based on what the VM is doing in real time.

If a VM starts gobbling up storage and bandwidth, it’s easy to see which VM is the problem. This should be tied to a good chargeback system as well.

There are many products that improve – often dramatically – VM storage and performance. But making the VM the unit of storage is brilliant and opens the door to many more enhancements.

Tintri is expecting to support other hypervisors in the not-too-distant future.

Courteous comments welcome, of course. I attended the Tintri presentation as a Tech Field Day delegate. The vendors who presented at the TFD paid for the privilege, which funded my travel expenses.

Our digital civilization requires data integrity and long-term preservation, and neither is assured by our current storage infrastructure. But progress continues.

Latest case in point: Amplidata. This 4 year old company, based in Belgium with a growing US footprint, brings a new level of erasure code goodness to the both problems with a cluster-based object store.

What erasure code goodness, you ask? The first – AFAIK – rateless erasure code, AKA fountain code, storage system in production use.

And that is a good thing because?
Robustness and efficiency.

Amplidata claims storage durability well beyond RAID 6: 10 9′s (spread across 16 drives with up to 4 failures) durability – though the spreads can be much larger logically and geographically. They do this by breaking the data object into segments and adding redundancy data.

The redundancy data adds about 50% to the object size – more efficient than mirroring or triple replication. The benefit is that the system can lose hundreds of segments and still reconstruct the data.

Each object is protected by checksums that can protect against more than 1000 simultaneous bit errors per object. And each write goes to at least to controllers before it is committed.

What kind of monster controller is able to perform all this magic? The minimum configuration is 3 Xeon-based commodity controller nodes with as many 10-drive Atom-based storage nodes as you need.

Amplidata is optimized for bandwidth, not IOPS. With their latest software update they now spec each controller at 750MB/sec, and you can have as many controllers as you can afford.

Sounds like Cleversafe
Cleversafe thought so too, and they’ve sued Amplidata for patent infringement. But Intel – who knows about patents and due diligence – invested after the suit.

Like NetApp’s suit against ZFS, this seems like a vanity project. Surely Cleversafe has more important things to invest in. If they don’t they’re in bigger trouble than we know.

The StorageMojo take
The need for robust, inexpensive and massive storage has been a theme of StorageMojo’s for years. Object storage is the best solution to the problem of scale, while the kind of redundancy and end-to-end checksumming that Amplidata uses seems as robust as anything on the market today.

As for inexpensive, that is in the eye of the beholder, but Amplidata tells me that their newest storage node lists for less than $0.60/GB while consuming only 60 watts. That should be attractive to people running tape silos who want faster access and better redundancy.

Courteous comments welcome, of course. I’m working with Amplidata to produce a video white paper on their technology, so stay tuned for more info on a promising company.

Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

Actually, StorageMojo is wrong, wrong, wrong, wrong and wrong. Umesh Maheshwari of Nimble Storage wrote a detailed and thoughtful response to the StorageMojo post Are SSD-based arrays a bad idea?

The StorageMojo take
Umesh makes good points, but perhaps due to Nimble’s hybrid disk/SSD architecture some seem to miss the mark. What’s missing is ample consideration of what the alternative to an SSD might be, a problem Nimble didn’t have because SSDs work fine, technically and economically, for their hybrid system.

For example, on the issue of reliability Umesh says:

Reliability. There are good reasons to be able to replace failed flash devices similar to how hard disks can be hot swapped. The raw bit error rate (RBER) of flash is actually worse than that of hard disks, and it gets worse as blocks are rewritten. It is also getting worse as manufacturers are moving to increase density. (See this paper from FAST 2012: The Bleak Future of NAND Flash and a related blog post.)

This is correct, but based on the Google/Bianca Schroeder research, the StorageMojo point is that the disk electronics – apart from the head/media pieces – are a major – 40%-50% – source of HDD/SSD failures. The flash controller has to handle the RBER and declining flash performance, but why add the other HDD bits that account for a substantial percentage of drive failures?

I could niggle about Umesh’s other points, but what fun is that? StorageMojo readers are encouraged to check out Umesh’s post and make up their own minds.

Courteous comments welcome, of course. I recently did a nifty video white paper for Nimble, which is a great intro to their innovative architecture. Check it out.

More is coming on SSDs RSN, but in the meantime there is the following piece from Virsto’s Eric Burgener on HA considerations for SSDs. Virsto is a software company focused on making VIRtual STOrage for VMware and HyperV much more functional than the physical kind.

Thus Eric’s response has a very particular POV: what is needed to use SSDs in a virtual environment to ensure high availability – from a company that DOESN’T sell HA hardware. One key point: virtual server environments are much more write intensive than most enterprise apps, so using SSDs as a cache is a losing strategy.

If that is intriguing, read on!

In Thinking About SSD, You Can’t Leave HA Considerations Out

In the “host vs array-based SSD” discussion as it pertains to enterprise accounts, the need for HA must play a critical role. This is true whether you’re working with physical or virtual environments. Any committed data that is not sitting on a shared, non-volatile, external storage device and accessible by at least one other node cannot be recovered until that failed node (on which it resides locally) is brought back up.

There are technical ways to solve this using synchronous replication technologies, but that’s an extra credit project you do yourself for now – as of yet, that hasn’t been built into any host-based SSD products. The reality today is that using host-based SSD precludes the use of HA (but not necessarily things like vMotion, which is NOT HA).

This was touched on in some other posts, but I think it’s an increasingly critical issue in virtual computing environments that may have been a bit downplayed in other comments. If you’re either thinking about moving production server workloads to VMs or have already got them there, HA is critical for a high percentage of workloads.

I can’t imagine an enterprise customer spec’ing out a production virtual server environment without asking about HA. True, there are workloads that don’t require it, but most do.

And it’s not just virtual server environments. We’re running into an increasing number of VDI environments where they want to enable HA for at least a small percentage of the desktops – usually executive desktops. HA isn’t a deal breaker for VDI like it is for “VSI” (virtual server infrastructure), but there are clearly use cases in VDI where you want and/or need it.

Today SSD is pretty much only used as a cache, regardless of where its deployed. And to provide a given level of performance speedup, caches generally have been sized at somewhere around 2% – 4% of the primary data store (it varies by application and exactly what you’re trying to speed up).

In virtual environments, write performance is much more critical because it tends to comprise a much higher percentage of the read/write workload – in VSI environments its not uncommon to see 50% reads/50% writes, and in VDI environments we’ve seen 70% write environments. Unless you’re using a write back cache (with all the attendant additional expense associated with that), you’re not going to get any write performance speedup from the conventional cache architectures, just read.

But now think about what a log architecture, applied at the storage layer, could add. Circular logs that are continuously draining (asynchronously) as they are filling need very little storage capacity to speed up ALL writes for ALL VMs ALL the time. In our experience, you need a log of about 10GB in size for each heavily loaded physical host.

Think about what that could mean for a 16 host environment with 20TB. You could get away with 2-4 200GB enterprise flash drives instead of the 10-12 that you might otherwise deploy in a 20TB environment. If you have a “linked clone” type snapshot technology combined with storage tiering, you could take the extra SSD capacity and create a tier 0 for critical VMs that need very high read performance, like for example the golden masters in a VDI environment or common templates you use to create your server VMs.

This covers both needs – read and write performance – using a lot less storage. That means pretty much the same performance you’d get with the more expensive configuration with more SSDs for a lot less money. If you want to use SSD efficiently, a log architecture is a great idea. And if the logs are placed in shared, non-volatile, external storage (like the SSDs hosted in a SAN array or SAN-based SSD appliance), you can fully support HA.

Host-based SSD cards are closer to the physical host so theoretically they’ll provide more performance speedup, but given Amdahl’s law, how much of that can you really use? Array-based SSD will still get you past storage latencies as your critical bottleneck, and if they’re implemented using a log based architecture you’ll get HA and large write performance speedups as well using a lot less of it.

The StorageMojo take
Write-through caches avoid a lot of sticky update synchronization problems, but as Eric notes they aren’t the best choice in write-intensive environments. And HA adds to the requirements: the cache must be network accessible.

But his larger point bears repeating: SSDs are wonderful for handling metadata. And as we move to object storage and more metadata that capability will become even more valuable.

Courteous comments welcome, of course. I think Virsto’s architecture is smart and fixes some real problems with VMware and vMotion.

StorageMojo offered its soapbox to any vendors willing to weigh in on the question of whether enterprise arrays should be built from flash SSDs or not. Ed Lee, architect at Tintri, formerly of Data Domain and a Berkeley Ph.D, elected to respond. It is a long piece but rich in insight.

Tintri produces hybrid disk/flash SSD appliances optimized for virtual environments, not Symm-killers. They use SSDs in their products, as do other folks like Nimble Storage.

No money changed hands between Tintri and StorageMojo or related entities. My accountant is weeping in the next room.

Begin Tintri’s response:

Outside the SSD Box: More than Faster Disk
Robin Harris of Storage Mojo in his recent article, “Are SSD-based arrays a bad idea? and Matt Kixmoeller of Pure in his response, The SSD is Key to Economic Flash Arrays, present interesting perspectives on whether or not SSDs are the best technology for building flash-based arrays. Robin argues that by rethinking how flash can be packaged outside the SSD box, you can achieve better performance, reliability, cost and flexibility. And these observations are supported by the experience of existing flash-based storage vendors who have developed their own custom flash modules and packaging. Matt argues that SSDs provide an industry-standard product that requires less investment to leverage, better economies of scale, and rapid improvement in technology. These are also very valid points, especially for startups with limited time and capital.

Latency
Taking latency as a point for comparison, flash-based storage vendors using custom packaging often quote IO latencies in the tens of microseconds versus SSD latencies of low hundreds of microseconds. While this is a notable difference, software and interfaces can also add overhead and the final latency seen at the subsystem level may differ by only a factor of two to four. Server-side flash products can avoid more of the software and interface overhead and provide better latencies – but may require rewriting applications to capitalize on this advantage. Keep in mind that hard disk latencies can easily reach tens of milliseconds under even moderate load. ALL of these flash-based products have latencies that are hundreds of times faster than disk.

In short, most of the performance improvement comes from simply replacing hard disk with some form of flash. This immediately shifts the performance bottleneck from storage to some other component in your system. As a result, you won’t be able to take full advantage of flash performance without also optimizing the performance of the rest of your infrastructure, and ultimately rewriting your applications as well.

The above phenomenon explains why replacing your hard disk with flash often speeds up your applications by only a factor of two to three rather than ten or a hundred. Congratulations! You’ve just moved the bottleneck from storage to some other component of your system. By Amdahl’s Law, further improving only storage performance has diminishing returns. So while custom packaging does provide significant advantages in latency, most applications are unlikely to benefit until the rest of the computing ecosystem is optimized to take full advantage of flash.

To take a closer look at SSD latencies, I ran the following simple experiment:
1) Erase an MLC SSD so that no logical blocks were actually mapped to flash, and then issue small random reads.
2) Overwrite the entire SSD so that all logical blocks are mapped, and issue the same small random reads in step 1.

The idea here is to measure the software and protocol overheads of accessing flash packaged as SSD separately from accessing the data on the SSD. Reads with no blocks mapped had latencies of around 70us, while the reads with all blocks mapped had latencies of 250us. In this case only a fraction of the overall IO latency was due to SW and protocol overhead, indicating that SSDs may still have significant room for improving latency.

Form factor
Another important issue discussed by both Robin and Matt is the relative cost of flash packaged in SSD versus non-SSD form factors. Robin argues that an SSD costs significantly more $/GB than the underlying flash while Matt argues that non-SSD packaging is expensive to develop, and SSDs provide useful flash management functions as well as hot-swap capability. It’s certainly true that developing custom packaging has a high up front cost, although this is likely balanced by lower unit costs. But as Robin points out, there are also standard packaging options available for non-SSD form factor flash, which may make custom packaging for non-SSD flash unnecessary.

A very important point to keep in mind when thinking about commercially available SSD vs. non-SSD form factors is that SSDs are designed as a substitute for disk, while non-SSD form factors are often designed as substitutes for memory. This means that SSDs focus primarily on reducing $/GB (its greatest weakness vs. disk), while non-SSDs focus on reducing $/IOPS (its greatest weakness vs. DRAM). This explains why SSD is currently much cheaper on a $/GB basis than PCIe flash, while PCIe flash designed as memory expansion is cheaper on a $/IOPS basis than SSD. This is not to say that you can’t build a non-SSD form factor that has lower $/GB than SSD, just that the primary applications for these non-SSD form factors today is usually not as a replacement for disk.

Whether flash in SSD versus non-SSD form factors is better for use in storage subsystems in the long run primarily depends on the relative volumes of these products, and the feature and price sensitivity of the applications these products serve. At this point the ‘winning’ form-factor seems hard to predict. So as a flash subsystem vendor, it seems desirable to keep your options open and ensure that your technology will work well with a variety of packaging options.

More than just a faster disk
But flash is about more than just performance and packing. Flash enables much more than just a faster, denser replacement for disk. With flash, we can finally remove a key mechanical barrier to scaling not only storage systems, but computing systems in general. Going forward, CPU, network and storage can now all scale with improvements in semiconductor technology. When transistors replaced vacuum tubes, we got more than just compact radios; we got simpler, more powerful computing systems. Similarly, flash is a catalyst that will enable far greater levels of automation and functionality for storage and computing systems than is possible today.

I tend to think of the value of new technology as the product of its simplicity times the functionality it offers. It’s clear why functionality is important, but why is simplicity so important? Technology that is simple to use will be used more often, to solve more problems, in less time. As a result, simplicity has a compounding effect on value:

Value = Simplicity * Functionality

How does one measure simplicity? One way is to list the basic steps it takes to perform a task and how long each step takes. One to three is good, four to six is manageable, and anything resembling a twelve step program will likely require written directions and a significant amount of focus. Note that in assessing the simplicity and functionality of a technology, one must do it in the context of the job that needs to be done. For example, a chainsaw has great features for cutting down trees but not for giving haircuts.

A common problem with many general purpose storage products when applied to applications such as virtualization is that they require executing long lists of steps to get anything done – and most of the features are not directly applicable to virtualization. Paradoxically, many of the features that try to make these products better suited to the application end up making the products more complex – resulting in little improvement in overall value. Kind of like adding too many tools to a Swiss army knife until you have so many that the attachments start to stick and rub against each other.

Flash as a catalyst
Flash eliminates a key mechanical barrier to scaling computing systems and is 400 times faster than disk. To keep things in perspective, the speed of sound is “only” 250 times faster than walking! If I could get to work at supersonic speeds, I would no doubt save a lot of time each year. But would I do no more which such an ability? Similarly, is flash just a faster replacement for disk? Will it make no significant difference in the way storage is managed and used? We obviously don’t think so. Flash will greatly increase the value of storage by improving both the simplicity and functionality of enterprise storage products. But these gains will not come easily or without their own set of problems.

An obvious way flash promotes simplicity is by eliminating performance bottlenecks, but as flash enables more dense storage systems many of those gains will be converted to problems in quality-of-service. A more significant way flash promotes value is by providing a better building block for constructing storage systems: flash promotes simplicity by enabling higher levels of automation and allows the implementation of more powerful functionality.

Flash will fragment the enterprise storage market. The general purpose storage systems of today will be supplanted by new flash-based products that are far simpler and more powerful for the specific application areas that they target. This will amplify the simplicity and power that flash already makes possible, and further accelerate the fragmentation of the storage market. This is precisely what happened in the 1980’s when advances in networking technology caused a shift from centralized computing to networked computing – and in the process fragmented the direct attached storage market into ones based on networked storage technology. Over time, the networked storage markets consolidated into the current general purpose storage market dominated by a few major vendors. And so the cycle is repeating itself.

We are at the start of a new technological shift. A shift that is made possible by flash and one that will disrupt the existing enterprise storage market. Just as transistors enabled new products such as personal computers and smart phones, flash will enable simple, intelligent and fast enterprise storage systems. In turn, this will lead to much higher value for end users, but only if we think outside the storage box and treat flash as more than just a faster, denser disk.

The StorageMojo take
For the record the original post wasn’t looking at hybrid solutions, although it is obvious that SSDs can help legacy designs stay competitive without replacing all disks for a few years. For folks like Tintri and Nimble who want to speed up disk storage to stay affordable SSDs make sense. Why engineer a small part of your system when an off-the-shelf solution will suffice?

But for high end transactional SAN storage I still don’t see how SSDs are the right way to go. But I’m expecting more responses, so stay tuned.

Courteous comments welcome, of course. I’m working on a post that reflects directly on Ed’s comment about SSD latency. You’ll like it.

Pure’s Matt Kixmoeller saw the Are SSD-based arrays a bad idea post and, unsurprisingly, responded. The SSD is Key to Economic Flash Arrays is a good post and I urge interested readers to check it out.

Pure has a stellar team with deep experience. Their views are worth considering.

As Matt notes:

This post caught our eye for an obvious reason: Pure Storage did start “fresh” to build an all-flash enterprise storage array, and we did decide to use the SSD form factor, after quite exhaustive looks at all the other options. Quite simply, we found that SSDs are the most efficient and economic building blocks from which to build a flash array. Let’s explore why.

After dismissing disk arrays that add flash drives – as I do – Matt focuses on (1) all flash appliances built from raw NAND and (2) flash arrays using flash SSDs.

SSDs are most efficient
Matt argues that SSD-based arrays have 3 key advantages:

• Economics. SSDs are a commodity product that raw flash arrays will have a hard time out-engineering.
• Flash controller complexity. Matt notes, correctly, that the flash controller is at the heart of argument. Better to use a controller that goes into millions of SSDs or one purpose-built for a single vendor’s array? How will the single vendor be able to keep up?
• Servicability. Pure’s use of SSDs enables them to offer a familiar hot-swap experience that higher density designs may not offer. Futhermore, Pure’s data reduction features increase effective density to rival raw flash designs.

In conclusion, Matt makes a couple of more points. First, that SSD form factors will become much more compact, such as Apple’s DIMM-like mini-SATA SSD used in the MacBook Air. Second, that the proof is in the pudding: Pure, he says, has “. . . delivered with break-through performance, at a cost below traditional spinning disk.”

The StorageMojo take
How does Matt’s response stack up to the criteria in the original post? Not that there’s anything magic about them, but . . . .

• Latency. No response, which doesn’t mean they’re worse.
• SSD bandwidth. No response, but to be fair with enough SSDs you should be able saturate 16Gb Fibre Channel.
• Reliability. No direct response. Instead a focus on servicability. More on that below.
• Cost. Says Pure is cost-effective using their data reduction technology.
• Flexibility. This is the heart of Matt’s argument: due to the commodity volume of the flash controllers flash SSDs will evolve faster – in functionality and cost – than any proprietary solution could. Proprietary flash controllers, he says, will be boat anchors for flash array vendors and are likely to end up controlled by flash manufacturers.

Servicability is an interesting response to the question of reliability. After all, the reason hot swap is important for some components but not others is because they either a)fail often – individually or in aggregate – b)failure compromises the product or c)online expansion, upgrading or reconfiguation is desirable.

Power supplies are routinely hot swappable because they have the lowest MTBF of any major system component. Disks are hot swappable because they come in multiples that reduce their aggregate MTBF while their standardized design makes hot swap cheap. I/O cards are often hot swappable because they are critical and needs change.

SSDs should be hot swappable because their failure rates are at best about half that of disks. But DIMMs, another critical component, especially if you invest in high-capacity ones, aren’t, because they rarely fail.

While I’m not aware of any non-SSD enterprise array vendor whose arrays don’t include hot swap components – love to be educated – which is more important: a short mean time to repair (MTTR) or a long mean time between failures (MTBF)? Because that is the argument about servicability.

I’d like to publish responses from vendors who feel strongly about this issue. Not in the comments, but as a blog post. Any takers?

Courteous comments welcome, of course. I was so impressed with the Pure Storage team that I signed a rare NDA with them last spring to get briefed, the first of 2 visits to their Castro street HQ.

A hardware storage controller is an expensive guarantee that you’re using old technology to handle your most important data. Hardware specs are frozen early in the typical 18-24 month development cycle so by the time you get your “new” controller it is already 2 years old.

But it may not have to be that way. In Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization researchers Muli Ben-Yehuda, Michael Factor, Eran Rom, Avishay Traeger, Eran Borovik and Ben-Ami Yassour of IBM Research–Haifa wanted to find out if virtualized storage controller features are feasible.

Short answer: with some tweaking, yes.

The big question is overhead. Storage controllers are typically in the data path, so latency, as well as compute efficiency on out-of-date processors, are real concerns.

Unlike the gateway approach of virtual storage appliances (VSA), the team ran the VMs directly on storage controllers using the Linux KVM hypervisor.

Overhead
The team identified 3 sources of performance overhead:

• Base. System work such as virtual memory managment or process switching.
• External communication. Important if a new function is layered on top of the storage system, such as a file server.
• Internal communication. Virtual machine coordination and communication with the hardware controller.

Reducing overhead
Different techniques are used to limit each type of overhead.

Base They statically allocate CPU cores to the guest to ensure sufficient resources. Memory is also statically allocated to the VM to reduce translation overheads.

External Device assignment is the highest-performing approach as it eliminates hypervisor intervention for physical events. This requires assigning the network device directly to the guest using an SR-IOV (single root I/O virtualization) enabled adapter which allows the guest to send requests directly to the device.

Internal communications To reduce internal communication overhead, they modified KVM’s block driver to poll instead of interrupt. This gives a fast, exit-less, zero-copy transport.

Results

By using these techniques, we show no measurable difference in network latency between bare metal and virtualized I/O and under 5% difference in throughput. For internal communication, micro-benchmarks show 6.6μs latency overhead, read throughput of 357K IOPS, and write throughput of 284K IOPS; roughly seven times better than a base KVM implementation. In addition, an I/O intensive filer workload running in KVM incurs less than 0.4% runtime performance overhead compared to bare metal integration.

That sounds pretty good.

The StorageMojo take
While the static assignments may reduce flexibility, the win is updating storage functionality on the fly. But are there viable use cases? The arc of controller history suggests there are.

The earliest disk drives were directly controlled by the host CPU. Over the decades that and much other functionality migrated to controllers and to disks. Lately that trend has slowed because of large investments in existing standards.

This paper shows that it is possible to migrate more functionality to controllers without lengthy development cycles, enabling architects to make different tradeoffs.

For example, big data requires big pipes, and big pipes are expensive. If volume-reducing preprocessors could be added to file servers, existing bandwidth could be optimized.

More importantly, it suggests that by virtualizing the controller’s applications, the underlying hardware can be updated more frequently. To be fair, that’s not what the authors suggested, but it certainly seems possible based on their work.

Courteous comments welcome, of course. Jeff Darcy of Red Hat has his own list of favorite papers from FAST ’12 here.

EMC’s Chuck Hollis blogged about The Vendor Beating a couple of months ago. The unspoken question in the post is “how do we understand what customers are telling us?”

He writes

As an employee of a large IT vendor, I’ve been at the receiving end of a reasonable number of vendor beatings.

Occasionally it’s richly deserved. But, sometimes, it’s masking a deeper set of issues that have very little to do any vendor whatsoever.

Unhappy customers, like unhappy families, are all unhappy in their own way. This customer appeared to be overstaffed, under-skilled and poorly managed.

Interpretation
Interpreting customer complaints and behavior is hard. When companies can’t decipher what customers want – which is usually what the company isn’t selling – it is easy and dangerous to tune them out.

Customers can tell you things about your company and products that you can’t directly discover for yourself, but what customers say may be different from what they think. And both are influenced by the customer’s context, which can include company politics, prior vendor experiences, knowledge deficits and employee level.

Diagnosis
Steve Jobs once said that customers don’t know what they want until you show it to them. Customers know what would improve the current product in the current use case, but they can’t imagine bringing multiple novel technologies to bear on a much broader problem.

Tablet computers flopped for years until the iPad crystalized the market. Everyone saw the tablet problems: thick; heavy; slow; clunky UI; poor battery life; and, thanks to low volumes, cost. Incremental improvements – faster processors, more RAM, larger disks – didn’t help.

Tablets required a deep rethinking and application of several novel technologies – flash, gestures, CNC case milling, an app store and an energy-efficient OS – to create a compelling user experience.

The iPad illustrates the problem of listening to customers: they described symptoms and suggest fixes, but couldn’t articulate the underlying problem: how the use case differs from desktop and notebook PCs. That requires an act of imagination, not transcription.

The StorageMojo take
In Chuck’s post an EMC presales engineer identified the root cause of the customer’s pain:

. . . the database environment had grown willy-nilly over the years — it wasn’t laid out well, the queries weren’t particularly well written, and so on.

Sure, there were things we could do on the storage side (e.g. faster storage, better layouts, etc.), but it was a bigger issue than just storage performance.

But the larger question is: with high-speed and high-capacity SSDs, why isn’t this customer moving to an infrastructure that doesn’t need this fancy tuning? EMC can’t manage the fight between DBAs and storage admins, but they could be making it less contentious.

From within the EMC ecosystem the solution is clear: more training, professional services and faster gear. But from the outside the question is: who is building “it just works” high performance storage?

Courteous comments welcome, of course. I admire Tucci’s innovative EMC business model: outbid everyone else for chasm-crossing companies; give them global distribution and support; and watch the bucks roll in. It may not be innovative technically but it is innovative.

Reader Kyle asks a good question:

SANs are advertised up the wazoo as having lots of internal redundancy such as redundant power, redundant controllers, etc. I’ve spent enough time with redundancy to know that having two pieces of hardware often doesn’t cut it. I was wondering what the real story is from someone who has spent a lot of time in the storage space. Do complete SAN failures really pretty much *never* happen or are they just on the rare side? If so what are the common points of failure? Perhaps people, the OS, non-redundant hardware parts?

Please, SAN folks, tell StorageMojo readers your experience. In the meantime, here’s

The StorageMojo take
Kyle asks 2 questions: how reliable and available are the individual devices that make up a SAN and how reliable and available is the system – the SAN as a whole.

Redundancy is aimed at ensuring availability. Because of the redundancy’s greater component count you also have more failures.

Failures of redundant components shouldn’t affect availability – assuming, that is, that failures are not correlated. That assumption turned out not to be true of RAID arrays, making them less available than advertised.

How much redundancy is enough? Customers generally prefer triple redundancy if they can afford it, partly for availability and partly for performance: losing ⅓rd of system performance is less painful than ½. But for the moonshots, NASA chose quintuple redundancy on critical systems.

Yet I’d guess that most are more concerned about SAN system availability – which includes not only what we consider SAN hardware, but also the server-side HBAs, drivers and management software. It is here that the nastiest bugs lurk: untestable interactions between applications, drivers, firmware and architecture that bite us hard – and crash entire SANs.

But what is your experience, gentle reader? Many of us would like to know.

Courteous comments welcome, of course. Update: Bayesian analysis is the best tool to evaluate system-level availability, as noted in this StorageMojo video. Sadly, the tool referred to is no longer online. Anyone want to take a whack at a new one?

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

Amazon Web Services architect James Hamilton has been posting on network issues for over a year and researching them much longer. As Ethernet becomes the de facto SAN technology, his views become more relevant to the larger storage market.

Critique
Part of Mr. Hamilton’s concern is the structure of the networking industry: the high margins; the dominance of a single player, Cisco; the closed technology; and the heavy vertical integration. All antithetical to the dynamics that have driven server costs down so successfully in the last 20 years.

These are issues the storage industry knows too well. But Mr. Hamilton is more concerned about the waste the current high-cost industry structure causes.

Waste?

Workload placement
The cost of network bandwidth leads to network over-subscription. Networks are configured as tree topologies: the further you move from end nodes the worse the over subscription.

As described in the 2009 Microsoft Research paper VL2: A Scalable and Flexible Data Center Network:

. . . the capacity between different branches of the tree is typically over- subscribed by factors of 1:5 or more, with paths through the highest levels of the tree oversubscribed by factors of 1:80 to 1:240. This limits communication between servers to the point that it fragments the server pool — congestion and computation hot-spots are prevalent even when spare capacity is available elsewhere.

This throttles data center performance by limiting server-to-server bandwidth, fragmenting resources and reducing network utilization. The latter reflects the redundant paths needed in case of switch failure: ≈50% or more of costly data center bandwidth goes unused.

As might be expected, big Internet data centers like Amazon’s have complex and unpredictable workloads. They need lots of bandwidth between all servers all the time.

A solution
The VL2 paper describes an experimental solution to these problems that includes location-specific and application-specific addressing, multi-path traffic load balancing and a novel directory design that efficiently handles lookups and updates to network mappings.

In an 75-node test cluster the design moved 2.75TB of data in 395 seconds – 94% of maximum network bandwidth – at a fraction of the cost of current enterprise networks. The paper calculates that a cloud-service scale network with no over-subscription could be built with commodity switches at 1/14th the cost of a traditional data center Ethernet.

Whoa!

The StorageMojo take
VC and engineering dollars follow high-growth markets. What Google, Amazon and Microsoft want, they get. With the rapid growth of public cloud services the network over-subscription problem will get solved.

Merchant silicon from Broadcom, Intel and Marvell is making a tried-and-true Moore’s Law attack on hardware cost. The protocol stack is tougher, but several open-source industry initiatives are under way with support from major companies. Progress will be slower than hoped, but within 3 years we’ll have a viable stack to build on.

Where does this leave the networking industry? That depends on where you sit.

Cisco will be the biggest loser, because they’ve been the biggest winner with the current model. They may need to pull an IBM and move big into services if they want to stick around. Ironically, Cisco’s UCS product line – which bakes in the tree-structured network – has further motivated broader industry action.

The rest of the industry can go after this emerging market with a lower-GM business model. Not all of them will, but it will be a critical success factor.

The big winner will be storage. Scale-out storage relies on spraying data across multiple racks for maximum availability, utilization and performance. Cheaper, faster, better scale-out networks will only drive storage demand.

For most of us this is an academic problem today. Lightly used systems – such as for backup and archiving – don’t see Amazon’s problems. But in 5 years this will be common even outside the public cloud providers.

Just as IT users have benefited from Google’s push on energy efficiency and much more, they will also benefit from much lower cost and more scalable networks.

Courteous comments welcome, of course. I can’t help but continue to marvel at how dumb Cisco’s UCS has turned out to be. It’s a gift that keeps on giving.

Sometimes in the midst of the endless tweaking needed to maximize storage performance one just wants to say “screw it! Put everything in RAM!” And that’s just what RAMCloud does.

Disk is the new tape, flash the new disk, DRAM the new flash.
RAMCloud is a research paper (pdf) and an open software project. The goal is enterprise-class availability with every bit of active data stored in DRAM, not disk or flash, for maximum performance. It is a key-value object store today, though as pure software that could change.

It’s the brainchild of John Ousterhout, a Stanford prof who invented Tcl back in the 80s at Berkeley.

Isn’t DRAM volatile and costly?
Right on both counts, grasshopper, so RAMCloud isn’t a 1 for 1 disk-style architecture. No Google FS-style triple replication here, or RAID-style erasure coding.

Instead RAMCloud uses buffered logging:

. . . a single copy of each object is stored in DRAM of a primary server and copies are kept on the disks of two or more backup servers; each server acts as both primary and backup. However, the disk copies are not updated synchronously during write operations. Instead, the primary server updates its DRAM and forwards log entries to the backup servers, where they are stored temporarily in DRAM.

Instead of working around crashes – using multiple object copies as scale-out storage does – RAMCloud recovers lost data from the DRAM logs or disk drives to replicate the lost data at high speed. That’s possible because all the log data is in DRAM or spread across many disks.

In a recent paper (Fast Crash Recovery in RAMCloud) (pdf) Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John Ousterhout, and Mendel Rosenblum (co-founder of VMware) go into more detail on this critical feature.

The key elements are:

• Scale. Servers scatter their backup data across all other servers so thousands of disks can serve the recovery.
• Log-structure. Reduces complexity and offers high performance.
• Randomization. Many decisions need to be made in a large cluster. Rather than CPU, time and bandwidth consuming determinism, injecting randomization speeds decisions with less overhead.
• Dynamic tablets. The key-value store tracks resource usage within a single table and ensures that no single partition is too large for fast restores.

DRAM is volatile so the log replication data is spread to other servers on other racks for redundancy before being committed to disk. Still, total system write throughput is limited by the disk write speed, whose limits are a key reason people are moving from disks. Flash drives may help, but other techniques, such as log truncation and sharding make it possible to get good performance from several thousand SATA drives.

How good? The team reports that in a 60 node cluster they recover 35GB in 1.6 seconds. With more nodes larger partitions should be restored even faster. Scale is good.

Lights out!
Power failures wipe all the data in DRAM. The obvious defense is to avoid failures: combine battery backup with diesel generator sets. Power ride-through will handle interruptions into the hundreds of milliseconds.

But who is going to trust that? That’s why future commercial implementations will insist on logging to stable storage, such as the flash SSDs.

They’re getting cheaper fast – faster than DRAM – which will make this a common approach.

Cost
Professor Ousterhout kindly sent a short note about cost, correctly noting that

. . . if you measure cost/operation, DRAM is roughly 100x cheaper than disk, since a disk can only perform about 100-200 operations/second. This is why RAMCloud makes sense for data-intensive applications. . . .

While you and I might find that persuasive, too many enterprises don’t. The deep conservatism of the storage culture – both figuratively and literally – makes cost a good excuse to stay with the tried and true, and easy to explain to CFOs.

The good news for the company I hope he is starting is that the primacy of $/GB is slowly eroding as customers see the system level savings from fast storage. SSD vendors and companies like TMS and Kaminario are breaking trail for RAMCloud.

The StorageMojo take
Make no mistake: RAMCloud is a research project, not a commercial product, years and million$ away from commercial application. But the concept is promising.

Imagine a world where data layout doesn’t matter, where apps are optimized for sub-millisecond storage, where 100 byte I/Os are faster and just as efficient as 8KB I/Os. The architectural implications are huge and would take a decade or more to absorb.

RAMCloud raises the thorny issue of tiering: getting hot data on the hot storage and everything else off to disk. There are OK answers for tiering but nothing insanely great.

RAMCloud shows we’re far from the end of the line in what storage can do. Faster, better, arguably cheaper: 2 out of 3 ain’t bad.

Courteous comments welcome, of course. A shorter version of this post appeared on ZDNet.

One more datapoint and we’ll have a trend: NoSQL databases managing metadata. It’s obvious in retrospect: use a scalable big data tool to handle scale-out metadata. Maybe not a requirement today, but surely will be with even bigger data tomorrow.

Metadata is a fraction of the user data set, but it gets hammered much more. As more metadata is found useful the hammering will get more insistent.

Nutanix
Nutanix, whose CTO and co-founder, Mohit Aron, was a developer of the Google File System, uses MapReduce. Nutanix achieves it scale due to its distributed metadata, masterless architecture – powered by MapReduce jobs that run in the background.

Druva
Druva, a backup company for mobile devices, also uses a NoSQL database to manage storage metadata. They say they’ve found that NoSQL scales over an order of magnitude better than relational in similar applications.

Somebody else
A company that shall remain nameless is porting Hadoop to their backend. The customer won’t be able to access Hadoop for their work – it is strictly for the system’s internal use.

It is a proof of concept so it isn’t a 3rd data point, but they see the potential advantages. Call it data point 2½.

The StorageMojo take
Small advances are the building blocks of disruption. RAID made it possible to build available storage using cheap disks. Consumer adoption of PCs made disks even cheaper. Moore’s Law made RAID controllers cheaper and faster, or faster and more capable.

A virtuous circle of disruption.

The basic architecture of scale-out storage systems – purpose-built software on clustered commodity hardware – has been stable. But this is the beginning of scale-out storage 2.0: taking scale-out technology developed for users and incorporating it into the storage infrastructure itself.

These ideas are bubbling up among the latest startups and among the establishment players. At some point the old RAID architectures will be well and truly broken, able to compete in smaller and smaller niches until the revenue can’t justify more investment.

Of course vendors have been making RAID controllers out of servers for years now, and those servers can run any software they want. But at some point the explicit and implicit assumptions in the old architecture crash into current realities – either in cost, development time, feature completeness or management overhead – and then we move on.

Courteous comments welcome, of course. I learned about Nutanix at the last Tech Field Day “The Independent IT Influencer Event” which paid for my travel expenses to Silicon Valley.