More virtual machines than physical machines were sold last year. What does that mean for storage?

As noted 4 years ago in The virtual machine I/O blender

Engineers have spent decades optimizing the OS, drivers, caching, controllers and disks for specific workloads.

Observed behavior such as locality of reference have informed many strategies. Like read-ahead.


But when you put 25 virtual machines on a single server, what happens to all this hard-won empiricism? It’s gone.

The new empiricism
Enter Tintri. Their mission: making storage for virtual machines more efficient.

Co-founder Kieran Harty was an engineering VP at VMware. The other co-founder, Mark Gritter, was part of Kealia, a server company founded by Andy Bechtolsheim and bought by Sun. Architect Ed Lee has deep storage experience dating from Berkeley’s RAID work in the late 80s.

Tintri brings several ideas to the issue of VM storage.

• A VM-aware file system
• Inline dedup, compression and a hybrid file system that manages flash and SATA drives as a single pool
• Managing VMs directly through interaction with vCenter
• Manage VMs, not LUNS or volumes
• Automatic VM alignment

The StorageMojo take
Much of this mirrors good ideas that other 21st century storage companies have adopted. The big win is the Tintri’s direct management of VMs.

What this means in practice is that an admin looks directly at the I/O activity of each VM, not on a LUN or volume basis, but at the VM level. Which means the array is doing the same thing, able to optimize performance for each VM based on what the VM is doing in real time.

If a VM starts gobbling up storage and bandwidth, it’s easy to see which VM is the problem. This should be tied to a good chargeback system as well.

There are many products that improve – often dramatically – VM storage and performance. But making the VM the unit of storage is brilliant and opens the door to many more enhancements.

Tintri is expecting to support other hypervisors in the not-too-distant future.

Courteous comments welcome, of course. I attended the Tintri presentation as a Tech Field Day delegate. The vendors who presented at the TFD paid for the privilege, which funded my travel expenses.

More is coming on SSDs RSN, but in the meantime there is the following piece from Virsto’s Eric Burgener on HA considerations for SSDs. Virsto is a software company focused on making VIRtual STOrage for VMware and HyperV much more functional than the physical kind.

Thus Eric’s response has a very particular POV: what is needed to use SSDs in a virtual environment to ensure high availability – from a company that DOESN’T sell HA hardware. One key point: virtual server environments are much more write intensive than most enterprise apps, so using SSDs as a cache is a losing strategy.

If that is intriguing, read on!

In Thinking About SSD, You Can’t Leave HA Considerations Out

In the “host vs array-based SSD” discussion as it pertains to enterprise accounts, the need for HA must play a critical role. This is true whether you’re working with physical or virtual environments. Any committed data that is not sitting on a shared, non-volatile, external storage device and accessible by at least one other node cannot be recovered until that failed node (on which it resides locally) is brought back up.

There are technical ways to solve this using synchronous replication technologies, but that’s an extra credit project you do yourself for now – as of yet, that hasn’t been built into any host-based SSD products. The reality today is that using host-based SSD precludes the use of HA (but not necessarily things like vMotion, which is NOT HA).

This was touched on in some other posts, but I think it’s an increasingly critical issue in virtual computing environments that may have been a bit downplayed in other comments. If you’re either thinking about moving production server workloads to VMs or have already got them there, HA is critical for a high percentage of workloads.

I can’t imagine an enterprise customer spec’ing out a production virtual server environment without asking about HA. True, there are workloads that don’t require it, but most do.

And it’s not just virtual server environments. We’re running into an increasing number of VDI environments where they want to enable HA for at least a small percentage of the desktops – usually executive desktops. HA isn’t a deal breaker for VDI like it is for “VSI” (virtual server infrastructure), but there are clearly use cases in VDI where you want and/or need it.

Today SSD is pretty much only used as a cache, regardless of where its deployed. And to provide a given level of performance speedup, caches generally have been sized at somewhere around 2% – 4% of the primary data store (it varies by application and exactly what you’re trying to speed up).

In virtual environments, write performance is much more critical because it tends to comprise a much higher percentage of the read/write workload – in VSI environments its not uncommon to see 50% reads/50% writes, and in VDI environments we’ve seen 70% write environments. Unless you’re using a write back cache (with all the attendant additional expense associated with that), you’re not going to get any write performance speedup from the conventional cache architectures, just read.

But now think about what a log architecture, applied at the storage layer, could add. Circular logs that are continuously draining (asynchronously) as they are filling need very little storage capacity to speed up ALL writes for ALL VMs ALL the time. In our experience, you need a log of about 10GB in size for each heavily loaded physical host.

Think about what that could mean for a 16 host environment with 20TB. You could get away with 2-4 200GB enterprise flash drives instead of the 10-12 that you might otherwise deploy in a 20TB environment. If you have a “linked clone” type snapshot technology combined with storage tiering, you could take the extra SSD capacity and create a tier 0 for critical VMs that need very high read performance, like for example the golden masters in a VDI environment or common templates you use to create your server VMs.

This covers both needs – read and write performance – using a lot less storage. That means pretty much the same performance you’d get with the more expensive configuration with more SSDs for a lot less money. If you want to use SSD efficiently, a log architecture is a great idea. And if the logs are placed in shared, non-volatile, external storage (like the SSDs hosted in a SAN array or SAN-based SSD appliance), you can fully support HA.

Host-based SSD cards are closer to the physical host so theoretically they’ll provide more performance speedup, but given Amdahl’s law, how much of that can you really use? Array-based SSD will still get you past storage latencies as your critical bottleneck, and if they’re implemented using a log based architecture you’ll get HA and large write performance speedups as well using a lot less of it.

The StorageMojo take
Write-through caches avoid a lot of sticky update synchronization problems, but as Eric notes they aren’t the best choice in write-intensive environments. And HA adds to the requirements: the cache must be network accessible.

But his larger point bears repeating: SSDs are wonderful for handling metadata. And as we move to object storage and more metadata that capability will become even more valuable.

Courteous comments welcome, of course. I think Virsto’s architecture is smart and fixes some real problems with VMware and vMotion.

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

A shout out to all ‘worlders: the StorageMojo analyst team will be descending en masse on the lush Nevada desert (yes, in Phoenix real estate people often tout the “lush desert” – don’t they know what desert means?) to take in our 1st VMworld.

We’re arriving the 29th & departing the 31st. Hope to learn a lot more about storage in the virtual world.

The StorageMojo take
Data storage is hard enough when your CPUs aren’t VMotioning across the globe. Dealing with the related issues of concurrency, consistency and limited network bandwidth makes it even more challenging.

I’m interested in seeing what Virsto is announcing as well as what is coming from companies I haven’t yet heard of. Do readers have any suggestions for vendors that shouldn’t be missed?

Courteous comments welcome, of course. Even though it is a hike to the Venetian and the Wynn, I’m going to stay in one of the new City Center hotels.

Robert Gray, a former analyst at IDC and long time friend, asked me to respond to a comment from NetApp president Tom Georgens.

Tom told Information Week that:

. . . virtualization is fundamentally changing the way data centers are designed: instead of individual infrastructure around individual apps, they’re building one single infrastructure that’s independent of the apps. And now they want the same with storage because customers are seeing that virtualization is the way to dramatically improve the flexibility of the business as well as IT costs.

(bolding added)

NetApp’s on a tear
NetApp is growing, approaching a $5 billion annual run rate. Mr. Georgens attributes part of that growth to the 50% capacity savings guarantee that NetApp announced two years ago.

That guarantee, which originally applied only to the storage of VMs, appears to have morphed into a more generic talking point: buy NetApp and we’ll give you the same capacity with half the boxes.

Gee, a marketing initiative had an impact on the bottom line? Don’t tell HDS!

The StorageMojo take
It is natural to conflate the impact of virtualization and the growth of single infrastructure thinking. But that is an accident of timing, not cause-and-effect.

The single infrastructure meme grows out of the obvious success of scale-out processing and storage architectures, as exemplified by Amazon and Google. Even without VMware the economic advantages of Internet scale cloud computing and storage would drive the industry.

What virtualization is doing is driving IT architects and CIOs to rethink long-settled – fossilized – infrastructure practices. As they do so it becomes clear that the standard stove-piped enterprise IT model no longer makes sense either economically or operationally.

True, virtual machines are making the desirability of a common storage infrastructure obvious to even the most hidebound IT shop: migrating storage with a VM is tough enough – why have a different infrastructure at the other end? But virtualization or no, the old model of bespoke infrastructure for each application was not economic. Today’s glass houses are factories, not artisanal workshops, and they need industrial scale infrastructure.

This builds on the long-term secular trend of the commoditization of IT. This is why all the array vendors have been busy migrating their custom hardware to software running on x86 commodity boards.

Ironically, NetApp doesn’t actually offer a single scale-out infrastructure, with the exception of recently acquired Bycast. The Spinnaker acquisition – lo, these many years past – was supposed to solve the problem but the integration issues have proven more difficult than NetApp ever imagined.

Enterprises will never reach the scale of a Google or Amazon. However, they will find that scale, even at the enterprise level, has its own dynamics. One implication is a widespread move to object storage.

More on that later.

Courteous comments welcome, of course. Robert, I hadn’t seen Tom’s words. Thanks for bring them to my attention.

Got a presentation from a company I hadn’t heard of before: VKernel. Their specialty is virtual machine capacity planning and management.

They demo’d their software, which ships as a VM. They offer free versions that are good enough to tell you if their paid tools will be able to help you.

VKernel has a customer sysadmin in the room, talking about how his company – which remains nameless, but has over 1,000 VMs on 40 C-class blades – uses the product. The user says that some of the big wins have been in reducing over-provisioning and quickly finding performance problems.

The user says that it is very lightweight and agentless. It installs in 15 minutes or less and quickly provides actionable data, even to junior admins.

Today the product supports VWware and their Capacity Optimizer also supports Hyper-V.

The StorageMojo take
Storage planning is a lost art: many admins grew up in a 1-app/1-server world where capacity management meant adding another drive to a server. Virtual machines are much more complicated.

There’s the I/O blender effect, which renders 40 years of I/O empiricism useless. The interactions between VMs on a single physical server. Contention for memory, bandwidth and IOPS.

It’s a new world. VKernel looks promising.

Courteous comments welcome, of course. This presentation is part of the Gestalt IT blogger tour that VKernal, among others, helped pay for. If you’ve tried VKernel please comment on them below. Update: And I think I’ve finally learned how to spell kernel. Thanks Joe, Stephen. End update.