Data services more important than latency? Not!

by Robin Harris on Thursday, 3 July, 2014

Yesterday’s post on IOPS vs latency provoked some controversy on Twitter. Kappy, CTO of a midwestern IT consultancy, asserted

@storagemojo Most AFA users don’t even care about latency. Sure there are latency sensitive apps, but data services are more important.

When asked what services, Kappy said

@lleung @storagemojo standard issue stuff. Snaps, replication, deep VMware integration, rich API access, etc. See:

I replied:

@Kappy Data services more important than performance? Really? What about the massive savings from lower latency?

Kappy replied:

@StorageMojo can you give more specifics as to what you see as “massive savings”?

The storage pyramid
Surprising question. As said in the StorageMojo post, not the tweet:

Lower latency means fewer inflight I/Os, less server I/O overhead and more server capacity. The systems can handle more work. With fewer servers, software licenses, network ports. Less power, cooling, floor space, maintenance.

Those massive savings. If flash isn’t making something better, you have to ask: Why has flash remade the storage industry in the last 10 years?

The StorageMojo take
Availability and performance are two sides of the same coin: low performance = low availability. That’s why tape’s high latency is slowly pushing it out of a market it once owned.

Most of the data services Kappy mentioned are designed to ensure availability because that’s what customers need. But the problem most data services help manage is simple: data corruption and loss.

There’s a reason we have a storage pyramid: if the cheapest storage were also the fastest that’s all we’d use. Flash has inserted itself between DRAM and disk arrays because it makes our systems perform better for a reasonable cost.

Data services are a Good Thing. But ask a customer if she’d rather have availability and performance or data services and you’ll find out quickly enough what customers care about.

Twitter isn’t the best place for a thorough airing of technical issues. But it can be educational nonetheless.

Courteous comments welcome, of course.

{ 12 comments… read them below or add one }

Fazal Majid July 3, 2014 at 4:27 pm

I understand where you are coming from, but he also has a point. Some reasonable performance hit is an acceptable trade-off for availability and manageability.

A couple of months ago at my company we refreshed our database server layer. In addition to upgrading the servers, we switched from PCIe SSDs (Intel 910) to enterprise SATA SSDs (Intel DC S3700). The reason why is availability – SATA SSDs are hot-swappable, replacing a PCIe SSD requires shutting down the server, opening it and swapping out PCIe cards. The PCIe card has better latency, but the (slight) sacrifice in latency is worth it for the operational considerations.

Now, as for the other features he describes, we get them for free from the OS (OpenIndiana and ZFS), so we don’t have to pay for outrageously marked-up flash arrays to supply them. The ability to snapshot/clone/replicate is also vital for operational purposes, that’s how we did the forklift upgrade with minimal downtime.

Johnm July 4, 2014 at 12:09 am

Why wouldn’t you want all three, low latency, high availability and rich data services ? As I see it high IOps allows you to put more things in the same bucket (consolidation) whereas low last latency allows you to provide a better user experience / application quality of service. Data services such as QOS allow you to guarantee those things whilst dedupe and thin give you efficiencies. But if you are going to consolidate or put your business critical apps on such a system you need basics like online upgrades, seamless scalability, data mobility replication & snapshot options etc provide business continuance.

Ernst Lopes Cardozo July 4, 2014 at 4:30 am

Dear Robin,
I would like to comment on the recent cluster of blogs. There seems to be a potential future where the cheapest storage technology is also the fastest and on top of that, byte-addressable. We could build a server with a petabyte of internal, non-volatile memory. In a way: magnetic core memory of the 60ies and 70ies reinvented. The underlying technology might be magnetic, spin or resistive – let’s not dwell on that.
The question is: how would such a device impact our architecture. Sure, the first implementations would be drop-in replacement for flash in SSD’s and DRAM. Just like the first magnetic disks were used as ‘high speed tape drives with automatic tape mounting’, e.g. the disk was used to simulate multiple tape drives. Then we learned how to use random block-addressable devices, developed file systems. Etc.
When pondering the question: what if all storage capacity is instantaneously accessible at the byte level? my first reaction was OMG, how do you prevent a misbehaving program to accidentally change a few bytes somewhere in that vast storage space and how would you ever detect that? Don’t the many software layers we now have between applications and storage protect the stored data?
But of course we would still need a file system. It might be a file system that maps the files directly in the address space of the applications, giving read or read/write access as requested and permitted.
If the new storage technology is as reliable as our server hardware, we could drop a lot of reliability oriented techniques like RAID, but there still would be a place for replication between machines and data centers.
Do you see such radical change in architecture on the horizon?

James July 4, 2014 at 5:08 am

Well i think a key factor is the latency nowadays but that doesn’t mean that the services are allowed to be crap. Imagine if you are working with big data, it’s important to have a reliable service but at the same time you will need a good latency.

Jerry Leichter July 6, 2014 at 5:35 am

@Ernst Lopes Cardozo: I was at the long-dead Digital Equipment Corporation back in the late 1970’s, when the transition from 16-bit to 32-bit address spaces – and, for DEC, the new VAX – was taking off. Gordon Bell, DEC’s “big idea” guy, sent around a memo posing the question: Now that we have these immense 32-bit address spaces and virtual memory, do we need file systems any more? Why not just map all your data into memory directly? Who would have believed then that 40 years later, 4GB of data would be available on a USB stick for a few bucks, and that also in your pocket would be a phone with a 64-bit processor?

“One layer memory” is a recurring theme. Bell was hardly the first to suggest it – the idea goes back at least to Multics, and others have implemented it, sometimes as research efforts, sometimes as actual commercial products. Various hardware technologies – large address spaces, virtual memory, large caches, SSD’s – have been seen as finally enabling the grand transition to “it’s all just an array of bytes”. At the operating system level, it’s been possible to “map a file into virtual memory” and view its contents as “just an array of bytes” for years.

Meanwhile, on the actual software side, when we deal with massive amounts of data, it’s almost *never* as “just an array of bytes”. It might be as relational tables; it might be as a hierarchical tree; it might be as a NoSQL-style key-value store; it might be as bunches of objects; it might even be as “files and directories” (which merges at the edges with object stores); but the first thing we do in software is add a layer of abstraction atop the “array of bytes” (or what have you) which typically completely hides the underlying structure.

Will some new technology give us bulk, persistent, byte-addressable storage at a competitive price? Maybe; but after all these years, it’s not at all clear that the “byte addressable” part will enable anything really new. Main memory sizes have grown by immense factors of the years – I started in 16-bit days, and even a number of years ago I would stare in wonder at my own code in which I regularly allocated buffers larger than the entire address space of the machines I started on. Today, the on-chip caches are much larger than them main memories of the 32-bit era. So in a sense, we’ve tried all this before. The difference is one of scale, not one of kind.

Perhaps there are surprises to be found at yet more extreme scales; but personally I doubt it. I look forward to new memory technologies because they’ll eliminate various delays, not because I expect them to open up entirely new techniques.

— Jerry

Ernst Lopes Cardozo July 6, 2014 at 9:53 am

@Jerry Leichter. Thank you for your response. I can easily relate to the history part – I started with 12 bit addresses from that same company.
I don’t propose to do away with software layers of abstraction – databases, file systems – we need them. But what about caching? The whole I/O stack in the operating system? SCSI, SAS, Fibre Channel, RAID, all the things that are so specific for ‘storage’ that the very word represents a separate discipline? Does that discipline have a future or are we the coal shufflers on the electric train?

Jerry Leichter July 6, 2014 at 8:08 pm

Caching isn’t going away any time soon: There’s no proposed technology on the horizon that matches main memory speeds. The OS I/O stack? Hard to say. Keep in mind that it’s come to play multiple roles over the years. As a way to access non-local data, it’s become pretty good – and these days, at the datacenter level, pretty much all data is non-local. SCSI/SAS/FC – specific hardware connection technologies that may get replaced – as they replaced earlier technologies – but if you want to access that non-remote data, you need *something*. RAID, if interpreted in its original meaning – where D was disk; let’s ignore the original I for Inexpensive – obviously vanishes along with disks. If, on the other hand, you’re talking about redundant copies of data, that’s here to stay – even if the detailed mechanisms change. (RAID to improve read performance is likely to disappear – it was never a great solution, just one of those things you got for free as part of the package, so why not use it?)

I think the question of whether we need a discipline of “storage” was answered by Codd many years ago. There are ways of organizing data that are appropriate for a single use. That’s what the whole rich study of in-memory data structures is about. When you need to store data for uses you can’t even imagine, by people you know nothing about, you’re talking “storage”. Also, when your data needs to survive indefinitely in the face of hardware failures, hardware replacement, technological obsolescence of the underlying mechanisms for storing the raw bits … you’re talking about “storage”.

These issues are independent of technology. They’ve been with us from the days of magnetic tape, and they’ll be with us even in the unlikely event that we eventually end up with a single technology that’s suitable for storing the raw bits of both main memory and bulk, long-term, reliable storage.
— Jerry

Paul Houle July 7, 2014 at 7:01 am

If there was one core principle of the corporate ideology it would be that “9 women can have a baby in 1 month.” The people that buy and specify enterprise systems care about throughput, but end users feel latency.

It drove me nuts in the 2000s when vendors and developers would be so proud of supporting thousands of simultaneous connections when, if they were serving the customers quickly, they wouldn’t need so many connections.

If you put latency first, throughput usually takes care of itself, but the opposite is not true.

I know so many salesmen spend all day in front of a balky CRM system, and I have no idea how many ulcers, heart attacks, back pain and marital disputes come out of the stress of using bad software all day. Latency isn’t the only thing that makes enterprise systems intolerable, but it’s the one that, on a technical basis, is easy to do something about.

Howard Marks July 8, 2014 at 7:19 am


The disconnect is that you as a theorist view performance as a spectrum and see how lower latency is a good thing. Faster is by definition better in theory.

As a practitioner/experimentalist I know that in the real world computers, and their subsystems, only come in 2 speeds. Fast enough and not fast enough. If I can keep the system in the fast enough my life will be easier with more and better data services.

Since most users running AFAs need 50-250,000 IOPS at 1ms latency they’re better off with a system that does 250 KIOPS at 900us and data services than one that has no data services at 500us.


Robin Harris July 9, 2014 at 10:21 am

Howard, I think you just said we agree. I did not say data services weren’t important. I did say that weren’t more important than latency.

As you point out, data services and a not fast enough subsystem are worse than a fast enough subsystem without data services.

Your larger point is, I think, that many all-flash arrays offer fast-enough and that among those buyers should choose the one with data services they want. Can’t argue with that. However, I think you’d agree that buyers should also understand the economic impact on their total infrastructure of higher latency and that today too few do.


Ernst Lopes Cardozo July 10, 2014 at 4:17 pm

@ Jerry Leichter: You start from a different presumption than I did. In fact, I believe that some of the new technologies “on the horizon” are fast enough (or faster) than current internal memories. And non-volatile. And bytes-addressable (if we want). And might turn out to be the cheapest form of storage. So why cache of you can read and write with the same speed directly from and to “storage”? I’m not talking about cpu cache – yet.
The advent of SSD’s necessitated the use of faster interconnects, but did not change the architecture in a more abstract sense. Storage remained distinct from internal memory (even if it is sitting on a PCI bus) because our software still contains calls to storage functions, reading and writing blocks or somewhat higher abstractions and marking the end of such calls as special events: the data is safe now. None of that happens when a program stores a value in an array.
Let’s forget for a moment all we know about “storage” and start with a clean slate. We have cpu’s, some IO to the outside world and an ideal form of internal memory of arbitrary size. I think we would arrive at a very different software architecture, supported by an extended form of memory management / mapping hardware in our processors. Such servers would exchange transactions for data replication, controlled at the application, file system or object level, rather than as a ‘storage’ function. I suspect that some of today’s very compact embedded systems have such an architecture: cpu, memory and IO, but no storage. Now scale that up by a million …
And to address Robin’s “ch-ch-changing” question – do you feel such topics should be part of today’s or tomorrow’s Storagemojo?

George Crump July 18, 2014 at 1:24 pm

Robin, Interesting points here. I agree on the ROI of IOPS but have side with Howard on the value of data services. See my full response here:

Leave a Comment

{ 1 trackback }

Previous post:

Next post: