For all the time and effort poured into the storage market over the last 20 years, surprisingly little innovation has come from storage vendors themselves. Why is that?

Hall of shame
EMC got its opening when IBM whiffed on the storage array business. IBM had no excuse, as the hard work had been done at Berkeley by Patterson, Gibson, and Katz, in their seminal 1988 RAID paper. Despite the clear guidance offered by the paper, it was a tiny minicomputer add-on memory vendor – EMC – that went big on RAID, giving IBM a thorough shellacking on a market it had owned for decades.

Back in the 90s most of the “advances” in arrays came from the rapid areal density increases in disk drives. Array vendors could tout larger array capacities at lower $/GB, without doing anything more than qualifying a new drive. What more could customers want?

Sun totally blew the filer market, after inventing NFS and having a nice line of servers to run it on. It took NetApp to bring file servers to enterprise respectability.

Sun also blew the chance to own a major chunk of the SAN market, despite being first to market with an all-fibre channel array, as management insisted on FC hubs, not FC switches. Brocade, a startup, did pretty well instead.

Google developed their own scale-out object storage system whose basic architecture has been the fastest growing part of the storage market for over a decade. Where was EMC or NetApp?

The PCIe flash drive was developed by folks from the supercomputer world. Fusion-io, now rolled into WD, brought high-performance internal storage to servers, starting the revolution that has caused storage to migrate into servers instead of away from them. That’s part of what forced EMC to sell itself to Dell.

What’s the problem?
In the Innovator’s dilemma Clayton Christensen theorized that the initial small size of disruptive opportunities made them unattractive to large companies that needed large revenue increments to sustain growth. And, of course, people are notoriously poor at estimating exponential growth rates, such as hyper-scale data centers and internal storage.

So not only does the initial opportunity look dubious, management didn’t properly estimate the growth potential of the new paradigms. Oops!

But there’s a larger problem with the storage industry. Vendors develop a certain mindset and simply don’t believe that their conservative storage customers are going to embrace a new and untried technology – until they do. Then it’s too late.

Instead, vendors focus on creating loopy justifications for selling more of what they already have – Information Lifecycle Management, anyone? – rather than looking beyond the past to create the future. It is left to non-storage people with a broader view – and ignored needs – to see how storage can help them get things done.

The StorageMojo take
Storage vendors are good at faster, better, cheaper. But they seem to be the last to know what the Next Big Thing is.

That’s partly because storage, as the only persistent part of today’s systems, is really hard. The corner cases can be excruciatingly complex. Many of the common assumptions underlying RAID, for example, turned out to be false in practice, which led to ever more complex and costly work arounds.

That, in turn, leads to a kind of techno-blindness, where the details of the old technology are so enthralling that the new technology doesn’t get a serious evaluation. Similarly, financial-blindness – “business is good, margins are great, why rock the boat?” – is an issue for financial management.

Customer conservatism is also a factor. Customers hate getting bit by storage failures, so when they find something that works, they tend to stay with it even if it is far from the optimal solution. That creates an echo chamber for vendors who don’t really want to hear dissenting views.

There is a new paradigm about to hit the industry, which will eviscerate large portions of the current storage ecosystem. Like other major shifts, it is powered by a class of users who are poorly served by existing products and technologies. But if our digital civilization is to survive and prosper, it has to happen. And it will, like it or not.

Courteous comments welcome, of course. I’ve done work for WD.