I got a note from David Flynn, co-founder and CTO of Fusion-io (disclosure: I’ve done work for them) in response to The new storage pyramid. He makes several points about the nature of the array model that I wish I’d made.
Well worth the read.
David Flynn’s note:
Geat analysis Robin.
And, great comments.
My $.02 ….
I think it’s not just about the proprietary nature, the somewhat better performance and features, and the high markups that differentiates “storage arrays” from “clustered storage”.
It’s actually more to do with the vertically integrated nature of the business model of the companies in the array building business. This leads to proprietary architectures, higher margins and, true, somewhat better performance and features.
Let me explain through an analogy…
We used to get graphics workstations from SGI, Apollo, and other vertically integrated vendors, who sold everything end-to-end, down to the monitors and their own proprietary OS’s. These guys commanded HUGE margins – partly to reward their risky investment in solving a worthy, complex problem.
Similarly, the military (and other few others who could afford a million dollar price-tag) used to get flight simulators from Evans&Sutherlands who were also vertically integrated and insanely expensive. You even had niche vendors like Intergraph doing 3D graphics information systems who could justify their own proprietary architectures.
At least for a while.
They were all doing 3D graphics in one form or another. And, now, they are all GONE – thanks to the emergence of a component, the 3D graphics card.
With enough capability to be applicable across all of these different verticals, the 3D graphics accelerator has now shattered the benefit of running a vertically integrated business.
Today, there are myriads of “integrators” who make graphics workstations, flight simulators, GIS systems, etc. at very low margin by comparison. And, they do it by pulling together off-the-shelf components – all commoditized down to the software that provides even the high-value features.
They might have been inferior to the proprietary solutions at first, but not anymore.
Now, what happens when you introduce to the storage industry a component that commoditizes and trivializes the linch-pin reason for expensive proprietary disk arrays, namely the caching tier – using NAND flash.
Once anyone can easily get the performance across any use case (OLTP, OLAP, Data Warehousing, BI, VOD, content caching, etc. etc.) you no longer need vertical specific, highly tuned, proprietary solutions from vertically integrated companies.
Every capability that doesn’t migrate into the component itself becomes nothing but commoditized software to be layered on top by any number of interchangeable integrators. Things like replication, disaster recover, backup, dedup, and so on just become commoditized software that can run anywhere.
This is a classic Adam Smithian market evolution. What used to be a single, vertically integrated provider becomes a layered market where some people build the components, others integrate them (with some bit of value add), and you go to having many players competing on many levels.
And prices go down.
But, thankfully, (for those of us in the business of creating this componentized building-block) volume, productivity, and efficiencies all go up.
So, actually everyone wins. Including society as a whole.
Well, almost everyone wins. Everyone, that is, except for the proprietary array vendors who get caught by the innovators dilemma and a business model that used to be the correct one, but no longer is.
This generally makes them the slowest to simplify their proprietary infrastructures around the commoditized component – to help justify their investment into their heroic proprietary solutions.
In an effort to protect their margins, they endeavor to make things seem as complicated as possible. They do this, say, by preferring that NAND be forced to pretend to be an HDD and be put into HDD drive bays behind HDD protocols, where it has little ability to simplify things or get much additional performance.
They are the last to come out and say it can be simplified. Instead they’ll tell you you must have features X, Y, Z. And, see, those aren’t as good as with our proven architecture.
Let’s take high availability as an example. They aren’t going to tell you that a “shared nothing” strategy – where two separate RDBMS servers with terabytes of direct attached NAND inside of each use off-the-shelf log-shipping for asynchronous replication (or query replication to do it synchronously) to get fault tolerance.
No, they aren’t going to tell you that it’s actually simpler, more cost effective, and, here’s the real kicker… more fault tolerant to share nothing, than to use shared storage – no matter how fault tolerant they claim their monolithic storage array is, it’s still shared.
I’m not saying this market transformation is going to happen by tomorrow. But, given the geometric growth of the performance gap between processors and storage, and the geometric decline in cost of NAND flash – leading to a “Moore’s Law Squared” effect in the benefit to cost ratio – it is going to happen faster than people would think. Even considering the “stodgy” nature of storage folks who are in the business of obsessively caring for precious bits.
It doesn’t hurt that in this global recession companies are looking for ways to reduce costs while still needing to grow throughput. So, there’s more of a willingness to look at different, innovative ways to skin the cat.
I agree with you Robin. It will be a fait accompli by 2015.
David Flynn
CTO, Fusion-io
The StorageMojo take
Technology diffusion is a complex mashup of secular trends, technology development, individual creativity and happenstance. But the current direction of the high-end storage market points to the greatest change we’ve seen since the early 90’s and the advent of arrays.
The “Moore’s Law Squared” effect is particularly intriguing. Humans are terrible at estimating the impact of power functions, so this one is likely to be even more surprising than we dream.
Courteous comments welcome, of course.
