It isn’t about the capacity. It’s the performance.
I few weeks ago I asked StorageMojo readers to help out Jim Handy of Objective Analysis, a semiconductor research firm. They did, and Jim got some surprising results.
Now that I have finished interviewing 21 PCIe SSD users (both Fusion-io and Texas Memory Systems users, thanks to help from StorageMojo) I can step back and see the common threads. One surprises me a bit.
The almost universal reply from users is that, although they were able to reduce the use of other hardware by adding flash to their system, they were unhappy that the price per gigabyte for flash is so much higher than HDDs.
In one or two cases the SSD displaced very expensive hardware. One company was able to re-deploy a $500K high-speed SAN which it now uses to perform backups.
A couple of others slashed their DRAM sizes by as much as 2/3, leading to power savings. Answers.com decreased the server count in each of the company’s five data centers from four to one by eliminating the need to shard its databases.
There are side benefits to such moves.
- Any future system expansion that uses the new approach will be able to get by with less hardware, saving money along the way.
- A smaller system is usually easier to manage than a large one, saving in HR costs.
- In some cases fewer software licenses will be required, cutting operating costs appreciably.
- Smaller systems are likely to consume less power than their predecessors.
This last may seem to be a small point until its effect over time is considered. The last three all accumulate over time to become important sums.
Yes, this is a TCO argument, I agree, but it’s a GOOD TCO argument. Although the cost per gigabyte of an SSD may be an order of magnitude greater than that of hard drives, very many systems can save money by deploying these devices. Unfortunately it’s difficult to tell exactly how this will come about until after the SSD is deployed.
As I knuckle down to compile the survey data into a report I’ll be considering the impact of SSDs on the data center, and how the findings of these 21 companies can help others who have not yet adopted the technology.
This exercise has reinforced my position that flash will become widespread in the data center, but not as storage; it will rather be thought of as “something else†that brings cost and power efficiencies that can’t be attained through more established methods.
The StorageMojo take
Jim’s findings surprised me. But thinking a bit I’m surprised that I’m surprised.
Why? Back in ’06 I wrote about the Capacity Illusion, the storage industry equivalent of the economist’s “money illusion.”
The capacity illusion: most customers need IOPS, but they buy capacity. And then they moan about how much unused capacity they have or, in the SSD case, how much SSD capacity costs.
While faster and cheaper SSDs are rewiring data center architectures, something more powerful is needed to rewire customers. A concerted industry effort to classify and promote on IOPS, perhaps?
Courteous comments welcome, of course. Speaking of research, if you haven’t done so yet, please support StorageMojo by completing the Internal Storage Survey. More about it in the previous post. Thanks!
“Answers.com decreased the server count in each of the company’s five data centers from four to one by eliminating the need to shard its databases.”
Do you mean a 4:1 reduction in the total number of physical servers, or literally 4 whole servers down to 1? If the latter, is Answers.com using managed hosting or simply running the servers from the founders’ bedrooms?
Great post Robin.
The capacity illusion is as big an issue today as it was in 2006.
The only thing that has changed is that SSD prices are dropping (slowly but surely) and sub-lun data relocation tools (examples such as IBM EasyTier and EMC FAST and HDS Dynamic Optimizer) are becoming more readily available.
It would be great to hear from an independent analyst what impact this is having on clients ability to buy for capacity (with a small amount of SSD to act like nitrous-oxide) versus buying for IOPS (by spindle count).
Hi Robin,
Great article! This aligns with Nimbus’ research as well. Customers focus on cost/GB first, with cost/IOps further down the list. However, random and unpredictable IO now dominate primary storage-based data. Consider databases, VDI, and virtual servers; these are almost 100% random IO apps, limiting the effectiveness of traditional read caches on HDD-based arrays. As a result, the IO hits spindles. Cost/IOps matters more and more.
The good news is that the historically huge acquisition cost premium for flash-memory storage is shrinking rapidly as well. In the grand total acquisition cost of an enterprise storage system, the hardware cost is only a minority part, with software and maintenance taking a huge chunk. As a result, we have some customers that have been able to make the move to flash storage at approximately the same cost per usable GB as 15K rpm disk-based systems. Despite the higher hardware cost, software and maintenance costs are less, driving down total acquisition cost.
Nimbus launched a survey on this topic one week ago called “2011: The Year of Enterprise SSD”. We plan to release our findings in a couple of weeks’ time. In the interim, the 3-minute survey is still open and I’d like to point your readers to take a look.
http://ow.ly/43yIb
Best regards,
Thomas Isakovich
CEO & Founder
Nimbus Data Systems, Inc.
(good) Flash is a good substitute for systems designed to hold everything in memory. At my last gig they had developed an in house distributed in memory database, primarily for performance reasons, they needed ultra fast response times but didn’t have a whole lot of data (maybe 2TB or so at the peak). I pushed hard to get Fusion IO in there but they re-developed the app to run against our SAN instead, and the results weren’t pretty, the SAN wasn’t configured, built or purchased to do such things so the IOPS were not sufficient. And it was going to take a lot of R&D to change the architecture of the product yet again. They didn’t consult operations when re-designing the product for use on a SAN vs in memory and the end results were quite terrible.
They thought the product as an in memory database was too expensive to operate, what they refused to take into account was the server infrastructure they were running on was nearing 5 years old and we could of easily have collapsed the 50 servers running it into a half dozen servers with either good SSD or plain ol lots of memory (can fit 4TB in 10U on 8 servers…)
But that made too much sense for the company to head down that path so they had to create another train wreck with another strategy…
WRT to SSD I still think enterprise systems will struggle for years to come to compensate on the controller side for the number of IOPS that modern SLC SSD can run. A good example last year(or was it the year before I forget) was VMware running against a bunch of SSDs to drive IOPS, my memory is fuzzy but they had something like 8 or 16 SSDs per storage array and had to have 3-4 storage arrays to drive the aggregate IO.
Enterprise storage systems at least need to have at least an order of magnitude increase in performance to really leverage SSD IMO. From 200-300k IOPS per storage system to 2M-3M IOPS, without an order of magnitude increase in cost btw, maybe make it twice as expensive for 10x the performance.
“While faster and cheaper SSDs are rewiring data center architectures, something more powerful is needed to rewire customers. A concerted industry effort to classify and promote on IOPS, perhaps?”
I seem to remember SanDisk (and others?) talking about “virtual RPMs” on SSDs, which seems like a decent idea if you can make the measure meaningful.
People have been thinking of drive performance at least in part in terms of RPMs for 20+ years now — might as well leverage that.
Most systems are overprovisioned for GB and bandwidth today. It has taken years to change the discussion from bandwidth towards IOPS. But what we really care about is latency. IOPS, like bandwidth, allow simple comparisons for simple systems, but both are trivially parallel. Latency is what matters and the best high-performance storage systems target latency.
The capacity illusion: most customers need IOPS, but they buy capacity.
Except where its not I’m deploying large 120 TB zfs stores weekly. Using every byte. I could care less about 4 kb random IOPs when I have to lay data down at 4 GB/s.
I really really wish data vendors would stop applying IOPs models to every industry. My particular one it doesn’t apply.
Using SSDs for Level2 cache keeping RAM as L1 is the right way to go. Check how SSDs help in building L2ARC for ZFS.
Doing what Nimbus does is just a complete overkill… You can have the same MBPS and IOPS for a fraction of cost.
Anton
Proud member of iSCSI SAN community.
Jeff,
I’d like to know more. Please give us what details you can on the application.
Thanks,
Robin
High end feature entertainment industry. Classic case of the technology has been there for years….and yet isilon still sells their crappy gig-e solutions to unsuspecting clients.
If I’m bitter its because I know that their solutions have put companies in my industry out of business.
Honestly I think that if a CTO isn’t looking at ZFS solutions and support models (if needed from nexenta or somebody) they’re doing their companies a disservice bordering on negligence.
As far as details…really we just have that much data…. a mid range feature film can use anywhere between 50 and 150 TB of data. Keeping it online introduces huge cost savings, if its done right.
Jeff,
Could you please provide a little bit more of details? I’m very interested in knowing what Isilon did (does?) wrong.
Thank you!
Anton
***
If I’m bitter its because I know that their solutions have put companies in my industry out of business.
Low throughput solutions being pawned off as acceptable to start. A focus on IOPs when throughput matters. Where to go from there… I’ve seen several severe data loss scenarios. Days of downtime when staffing overhead with freelancers is extreme.
Watching artists choke through HD or 2K uncompressed frames off of a single gig-e port, because management can’t afford more storage at a price tag that Isilon puts it at, yet because of vendor lock in to a single volume strategy, not being able to break out of the mold.
Last but not least telling me I needed to buy roughly 200K worth of gear to what I ended up engineering for under 20.
I’m not saying they don’t have their purposes in other industries I don’t have experience in, but considering how stupidly they’ve treated mine I have to wonder.
I just hate this large vendor storage lock in that to be honest is roughly 5 years behind on the technology curve, where they spend more money on sales people then on making their technology worth anything.
“Time is Moneyâ€
Robin, I really enjoy reading you blog and would like to comment on Jim Handy’s observations. Disclosure: I am the VP of operations at Virident (www.virident.com), a high-performance PCIe SSD provider. We address the TCO argument for our drives all the time. I felt he correctly identified the TCO benefits these drives bring. However, there is another factor, the “time is money†factor.
In many applications, response time is the most critical element. Certainly it is intuitive in financial services, for example in the high-frequency trading area if you can find a difference in pricing of a stock, bond, commodity, etc. between different exchanges and buy from the lower and sell to the higher before your competitor can, you win.
In the Computer Aided Engineering space, if you can perform your analysis faster, especially iterative ones, you can release a better product sooner. Time-to-market has always been a key economic driver.
Processors have increased in performance at a rapid pace, as they follow Moore’s Law; unfortunately, Disk Drives don’t and haven’t increased in performance at anywhere near those rates. The advent of high-performance SSDs put storage on the same Moore’s Law curve. In these cases, faster is better, which is why PCIe direct attach SSDs, no SAS or SATA interface stealing latency, is the way to go.
As the benefits of reducing the IO bottleneck become more prevalent, I believe you will see the pendulum swing more to the business revenue benefit, not just the cost benefit.
Robin and Jim,
Perhaps I can help illuminate the push-back you are seeing from customers (including myself here) on SSD solutions in general, and Fusion-IO in particular. It’s not that we’re dense.
We’re not pushing back on Flash SSD solutions because they are too expensive on a cost/capacity basis, we’re pushing back because Flash SSD solutions are still too expensive even on a cost/performance basis.
Jim’s assertion, that Flash ‘brings cost and power efficiencies that can’t be attained through more established methods.’ simply hasn’t been shown, and all the hard evidence shows the opposite is usually true.
A look at a pair of nearly identical TPC-H (300GB) results illustrates this perfectly. HP did these benchmarks with only two significant differences between configurations:
– first configuration used 512GB of DRAM, Fusion-IO SSD and a half-dozen spinning disks.
– second configuration used 640GB of DRAM, eight spinning disks and….NO SSD.
The system that did >>not<< use SSD was 20% faster, and 40% cheaper than the "SSD-burdened" configuration.
http://www.tpc.org/results/individual_results/HP/HP_DL580G7_300GB_4P_2.27GHz32C_ES.pdf
http://www.tpc.org/results/individual_results/HP/HP_DL585G7_300GB_4P_2.3GHz12C_100621_ES.pdf
I'll say it a different way. In these two identical tests, taking out the Fusion-IO SSD and replacing it with more DRAM reduced cost and improved performance — more than 40%.
Power savings from Flash? No…there weren't any.
Flash-based SSD will succeed when it is shown to actually improve cost-performance of the total solution. I've yet to encounter a single real-world application that cannot be architected more simply and more cheaply without Flash…by merely optimising the balance between good 'ol DRAM and good 'ol spinning disk.
The mid-hype-cycle popular wisdom around Flash ignores the fact that inserting a third layer between DRAM and Disk introduces new complexity, new failure points and more stuff to procure, manage, update and all the rest. That's the bad thing about adding a third layer to anything — the extent to which the third layer increases costs and complexity sets a very high bar for the performance gains needed to produce an ROI. The rule I go by is "don't ever do anything with more elements than you need".
For credit where credit is due…Dr. Steve Hetzler of IBM's Almaden research was (I think) the first to quantify the problems with "third-layer" Flash economics.
http://www.idema.org/wp-content/downloads/2007.pdf
@KD Mann,
Your TPC comparison are between an Intel Xeon and an AMD Opteron system.
One can make the point that the number of cores balances out architectural performance differences but I would argue that Intel Xeon is a much faster beast than the AMD Opteron especially for TPC type workloads.
Back to the point you were trying to make…
Cost/Performance:
For a few cases, FusionIO can have a huge ROI; a lot depends on if you have a need for that many IOPs and reduced latency.
For most other cases, (those that don’t need that many IOPs or reduced latency) FusionIO style flash systems don’t provide a lot of value.
Aside from the above use cases, there is value in adding a flash cache and/or tiering layer to storage systems. Case in point…
+ Oracle/Sun’s ZFS storage systems’ use of flash as read and write caches.
+ Nimble’s storage systems
In the near term, what I see flash doing is significantly reducing 15K and 10K spindles.
@Jacob,
Thanks for pointing out…the Opteron/SSD system ran 48 threads, the Intel/no-SSD system ran 64 threads. However, this difference is largely accounted for in the raw performance result. What remains is still the >40% advantage in terms of cost-per-query-per-hour for the no-SSD system.
Re: “…Intel Xeon is a much faster beast than the AMD Opteron especially for TPC type workloads.”
I used to think so. But in a pure apples-to-apples comparison using the identical matchup of Opteron vs. Xeon processors (and in this case, also matched up thread-for-thread), the Intel and AMD processors delivered virtually identical performance, on the same TPC-H benchmark:
http://www.tpc.org/tpch/results/tpch_result_detail.asp?id=110070201
http://www.tpc.org/tpch/results/tpch_result_detail.asp?id=110071401
…so I don’t think the argument that a faster Intel proc was responsible for the sans-Flash price/performance improvement can be made.
Regarding Sun/Oracle and Nimble, both are in the business of profiting from Flash (and the former manufactures their own Flash SSDs), so I’m not personally swayed by their enthusiasm around incorporating Flash in their solutions — we are in the middle of a Hype-Cycle after all, and there is money to be made.
The point I was making is that customers are still looking for hard, quantifyable examples of ROI for Flash — huge or otherwise. I think that’s what the customers that Jim Handy surveyed are trying to convey.
Just a few days ago, SNIA finalized their SSD Performance Test Specification. It was Fusion-IO’s very own Jonathan Thatcher who wrote the original proposal for this, and (quite eloquently) made the case for a uniform benchmark that we can architect solutions around. One can only hope that Fusion-IO will also be the first to publish a result. This will go a long way toward helping customers quantify the real-world cost/performance benefits of Flash.
http://www.theregister.co.uk/2011/05/24/snia_ssd_pts/
@KD Mann,
Your new TPC comparison is two 6core Intel Xeon 3.33Ghzs (with hyperthreading) vs two 12core AMD Opteron 2.3Ghzs.
+ Depending on workloads, Hyperthreading can be 10 to 90% the performance of another core. For computationally intensive workloads, the performance is much closer to 10% of another core.
+ There is a significant clock disparity between the two systems.
Let me give a counter example.
TPC-C Top Ten Price/Performance Results:
http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp
+ The three that list Watts/KtpmC (power efficiency of system) all use SSD.
+ Of the top 11 systems listed, 7 use SSDs to store the database.
+ The 4 that do not are entries from 2009 or earlier and use close to 200 or more spindles to achieve their result.
Still think Flash can’t be justified for at least some database applications?