A recent post on the dumb disk fallacy argues that enterprise storage isn’t overpriced. That misses the point: enterprise arrays may not be overpriced – but they overshoot most market requirements.
That’s why there’s so much innovation in the high capacity/low cost end of the market. And why high-end monolithic arrays are the mainframes of tomorrow.
Background
Disk drives are only 5-10% of an array’s cost. Since dual-parity arrays protect against 2 drive failure/read errors, the logical question is “why not just replicate the data 3x for 15-30% of the array’s cost and get better protection?”
Comparing bare drive costs to array costs isn’t realistic. Power supplies, chassis, cabinets, processors, interfaces and firmware all cost money.
Servers have it all
Which is why I prefer to look at server costs – especially servers with lots of disk slots. Like a Supermicro Superchassis 846E1-R710B 4U 24-bay storage server.
For $5500 you get:
- A 4U storage 24-drive chassis w/ redundant power & cooling, 2 GigE ports
- Mobo with dual quad-core Xeon 2.5 GHz processors and 16 GB ECC RAM
- 24 1 TB 7200 rpm SATA drives
- Dual 12 channel PCIx RAID cards
- Support for 15k SAS drives if desired
Update: Some folks asked where that pricing came from. It is from Priority Computer & Networks. I have no experience with them, but I like the nifty online configurator. End update.
The drives are 40% of the configuration’s cost. And I’m sure you could do better. Throw Linux or OpenSolaris storage stacks on the box for free, or, for example, Nexenta’s Enterprise Gold Edition for another $4k and you’ve got a nifty NAS/iSCSI array.
Buy several and layer HP’s IBRIX or ParaScale software on them and you’ve got a scalable file cluster with redundancy and performance for way less than monolithic enterprise arrays. Or even mid-range modular arrays.
When are enterprise arrays better?
Enterprise arrays aren’t designed to compete with Supermicro boxes and FOSS. They offer benefits way beyond commodity-based storage – but at a price.
- Performance. Big redundant write caches are perfect for transactional apps – but the corner cases make engineering them a nightmare.
- Scale-up architectures. Embedded switches, star networks, high-performance FPGA controllers, FC and Infiniband – all the hot-rod, go-fast, low-volume tech give big arrays a scale-up capability that enterprise IT likes – until its gone.
- Bullet-proof hardware and software. Years of tweaks to the exception-handling and careful drive qual and firmware control makes these systems more reliable.
- Layered software. Database and email backup, LAN and WAN replication and all the other options give enterprise IT a warm feeling and operational flexibility.
So what’s the problem?
On the Supermicro the storage is $0.50/GB, while on the enterprise array it’s $5 GB or more. And enterprises can’t afford that for everything.
As intuition suggests and recent research confirms data is getting cooler. We store more and more and access it less and less.
Translation: we need cheap capacity, not high performance. And as Google proved years ago, just because it’s cheap doesn’t mean it is slow.
The StorageMojo take
I don’t know if enterprise arrays are “overpriced” – if people buy ’em, how bad can they be? – but I am confident that they overshoot the performance requirements of most applications by a wide margin. And that performance is very expensive.
But enterprise IT learned long ago that it is better to be overconfigured than to be caught short. And in good times who cares?
But times aren’t good and despite the Q3 numbers we haven’t seen the end of the Great Recession. The good old days aren’t coming back.
The point behind comparing array prices to dumb disks is to remind people that there might be a better way to achieve their goals than spending $4.75 per GB on performance that most of their data doesn’t need.
If it is redundancy you’re after there are better alternatives than RAID 6. With the arrival next year of pNFS and the commoditization of 10GigE we have many more options for high performance at a low cost per GB than ever before.
Courteous comments welcome, of course. I once worked for Sun and have done work for IBRIX and Parascale.
Absolutely agree!
If you add the fact that some large scale cloud storage computing will rock the world one day (we hope). They are NOT running or using big iron such as EMC or HDS. Some are using clustered storage software like you describe and using cheap SATA disks combine with SSD for faster data retrieval (cache).
Some file systems allow data movement between SSD and SATA without having to write code, buy software or setup complex data movement schema…(Ex: ZFS, Oracle SAM). Totally free!
No more storage tiering technology options or requirement since you have both level of performance without having to do anything. That force high end storage vendors to be caution about their software options at big expenses. They can’t beat Free…
Apps NEED to be MORE data storage aware like MS Exchange and Oracle did recently. These two work great on straight JBOD…and data is protected too if setup properly…They no longer rely on high end storage to be resilient and have storage replicated safely under their own controlled environment. Free!
How tape will be used in cloud storage? It will be highly challenged but it is feasible with changes in apps behavior of course.
I’m a strong believer that tape need to be around for safety. I’ve seen to many users and configuration mistake in my carrier life. Tape remain the last storage solution against this type of human error…
Worked at Sun/StorageTek for 25+ years.
http://www.thematrixstore.com (pre-production/nearline disk based archive in Media space) is built on the Supermicro platform. Off the shelf quality components that are more than appropriate to provide a solid, reliable node in a self-healing, redundant clustered architecture.
High end monolithic arrays are the mainframes of tomorrow? I thought they were the mainframes of yesterday, maybe high end modular arrays are mainframes of tomorrow..which may be true, at least for the ones that don’t have a reasonable entry level price(say sub $150k) that allows you to grow without rip & replace.
I think enterprise modular arrays will be around for a while simply because it’s not a simple feat to integrate real storage intelligence into apps, especially not when you consider you can go buy such intelligence and apply at at a more generic level rather than for each and every app. So most app vendors would rather focus on their app and let someone else focus on the storage, there are exceptions of course Oracle being the biggest, but for the price you pay for Oracle EE you can have yourself a bad ass enterprise storage array. I’m one to vote for go for Oracle SE, and get more/better hardware and silo your applications across multiple DBs if you really must scale beyond 4 sockets, 4 sockets with multi core processors is an amazing amount of horsepower for a DB. Just a few years ago I worked at a place with the largest OLTP DB in the world(probably still is), and it ran on a 8-way Itanium, a 24-core Opteron would run circles around it. And early next year you’ll be able to get 48-core Opterons in 4 sockets. Or go with two 2-socket systems and run them in RAC, since with Oracle SE RAC is free.
Robin, The future is inexpensive servers and smart software. No doubt. Another thing for customers to consider is how much they want invested in an individual node and if that node should have 2 drives (seems to be Google’s approach), or 4, 8, 12, 16, 24 or more. Some attention should be paid to both the speed of node repair/replacement/upgrade. This comes in to play when you have node failures, but also in the course of doing business when you have to upgrade nodes on the fly. The smaller the drive count the easier it becomes to NOT rely on any single node. And 2+TB drives help on the cost/capacity front! Since many of these overall systems (I’m talking applications here) will last far longer than the individual life of any one server, folks should think about how the system will survive 2-3 complete server hardware upgrades while online.
I have hands on experience with SuperMicro 846A-R1200B/846A-R900B chassis combined with 3ware 9650SE-24M8 raid controllers (with battery backed write cache). If you want RAID6, I think this combination kind of defines the price/performance/reliability/robustness sweet spot.
We like to think that in the future this or that solution will “win”. Not the case, as you pointed out. Discussion about which storage solution is best sounds very much like today’s Healthcare debate.
One goal is universal coverage – “good enough” solutions that reach every byte. For those that are willing & able, there are higher cost options that provide much more complete performance and protection.
While it would be nirvana if every data set could afford to be on the best and fastest solution, the math just doesn’t work.
“On the Supermicro the storage is $0.50/GB, while on the enterprise array it’s $5 GB or more. And enterprises can’t afford that for everything.”
While I agree with the tone and tenor of this article, if one is paying more than $2.00/GB(usable) for one’s enterprise tier 2 SATA storage, one is paying too much, at least in any reasonably large deployment.
Joe Kraska
San Diego CA
USA
Robin:
You hit the problem right on. That’s exactly how NetApp and Sun’s open storage is going, using commodity servers with SAS adapters connected to loads of cheap SATA drives through SAS Expanders.
I am in love with ZFS. Unfortunately, Sun’s Open Storage strategy is being killed by their non-transparent marketing and advertising, and most Sun people are in denial. If Sun allowed Fishworks software to be purchased separately and installed on any SunFire X86 or Sun Sparc Machines, the open storage movement would instantaneously capture a majority of the enterprise storage market. Instead, as I have said before, it is ruined by “closed” hardware design choices, and basically non-realistic list prices.(Even though nobody pays list prices, I would prefer to have the actual price listed. )
Case in point, the 7310 is listed on Sun’s website for $40K for a single head node, here, $75K for a HA pair, and 111K for a loaded HA pair
http://www.sun.com/storage/disk_systems/unified_storage/7310/,
People can actually get the 7310 from resellers for $12.5K each from CDW.
http://www.cdw.com/shop/products/default.aspx?EDC=1897910
That is nearly 70% off of list prices. And that’s still not good enough. For people with a brain, they know that the 7310 is just a SunFire X4140, selling on CDW for as low as $2.5K
http://www.cdw.com/shop/products/default.aspx?EDC=1546283
On the storage front, the J4400 JBOD is only $3.6K
http://www.cdw.com/shop/products/default.aspx?EDC=1554430
Yet, filled with 22x 1TB Hitachi UltraStars + a couple of stecs,it suddenly becomes $32K
http://www.cdw.com/shop/products/default.aspx?EDC=1690801
People do know how much the 1TB drives are worth roughly $200 each for the Hitachi UltraStars. 1TB Western Digital and Seagates SATA can be had for $90 if you look around.
What it does is that it forces people to adopt NexentaStor’s OpenStorage appliance software, even though Nexenta piggy backs off of ZFS and Solaris Engineering costs.
So sad. I feel bad for ZFS engineering team. It is absolutely the greatest engineering feat, ruined by moronic MBAs. If Sun would wise up and truly open up the Fishworks software. All they have to do is sell the Fishworks as a software upgrade($2000) to any Sun Server, it would take off very quickly.
Aren’t the big guys already doing this? XiV from IBM is eactly like this (in fact using supermicro computers inside it). EMC’s Atomos, and HP/Oracle’s Cloud offering.
BTW, if one as a buyer is not comfortable with going the the second tier (SuperMicro, etc), one would do oneself a favor in talking to the Dell DCS folks. They only take calls from folks who buy in multirack commitments, but if you’re this kind of buyer and need a tier 1 partner, Dell DCS is quite interesting. As a hint, you’re going to find some very competitive commodity based solutions over yonder…
Exactly! Enterprise arrays were designed for unstructured data (database content). It was all about high reliability for small capacity (less than 2TB). So enterprise storage vendors focused on solving the availability and performance problems at the hardware level (FC for low-latency, redundant RAID controllers for availability.. ). There is only so much you can do at the hardware level. As you keep adding hot-swap memory or hot-swap CPUs, it gets incredibly expensive.
Today’s storage requirements are way different. Most of the data is unstructured (office documents, images, music and video). 100TB is normal to start with. Some applications demand beyond PBs. There is no way enterprise storage array vendors can scale their technology upwards (scale-up). There are physical limitations to number of CPUs, memory or I/O ports a single chassis can take.
Answer is to scale at the software layer on top of redundant array of inexpensive comodity servers. Let software do the job. It is scalable to multi Petabytes.
2 weeks before we deployed GlusterFS on 8 server storage cluster of 142TB capacity and 384GB memory cache. It delivered 12Gigabytes/s read throughput and 8Gigabytes/s write throughput with iozone benchmarking. 1000 clients pounded on the storage cluster.
$10K for Fishworks doesn’t sound that bad; wouldn’t an equivalent Data OnTap license cost over $50K?
I would appreciate it if you could provide a link to or source for the Supermicro configuration you quoted. I certainly don’t doubt your pricing, I’m just lazy & it would help to know where you got it. Supermicro direct, or….?
Keep up the good work, thanks.
—
David Strom
I’ll post the source after I’m back home with the browser history.
Robin
On the open source end, Matthew Dillon started DragonFly BSD in 2003 because he believed that clusters of servers was the future. He built a new file system designed for mirroring between servers.
Sounds like the secret is getting out.
We run high end transaction systems running millions of revenue transactions a day with millions of online accounts.
We’ve used EMC and Netapps, but recently investigated “open storage.” running on 10GBe iscsi. Our benchmarks show that open storage on 10Gbe iscsi is comparable to EMC/Netapps. It is also a fltat architecture without the bottleneck of the storage processor. With the modern mirroring technologies provided by VM OEMs and by Database vendors, we get HA from them anyway. We can upgrade our hard drives and storage cards very cheaply – just swap them out.
From a cost perspective, the lifecyle cost of open storage is 1/15th that of closed systems. Its also less cost to maintain because any good SA can build these boxes or support them – its just a server with a lot of drives.
We moved all our non-prod systems to open storage and all our new production systems are now on open storage. It has saved us a ton of money and buying storage is now a signature process, not a procurement process.
I do recommend going with established open storage vendors as they have accumulated expertise with these systems. They wills save you some time and if something is messed up, they will fix it – otherwise you are on your own.
These are the ones we have used.
http://www.emdstorage.com/
http://www.aberdeeninc.com/