Enterprise Storage On a Shoestring, Pt. I

by Robin Harris on Wednesday, 29 November, 2006

Is it really possible to build enterprise-class storage using cheap commodity products? Some folks at HP certainly think so. They call their design FAB, short for a Federated Array of Bricks. Ignatz, meet Krazy Kat. Of course, HP is only the world’s largest data storage company, so what would they know?

And what is a brick? A server with disks. Instead of $10/GB, perhaps $1/GB? I’d love to see Storage Markets ask when this will come to market. Since HP doesn’t actually build big storage arrays – they OEM from Hitachi – they could do this. If they really want to take the data center away from IBM they have to do this.

Let me count the ways. . . .
Google has *almost* created enterprise-class storage from commodities. Microsoft has Boxwood. Amazon has, but they aren’t telling. Now HP has people saying “we can build enterprise class storage from commodity components.” And they’ve done it. They said it, and a lot more, in this paper, FAB: enterprise storage systems on a shoestring (pdf), by Svend Frølund, Arif Merchant, Yasushi Saito, Susan Spence and Alistair Veitch.

FAB: the mission
The authors begin by noting what should be obvious to everyone in the storage business: enterprise disk arrays are expensive. Why?

. . . disk arrays are constructed from customized, very reliable, hot swappable hardware components. Designing and building the hardware components is time-consuming and expensive, and this, coupled with relatively low manufacturing volumes, is a major factor in the high price of storage systems — high-end arrays retail for many millions of dollars.

Another cost factor, and problem for customers, is the lack of scalability of a single system. There is a high up-front cost for even a minimally configured array, and a single system is limited in both capacity and throughput. Many customers exceed these limits, resulting in poor performance or a requirement to purchase multiple systems, both of which increase management costs. The lack of scalability forces manufacturers to build multiple products, or even entire product lines, each targetted at different system scales. For example, Hewlett-Packard sells three different array lines, each of which effectively multiplies the engineering effort required — for hardware and firmware development, for integration and testing, etc, costs which are reflected in the price paid for each system.

Which doesn’t even mention the storage management nightmare this crazy-quilt engenders.

FAB: the hardware
Frølund et. al. set a high bar for enterprise-class storage:

. . . absolute reliability: a disk array must never lose data or stop serving data, under any circumstances short of complete disaster.

Using commodity components – a rack mount box containing disks, CPU, NVRAM and network cards – a FAB is a decentralized cluster that presents logical volumes to servers across Ethernet or Fibre Channel. The FAB interconnect is Ethernet, cheap and, with the arrival of 10Gb Ethernet, plenty fast. And that’s about if for the hardware. The magic is in the software.

FAB: the software
FAB is a decentralized cluster and clusters have challenges that uniprocessor or SMP systems – i.e. standard storage arrays – don’t. These challenges include:

  • Failure tolerance: commodity hardware is less reliable than enterprise hardware, so FAB must handle hardware failure gracefully.
  • Single-system view: FAB needs to look like a single array to the server, even though there is no centralized software that manages the presentation of FAB to servers.
  • Asynchronous operation: Timeliness cannot be assumed, so data replication must be coordinated without assumptions about speed or connectivity.
  • Hardware independence: Brick technology changes often, so the brick’s rev level or storage capacity must be discovered, not assumed, and then used appropriately.

Part II of Enterprise Storage On a Shoestring coming soon. Comments welcome, of course, with moderation turned on only to deter comment spam.

{ 1 trackback }

Federated Array of Bricks - Deja Vu? at Oracloid Blog
Saturday, 5 July, 2008 at 5:58 pm

{ 5 comments… read them below or add one }

Robin Harris Thursday, 30 November, 2006 at 8:13 am

WordPress commenting burped, so loyal reader Lothar sent me this comment:

” Very interesting read – as usual i might add ;-) being a software developer in java land i would predict a bright future for this approach. in java development there exist some products (terracotta, tangosol and some others) which take (some of) the complexity out of distributed app development by offering APIs and taking care of the heavy lifting (sync, failover, etc.). exactly what you are talking about here. so i think probably the biggest win will be reduced complexity, the cheaper price tag is more of an extra.”

Lothar, the flattery thing definitely works (see Kevin Closson is a Golden God! Thanks!


Dave Holland Tuesday, 5 December, 2006 at 7:18 am


that paper was presented at HOTOS03. Do you know if the prototype implementation has gone any further since then?

Robin Harris Tuesday, 5 December, 2006 at 1:23 pm


I sent the authors a note to alert them to the posts, so let’s see if ther respond.

They have done some more recent work on a new file system and some other stuff that I’ve been looking at.


Kemp Watson Wednesday, 6 December, 2006 at 10:00 am

Sounds great, but why wait? It’s available right now from Zetera.

jay Monday, 21 May, 2007 at 8:57 am

check out PetaBox (http://www.archive.org/web/petabox.php). It is commodity storage solution spun off from Internet Archive.

Leave a Comment

Previous post:

Next post: