Choice is a great thing, unless there’s too much of it. And choice is what we have a lot of in today’s data storage market.

A longtime StorageMojo reader has an interesting problem: architect a 3PB data storage facility. Can you help?

Here’s what he wrote to StorageMojo. His email has been slightly edited for clarity and length.

One of my current problems is to design one of the nodes for a large research data storage facility. I’ve had to do this stuff in varying degrees, varying modalities and varying tech in times gone by.

I’ve been given a number and “capacity” to look into – somewhere near or around 3PB to begin with. We won’t even go down the path of discussing workloads or disk technology fit for purpose at this stage, but, something has struck me as interesting.

There is this clear divergence in disk technologies at the moment and I’m finding it hard to resolve what is the “right” one of the task.

Currently, I see:

  • Heavy-end storage virtualisation frames [VSP, Symmetrix et al]
  • Big grid-ish things [IBM XIV etc]
  • Weird “stacked” commodity LSI Silicon [NetApp E5400/5500, SGI IS5500/IS5600, Dell MD3660F etc – all the same silicon I think?!]
  • Quasi virtualisation arrays with modular form factors (Hitachi’s HUS-VM?)
  • High performance dense trays in modular form factors [DDN’s SFA-12K Exa and Grid scaler tech?]
  • Bog-standard performance dense trays in modular form factors [Hitachi HUS, EMC VNX, HP EVA, Dell compellent etc etc]
  • That wild crazy pure flash/RAM/SSD/NAND world that guys like Violin inhabit.

Currently I’m trying to rationalise what I should be using for a storage platform that needs to scale big, but do it in a sensible economic standpoint, with density, performance of interconnect and throughput with gross mixed workloads being all big factors.

Some folks suggest to me that I should be happy enough with the LSI horizontally stacked 60-drive trays, but I am not sure the technology is tracking too well in terms of performance or density (Hitachi, DDN and maybe some others can now do 84 drives in as little as 4-RU!).

I guess my question to you is – where do you see that dense high performance market heading? I know the guys at the LLNL over your way were crowing about the NetApp E5400 LSI stuff where they managed their “1TB/sec” file system (I think it was Lustre based?), but I have to wonder if that could have been more efficiently carried out using a DDN GridScaler/SFA-12K-E etc.

The StorageMojo take
Two issues here: is the segmentation our correspondent offers realistic and helpful? And what are the core architectural issues he needs to think about?

For the first issue an object store or a highly parallel NFS – like Panasas – seems to be indicated.

Given that this is a general purpose high-performance system, the critical problem seems to be how the system – however architected – handles file creation/update/deletion metadata. String enough disks together – 1,000 to 2,000 – and you can get a reasonable # of IOPS and, if you need more, put some SSDs in front.

There are a number of scale-out storage systems that will credibly and economically grow to 3PB. Metadata is often the bottleneck, as Isilon buyers have found when creating many small files.

A maximum performance spec – including file creation etc. rates – will probably help eliminate likely laggards, while a budget $ per usable TB/PB will eliminate the uneconomic products.

Vendors are welcome to offer their perspectives. Please just identify your company so we know where you’re coming from.

Practitioners who’ve done this, or something similar, are encouraged to share their hard-earned wisdom. 3PB is non-trivial today.

Courteous comments welcome, of course. I’m going to start offering almost-free consulting for end-users. Stay tuned!