I/Os Per Second
5,000,000+

Capacity
100 TB of Flash

Bandwidth
60 GB per second

Latency
Writes: 80 microseconds
Reads: 250 microseconds

Sun has won few TV stations with their Unified Storage into editing sections with Appel FCP or Avid. A single 7410 with 128GB ram with few SSD head can do 1.2 to 5.5GB/sec depending file size and number of disks behind. No complex setup like QFS or any similar disk sharing here. Huge saving.

So with that said I’m about to embark on a dyi with SSD. I’m projecting 6 workstations at 10 bit 4K with intel SSD’s. One storage vendor put a price tag on that throughput at north of half a million… the DYI at under 40.

Panasas is remarkably affordable these days, offering most of the benefits of Lustre in a turnkey appliance form factor, without the baggage. At the prices Panasas is able to offer, I’d suggest you consider only tiering off old data, and do it in a separate silo: i.e, don’t try to get fancy. Caveat, they have no Mac driver, and their Windows driver is not QUITE out yet.

DDN is entering this market with a clustered storage appliance also. It, too, will be an easy to use turnkey clustered storage appliance…

Others have pointed out Isilon, which is the easiest to use of the bunch, but cannot approach the performance requirements that the first two can. Note that if you don’t need the highest performance that Panasas and Lustre offer you, you should do some thinkin’: when Isilon says they are the easiest enterprise storage to deploy and use, they ain’t lyin’.

As for ILM, I might suggest you just go for some software that moves older files over to something cheap when the time is right. I think IBM will happily sell you TSM for that… and it’s cheap.

So, make your own:

Put Panasas, Isilon, DDN in Tier 1
Put a Tape Robot (Sun, IBM, Quantum/ADIC) in Tier 3

Notice I do not list a Tier 2. I do NOT think this is needed. I believe you can negotiate Panasas, Isilon, DDN, and the like, to be close enough to Tier 2 prices so as to not need a classical Tier 2 at all…

Joe.

The data volumes here are seriously crazy – like my 12MP DSLR taking 30 frames a second. At 1.2GBps per second then that’s over 1TB just for 15 minutes.

But this also shows the problem with spinning disks – 1.2GBps would take perhaps 20 of them in parallel to have a chance of sustaining just one stream. Pulling 60MBps reliably off of 20 devices all at the same time is difficult enough. Add into that contention through multiple streams, small random I/Os for file system, meta-data and so on and there would have to be many dozens of disks. I suspect that the strain of multiple streams and all the buffering going on could also saturate the memory bandwidth available.

Ahead of pNFS if somebody can invent some form of layer that allows for the interleaving of files over multiple, independent links and storage devices then I suppose the problem might be soluble using several commodity NAS units whilst allowing for full sharing. Using a file-based protocol rather than block-based at least makes the shared access easier. I suspect it is fairly easy to kludge together, but whether there is enough CPU horsepower in a client machine to do all the buffer shuffling at that speed, I don’t know. Also there are probably single threaded parts of client IP stacks that could throttle the whole thing, even if there were multiple targets.

Steve, I don’t see how you’d get multiple 1.2GB/s streams out of Sun Unified Storage (short of time-traveling into the future to get pNFS).

- Trying it out is a simple software download and install on any commodity hardware running Linux. You can try us out by downloading a free 4 TB cloud at http://www.parascale.com, and grow it to PB scale on demand. You choose your favorite hardware vendor and can select any hw configuration and SATA/SAS/FC based on your performance requirements/cost. You purchase any server that can run RHEL or CentOS (e.g. SuperMicro xxx, white boxes). You can also mix and match servers, repurposing servers into the cloud if needed.

- The cloud offers a single namespace and every node within the cloud can serve client requests independently. This is not a gateway implementation and there is no chokepoint.
- The cloud software performs load balancing automatically, eliminating the need for an external load balancer. Each user who uses NFS will get redirected to a different part of the cloud and the load will be spread across the cloud.
- We work over regular bonded interfaces, and so the network will also not be a bottleneck with our solution. GE, 10gE, IB etc – whatever Linux supports we support
- We simplify managing PBs of data by supporting policy based management operations like automatic data migration, capacity balancing and replication (if required). All this is included in the free download, and you can try it without having to install any add-ons. If you are familiar with Redhat Linux or CentOS admin, that’s all that is required.
- We are optimized for large file reads/writes
- We do not support CIFS today – but windows clients can access the cloud via WebDav quite effectively. We have also found Windows NFS clients to be quite effective and with good performance.
- You can eliminate “forklift” from your vocabulary. The ParaScale software provides a layer of persistence, permanence. Underneath this software layer you can add new servers as old ones die. The cloud continues with the latest and greatest hardware out there.

It might also be that some of the built-in functionality like snapshoting and will be useful too and the HA clustering might be important too.

Archival software is a different issue – I’m old fashioned enough to like my layering approaches clean and tend to want my basic storage devices do that well rather than try and perform too many functions. When it comes down to it, a storage device offering a standardised interface can be swapped out – something which has lots of added functionality can lock you in.

First, they need up to 1200 MB/s per user, using Linux, Windows, and OS-X. Up to a dozen people now (potentially 14.4GB/s, not 1.2GB/s), possibly larger. So smaller bandwidth/non-scalable bandwidth really need not apply. This (strongly) suggests a cluster storage system, not a single box.

Second, they need 100TB+ now.

Third, they need global namespace … and direct access for Windows … which is the odd one out in terms of client support. Lustre is mentioned.

Ok, first, the only way you are going to hit these bandwidths is with a distributed model. I am talking the 14GB/s with 12 users simultaneously slamming on the 1200 MB/s.

Second, don’t disparage Infiniband, just because your preferred vendor/solution doesn’t include it. QDR IB can handle these rates, and you can put it on desktop units. We do. You can do longer runs with DDR IB and fibre … we quoted this out for a customer. A bit more expensive than CX4, but if you need it, you can have it. Moreover, you can channel bond the IB and do multi-rail stuff. IB is the indicated solution, likely with the fibre links. ConnectX or Qlogic cards in the desktops. Nice Mellanox/Qlogic based switch doing QDR on the back end. Add in a 10GbE blade into the frame for multi-protocol switching if you want to tie this into your gigabit ethernet network (don’t go there, 110 MB/s won’t cut it).

Third, global namespace (e.g. single file system) suggests a distributed or centralized metadata service. One may be better than others for this use case. I’d argue for distributed in this scenario, and that suggests solutions like GlusterFS. No current windows client, ask Anand if one is planned. Lustre is possible, but again, that windows client gets in the way. And I should point out that Lustre does not require a team of Ph.D.s to administer it.

Ok, hardware side. FWIW, we have designed a number of storage clusters as of late, including smaller sustained 4 and 8 GB/s performance using Lustre and GlusterFS respectively. There is nothing magical about this, other than using good hardware as the basis. We have units that will provide 96TB raw in 5U at ~2GB/s per unit, that you can use as the basis for this (see this link for details), without breaking budgets, or causing CFOs to go catatonic.

All told, if this person wants to solve this, their best solution would likely look a great deal like a GlusterFS cluster, followed closely by a Lustre cluster. Its just that windows issue. Thats going to be the hard aspect to this. There is no simple solution … possibly apart from running windows in a VM on a muscular Linux desktop with an IB card in it. Not perfect, but much better than “No”. And for the highly reluctant pure windows users, you can have the bootup log in to a VM user, and fire up VMWare workstation right away, so it looks simply like a slow booting, but otherwise, very fast windows box.

Just my thoughts, and of course I am biased given who I work for …