http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

1) IBM’s SONAS (Scale Out NAS): infinibanded HPC derived clusters with DDN storage on the back-end. Just saw these with my own eyes last week — way fast, way scalable, way efficient, way cool

2) IBM’s SAN Volume Controller; especially now that they’re able to license per spindle instead of per terabyte. SVC’s 380,000 IOPS in SPC-1 is almost twice as fast as anything else ever tested on spinning disks, and is even about 30% faster than the big SSD arrays recently tested by both IBM and TMS. All that, and you can even put your existing spindle-farm underneath it.

Finally — wouldn’t Isilon be DQ’d here on performance? Isilon is all about cheap, massive capacity across a single namespace, but Isilon performance is not anywhere near the rest of the solutions discussed here.

I would definitely look at SSDs, since you have a mountain of metadata to churn through before you can get to your petabyte. Of course you’re talking to FIO. They love science experiments. Their ioDrive cards are the IOPS kings at the moment, as their pricing suggests. One caveat. FIO’s model of “an ioDrive in every server” is going to be at odds with your existing environment running SAM-Q on Solaris. Better to look at stuffing a box with a number of SSDs (disk form factor, PCIe, etc.), then present them out as storage target LUNs for your SAM-Q server to use for metadata store. You want to disrupt your fragile HSM server as little as possible. This way, you can remain flexible with respect to SSD adoption and take advantage of the SSD’s reapidly falling price/(capacity:performance) curve.

Big Physics Centre CERN did a study on this, and on 3000 hardware raid rack servers, they found 152 instances of bit rot, where the data was altered without the hardware even knowing this! The sysadmins didnt get noticed. CERN discovered this by using a program that wrote a known bit pattern and then compared to the expected result.

All hardware solutions have some rudimentary protection against bit rot and silent corruption, but no one protects completely, except ZFS. Here is two more articles for you to read, if you want to learn more about CERN and bit rot (he concludes end-to-end checksums are needed, ordinary checksums will not do – he suggests ZFS)
https://indico.desy.de/contributionDisplay.py?contribId=65&sessionId=42&confId=257

ZFS is designed from scratch, to NEVER EVER trust the underlying hardware (cosmic radiation might flip a bit, power spike, bugs in BIOS, not really connected card slots, etc etc):
http://queue.acm.org/detail.cfm?id=1317400

In my opinion, ZFS protection against bit rot is THE main reason to use ZFS. Why use fast and unreliable storage? Better to use safe storage which guarantees that your bits are not altered. Read those links for more information.

Finally, there are other ways you can reduce the cost of your infrastructure while using inexpensive SATA drives such as deduplication and compression. There are two companies that do a really good job at this independent of the file based storage you have, Ocarina Networks and Storwize. Ocarina is a post process for more static content and is able to optimize images along with text files and other precompressed files. Storewize is an inline compression engine that is optimal with a variety of file excluding precompressed ones. By using these technologies, you can reduce the footprint of data and the cost of the storage and its environment.

So to not to forget though….Open ZFS has been adopted by a few vendors who might be good for tier two including Greenbytes who added deduplication inline and Nexenta. Might be an interesting option.

First off, we are a vendor. We build very nice, dense, and fast storage boxen/targets/storage clusters and systems for this sort of work. We deliver and sustain data rates that are quite good (e.g. non-marketing numbers, real, end-user-repeatable results). We have units in production that easily supply more than 10GB/s to large HPC cluster systems, including for informatics analysis applications.

The issues I see being requested to be addressed are 1) existing meta data servers are barely able to keep up with load, 2) a need to expand the capacity (increase the density?) . I don’t see the tape as an issue, event though it is listed as a problem.

Other comments seem to suggest replacing the existing infrastructure with something new. Choose your flavor and go forth or something like this.

If the issue is to solve #1 and #2, this is easy to do without replacing much. The question I would have is how this would scale going forward. If you see going from 1PB to 10PB as problematic, then a new architecture is probably not a bad idea. More on that in a moment.

Question #2 is “solvable” by keeping the cost of additional nodes down. This may be at odds with some vendor solutions (not ours).

Question #1 is “solvable” by replacing the MDS in the current design with something faster. We just benchmarked one of our JackRabbit-Flash units, with 1k random reads against 256GB of data at a sustained 180k IOPs. This isn’t a terribly expensive unit, and its flash drives obviously run circles around the 15k RPM drives. You can’t repeal physics; mechanics will not be as fast as electronics in most cases that I am aware of.

Ok. Onto design issues. If as you scale up, the MDS is only going to get worse (as all centralized designs will), then replacing it provides only a bandaid over the issue, and avoids solving the real problem, that of good design. #2 isn’t affected as much as #1 on the design side. Bulk data storage should be lower cost and fast. But, if you have a single point of information flow in your scale out process, your design will eventually fall over.

So if you do plan to scale up well beyond 1PB, the centralized MDS has got to go (and any design that utilizes a centralized MDS is likely to have the same issues during scale up). Here things like Gluster (which we sell/support/integrate into our offerings) and a few others make a great deal of sense. You scale up as you need, with reasonable economics.

Feel free to ping me on/offline if you need to talk about these designs. Basically, if you are not trashing your existing infrastructure, you need to have a clear conception of how much higher it can scale, and whether or not an SSD replacement will help your MDS for your planned future. If you really do need to scale up/out, our siCluster (info to appear soon at http://scalableinformatics.com/sicluster) product is certainly one worthy of consideration, providing some of the best end user achievable scale-out performance we have seen on customer applications to date.

As Rob of Xiotech mentioned their ISE 5000 is an awesome array, and adding some DataCore SANsymphony to complement it you’re increasing your performance even more. If you want to add SSD, you can do that today, just connect up SSDs from any of the vendors to your commodity server hardware running Intel or AMD and you can use things like STEC’s 3Gb or 6Gb SAS connected SSDs, or their 4Gb FC SSD. Pliant just released some and Intel has them as well. With DataCore, you can pool all those and any other disk type from any vendor into as many pools and tiers as you’d like.

Just as the CPU has evolved over the years from vacuum tubes to silicon to a small internal cache to now three layers of cache, look at storage in the same way with DataCore’s software being your fastest Layer 1 cache. The software will not slow things down but speed them up, typically taking up to an order of magnitude of I/O latency off over a normal cached array controller. There are many more features and capabilities of the software than I’ll write about here although if you’re interested in finding a solution that offers you the scalability, performance, ability to fit into your exiting storage environment (DataCore can present storage to any Open System host) and doesn’t tie you to a monolithic stack of storage, you should look at DataCore.

Dave Brown
dave.brown@datacore.com

IBM Information Archive appliance supports both compression and deduplication of files archived. This provides from 20-80% reduction in storage size. The compression and deduplication is performed either at the client or upon ingest to the archive through up to (3) 8-core IBM System X servers, providing plenty of horsepower for the job.

Paul Hewitt
IBM Data Archive Institute

Having gone to the effort of implementing an HSM deployment (easier said than done), one way to take advantage of that is to optimize your tier 2. You can get more cost-savings than just using cheap SATA disk. For scientific data sets – such as next-gen sequencing, mass spectrometry, images that come off of many types of instrument – it is possible to integrate content-aware compression transparently in the second (SATA) tier.

This kind of compression recognizes specific file types and can get up to 75% compression at close to wire speed performance. This won’t solve your IOPS problem for the metdata slices, but saving 75% for a petabyte of storage at that tier could help you find the money to buy the cool stuff for tier one. If your data looks like TIFFs, SAM/BAM/SRF for genomics, or any other scientific image or coded data set, this would be worth looking in to. If it’s just alphanumeric, then generic compressors (such as those in ZFS) could be turned on.

As it happens, we do have some experience with the AMS 2500. I can’t compare it directly to the STK models, which I believe are made for Sun by LSI. We’ve found the HDS a bit difficult to get configured and ordered, but once it is in place, it’s rock solid. Super highly available, no disparity between vendor performance claims and actual performance, and very good at using intelligent cache to get that claimed 900,000 IOPS out of an array with SAS drives. (We’ve found it’s possible to get just as much performance with the 15K rpm SAS as with Fibre Channel drives.)

Cache is important in the HDS scheme, so you’d want to get the full cache size on offer. Finally, I am a bit surprised that HDS has not qualified any SAS form factor SSD’s in this array yet, as it would be a natural thing to do. SSD would be a good fit for high IOPS to a read-mostly metadata slice.

Although we have not used the AMS with SAM-FS, we have used it with multiple cluster file systems (Ibrix, PolyServe, Lustre) to good effect.

Carter George, VP Products, Ocarina Networks

http://www-304.ibm.com/jct01003c/software/data/smart-archive/

This is a next generation archiving solution that delivers on the promise of “ILM” with software integration to almost any structured or unstructured data/application in the enterprise. The infrastructure consists of IBM 2U servers which FC connect to either the IBM TruMAID (Massive Array of Idle Disk) storage and/or to your tape libraries. The IA pools could look to SAM-FS as another tier, but with much lower TCO than any of these other scale out NAS or internal cloud solutions.

TruMAID provides 179TBytes per square foot using 2TB SATA drives, and a full cabinet (10 sq. ft.) with 1.79PBytes requires only 5.5kW maximum power.

The IBM Information Archive is a complete software and hardware solution that would integrate well into your existing SAM-FS environment, giving you full protected archive features for compliance and eDiscovery requirements, and provide a very compelling TCO vs. standard NAS or cloud storage.

I am the IBM Data Archive Institute storage consultant for the western U.S. Send me your contact information if interested and let’s schedule a meeting to explore further.