1) following our paper, we published another DSN 2009 paper on how to practically increase encoding performance of popular RAID-6 codes:
http://nisl.wayne.edu/Papers/Tech/dsn-2009.pdf

2) a while ago ( 2005 ), we published a RAID-7 code, which we call STAR code at FAST 2005:
http://nisl.wayne.edu/Papers/Tech/star.pdf

so more choices are available. And coming soon, we will publish another paper on how to use the STAR code to tolerate both a single (whole) disk failure and silent disk errors at the same time.

Enjoy.

Disk Magnetic R&D and going down from what I can see. Hitachi and others never fully recovered from several disk technologies (1 and 2 in) and capacity increase failures rate. Millions $ lost here. I start to see slow capacity increase on magnetic disk over the years. We used to see 2x increase per year. Now it start to be 18 months or so.

SSD should definitively kill magnetic disks. Market will drive this and not technology. With millions of small devices the R&D will be in these products and not on the hundred of thousand disks enterprise grade ship per year. More competitions will also drive this market. There is more SSD vendors than magnetic disks. Simple maths should work on SSD.

Tape capacity should double or more next year (from 800GB to few TB). HP, IBM and Sun should come out with their new devices by mid next year. As storage capacity increase, media cost will be the key here. Not the device themself. Do not forget media or disk migration cost.

I agree with Robin, with a twist. “It will be many years before flash competes on a $/TB basis”. Here’s the twist. I predict at least 5 years, and that’s an eternity in tech development time. During the next five years, the magnetic media vendors will be working density improvements like crazy. There are improvements in the lab that would permit increases as much as 1000X for magnetic media. While these lab improvements are too much like science to save the high speed (10K, 15K) segment in the near term, within 5 years they could change everything. Hence, do not count, at all or ever, on FlashSSD’s taking the bulk storage crown.

Joe.

To overcome this disadvantage, the semi-conductor route has to come up with storage densities much higher than can be achieved with magenetic media. Fujitsu have announced magnetic recording technology with a 25nm pitch. Samsung have announced NAND flash using 30nm fabrication. To get the same cost per GB with SSDs, then the semiconductor side would have to move on far faster than disks, even when things like MLC are used.

This feature density issue also points out the fundamental problem with optical media. The feature density cannot be be greater than that dictated by the wavelength of light used. There are some major technical problems about suitable cheap optics working down into the type of far ultraviolet region to get near what is possible with magnetic media, not to mention issues regarding the optical media itself. Only be exploiting 3D recording would it appear that optical media can compete.

Of course it might be that SSDs get “big enough” to solve the nearline/archive problem, but I think that is only true if volumes of data following the exponential path of the storage medium itself. With consumer level devices capable of HD video (and no doubt that will be supplanted by yet higher formats), then the trend will continue for some time yet.

I do fully expect SSDs to effectively kill HDDs for transactional I/O. The fundamental problem with HDDs is that the IOPs per GB figures is dropping like a stone as capacities increase and that latencies have barely improved at all over the past 5 years. It is already the case that on a cost per IOP basis, enterprise SSDs are way ahead of rotating magnetic disk (and possibly even on GB/s) as is power used per IOP. Those distances will get even bigger in favour of SSD.

But for nearline/archive storage – I think tjhat SSDs are way off the mark.

Of course something might come along (like dataslide) or a completely storage system which changes all the rules. But for now we are seeing a shuffling of the boundaries with semiconductor storage gradually displacing HDDs at the high I/O density level and HDDs supplanting tape at the archive/nearline. For other than removable storage, optical is going nowhere and it is conceivable that flash will take that market.

In the medium term, if Heat Assisted Magnetic Recording (HAMR) gets productized, that will reduce bit rot.

Gary, I agree that disks have many advantages over tape. Today’s disks have to do all that stuff because they weren’t designed for archiving – but they could be. There is a huge home archive market and IT would eventually warm to the idea – especially if erasure codes made the loss of multiple drives a non-issue.

Charlie, most flash drive go into applications where disks don’t compete, such as the below $50 consumer space or the high-performance IT space. It will be many years before flash competes on a $/TB basis – even with the power advantage.

Robin

I realize people think of them as being at the other end of the storage spectrum, but they have several attributes that make them fit here. What do you think?

I can only speak for Nexsan (though I think other vendors would agree) in saying that deep disk archives should be periodically powered up and surface scanned every 2 to 6 weeks. During this process, the spinning disk activates the drive filtration systems to help remove any outgassed material that’s condensed on the media, we re-map and repair and report any grown defects, and of course dead drives will cause a rebuild on a spare. With RAID-6 you have a very high confidence that defects can be repaired with no loss of data. Our AutoMAID technology does all of the above automatically (though I don’t want to exaggerate the brilliance of this as RAID controllers have been routinely surface scanning disk packs for at least 10 years that I know about — the only new twist is that we might have to wake up the array to do it). We wake each array up one at a time, do the scan, and let it go back to sleep. The added power consumption is very minimal in a deep archive scenario as typically an array only needs to spin for an hour or two per month.

Anyway, this is the advantage that cold disk has over cold tape — it is very labor intensive (and risky to the media) to do a similar kind of scan of offline tapes, especially if not in a library slot, and in most cases there is no concept of RAID-6 on tape which would allow the system to periodically repair minor defects that appear over time. The best you can do is wholesale replacement of tapes or failing that, receive an inventory report that tells you how many tapes you have lost this month.