On October 10th at Oracle Openworld, Larry Ellison announced a new world
record (audited) for TPC-C using enterprise grade Flash.

Flash is the new disk and disk is the new tape. That’s how 2009 ends.

For top stories 2008, you said:

>>High-end SSDs will displace 15k high-end disks in the next 3 years.

I don’t think 15K disks are a long term good bet, but I don’t think it’s because Flash SSDs will replace them. There is a large and growing body of evidence that write performance in synchronous workloads — where application IO streams are transactional “threads” of interdependent reads and writes — are so problematic as to reduce performance of a Flash-SSD often to as little as the equivalent of just a few 7,200 RPM SATA disks.

Gene Ruth recently tripped over this problem when he observed write performance of a RAID5 array made up Intel’s much-ballyhooed X25-E “enterprise” Flash SSD.

The asymmetry of read vs. write performance is nothing new. But what has gone unnoticed by too many for too long is not the asymmetry of performance when reads and writes are measured in isolation, it’s the dramatic FURTHER decline in performance that happens when Flash SSDs encounter mixed read-write workloads — especially when these are encountered in the “synchronous IO” context of ordered and interdependent reads and writes typical of transactional workloads. This effect is so dramatic that in RAID-5 (which is a good model of this type of IO), write performance of a four-SSD array has repeatedly been observed to drop BELOW that of a similar sized array of 7,200RPM SATA disks.

At an average now of 300 15K spinning disks for each CPU socket in the server now, TPC is probably the most IO intensive, actuator-bound workload on the planet.

So allow me to offer a couple of my own predictions.

1) 2009 will come and go, as 2008 did, without a single published, audited TPC result using Flash SSD in place of disk, and the silence will become deafening.

2) The industry will learn a lot more about the intrinsic and fundamental limitations of Flash at the silicon architecture level, and we’ll begin looking for other alternatives.

Flash SSD will find lots of new uses, but not among brick-and-mortar enterprise applications. The enterprise data-center markets for Flash are going to be much smaller and applications far more limited than anyone expects.

In general, applications tend to use files, and that’s the abstraction layer that matters for most purposes. It will, of course, be possible to design storage arrays where flash is more closely coupled with the internal bus structures of the devices. It would be somewhat surprising if the likes of EMC, EDS and NetApp were not looking at exactly that. That way they could hide away the flash hardware interface from the more standardised storage services offered to applications. It’s also possible to write a “flash friendly” file system (OnTap and ZFS are a very good fit for storage devices with very good random read performance as both have a tendency to fragmentation with some access patterns and they are also more tolerant of poor random write performance).

Applications are notoriously slow to change, and there are plenty of legacy applications out there that could do with a simple speed boost, especially in the area of reduced latency. Existing network storage protocols and paths through device drivers and so on probably means that it will be hard to get lower than about 0.5ms, but that’s still an order of magnitude or more better than we see on random access on 15K drives (and the IOPs possible increase still more). There is, of course, nothing stopping a commodity approach to this, once the appropriate hardware, software and standards emerge.

Outside the realm of dedicated storage array hardware and embedded server storage, then I don’t see and obvious winner just yet. We still need this shared access model, and unless a very high speed, low-latency shared flash storage hardware model appears (with drivers), then I still perceive that the vast majority of large applications will access their storage over existing networked storage protocols.

If we want to see the persistence of legacy access methods, then look at how the vast majority of removable flash media is formatted. It doesn’t matter if it’s USB, CompactFlash, SD, memoryStick, you will find that the various generations of FAT dominate. It’s perfectly possible to embed abstraction layers into storage devices to optimise performance for an alien storage type. Quite what the physical interace to the actual storage device is then an internal only matter.

I’m also not as optimistic as you are on zero maintenance storage. I remember the IBM demo product (stackable cube of storage) where you were left with dead storage bricks in the middle because you didn’t want to unstack 20 – 30 TB of storage to get to the place where you could remove it. I do agree the days of carefully removing one drive at a time out of a shelf of drives may be on the decline. Give me a 10 – 15 drive “sled” I can pull out and “remanufacture” after its more than 33% dead that would be fine.

And finally you haven’t yet touched on what seems like a growing trend which is SiTH (my silly marketing acronyn for Storage in The Home). Cable DVR penetration plus game consoles plus media servers and having your media in central place is starting to make sense to non-nerds. The who netflix/blockbuster/amazon/itunes media on demand markets, the availabilty of an “always on” appliance with a relatively large amount of storage in it and a fat download channel available (cable/sat). Given Intel’s recent numbers and the proliferation of notesbooks (vs desktops) nearly everyone I know now has at least 2 USB attachable external hard drives, some folks many more, and the pain is growing around managing that storage.

Interesting times, especially with money being so tight.
–Chuck