Do you think it is time to reinvestigate the SSD place in the storage hierarchy when Linus Torvald is impressed by new SSD drives coming to consumer market ?

http://torvalds-family.blogspot.com/2008/10/so-i-got-one-of-new-intel-ssds.html

JPh. Papillon

Why? A couple of reasons. First, the array vendor has complete control of the environment that the SSD lives in and can - if they are smart - ensure that their software uses the SSD to good advantage. By replacing a few dozen short-stroked FC drives the array vendor provides clear power-footprint-performance advantages to the customer without asking them to change anything in their infrastructure.

Second, the array vendor shoulders much of the risk of the new technology for risk averse - aren’t they all? - enterprise customers. Persuading them isn’t a slam dunk, but the inside-the-array strategy supplies the one-throat-to-choke beloved of CIOs.

For the longer term the key question is: if flash had shipped in 1957 instead of the first disk drive, would we be trying to package flash into things that look like SSDs today? I don’t think so.

I welcome everyone - system, array, disk and startup vendors - to the scrum. There will be winners and losers among vendors, but in the end buyers will win. And that will keep them buying even more.

Robin

I think that maybe what I’m after is the impossible and amounts to a complete storage revolution (I suppose by getting rid of revolutions in the form of disk drives). Personally I feel that, ultimately, disk storage is doomed as a truyly high-performance storage medium as it is doomed by the fundamentals of geometry. Essentially as capacity increases through data density increases, the throughput and IOP per GB inevitably worsens (given there are physical limits to the speed at which disks can be spun). Lots of clever things have been done to try and overcome this - file system organisations, caches at various levels and so on, they just optimise things and can’t overcome that fundamental issue.

Now flash may not be the answer, but it at least adresses one of the issues (namely random read access). I think that disks, like tapes before them, will gradually get relegated to the role of relatively low performance, semi-archival type functions.

Anyway, on the subject of flash and caches, consider architectures like Compellent’s or Equalogic’s. These systems will, when given a mix of storage with heterogeneous performance properties, do various levels of automated block-based ILM internally. I.e., discarding the babble speak, they put your most frequently accessed blocks on the best class of storage in the system, when there is a mix of storage types. You, the user, are required to do nothing.

I find that to be pretty interesting, particularly if you’re looking forward to a day where you have a tray of storage in your enterprise.

I’m thinking in particular for the virtual machine provisioning use case, where if you are “really load’em down,” you are starving for IOPS. Well, I may be *able* to put my 40G vmdk files on flash, but man I really donwanna. That would be a waste. If, however, the frequently busy parts were there: win.

Your “cache” discussion made me think of this, because it’s sort of cache-like in its behavior, but… it’s not.

Joe.

Another approach, of using Hierarchical Storage is poss9ible, if you have the applications which can make use of it. In this case, the flash isn’t a “layer” - it’s a separate storage pool and is used to service (say) requirements for low latency OLTP systems. In this case, then it doesn’t make any sense to have a flash layer - it makes sense to have a higher performance storage pool and storage systems and databases which can automatically manage data placement accosrind to service requirements.

In the case of difference between disk and flash, applications talk to file systems, not devices (in general). I’m sure that current file systems are optimised for the performance limitations and characteristics of disk (minimising seeks, avoiding fragmentation etc.) . A file system optimised for flash would be a good idea and I’ve no doubt that many of the known weaknesses of flash on random writes could be addressed (e.g. WAFL-type files systems and roll-up optimisation of writes with the use of a small amount of non-volatile DRAM).

As for bringing back 3600 RPM drives, heaven help us. There is already a huge problem that the serial I/O speed is failing to keep up with increased capacity for unavoidable geometric issues. Disk sequential read rates go up in line with linear bit density whilst capacity goes up to the square. The result is it now takes 3 or 4 hours to read the whole of a 1TB drive. Rebuild a RAID set (especially a large RAID-5 set) and see how long that takes. Go to 3600 from 7200 and you’ll double that again. The number of random IOPs possible will reduce markedly (perhaps by 30-40% depending on access patterns). This is against a backdrop of increasing capacity so the number of IOPs per TB stored will continue to go down and down (as it has been doing since disks were invented) . I’d also be interested to know if there really would be any substantive difference in reliability, power consumption (or costs) between 3600 & 7200 RPM disks.

Possibly, just possibly, 3600 RPM drives would have a place for really low access rate, low throughput archive data, but It’s difficult enough to make use of 1TB 7,200 RPM SATA drives in the enterprise without hitting performance issues . That will get more and more difficult. The 2TB disk is here, and such a monster at 3600 RPM could take 10 hours to read.

Nb. I don’t think this is an issue for Enterprise storage only - performance on my quad-core 2GB three-disk desktop is often crippled by disk contention.

Those of us who’ve been around a long time might remember an earlier “extra tier” of memory: The CDC 6600 and its successors back in the late ’60’s/early ’70’s supported “extended core storage”. This, like the main memory, was magnetic core. It was organized in small blocks (the CDC’s had 60-bit words, and if I remember right, ECS blocks were 8 words) and accessed by doing moves back and forth to main memory. So, of course, ECS was slower, not as good at random access, but cheaper. The software didn’t see it as an I/O device, but had direct access; the hardware let you make ranges of ECS accessible to a particular program. (Actually, these weren’t virtual memory machines even in main memory - it was one contiguous range of real memory per program, I think.) The cycle of idea reincarnation takes another turn….

One can imagine a path to a new storage layer as follows: First, use the memory-mapping interface to directly map files on the flash to memory. Now, replace the back end of the memory mapping for flash with something other than existing disk drivers. This gives general programmers the simplest interface - everything is just memory, if that’s how they want to use it - while hiding the changes needed to treat flash as flash, not as a disk. Of course, then you can make some of that memory transactional - it’s backed by flash, after all - and things start to get really interesting.

— Jerry

For read cacheing, taking the risk of an hypothesis: you might set your swap to be very very large and put it on a fusion IO device or other some such.

Joe.

I am wondering about this myself. In my dreams I am hoping for “blade flash” that has modular, removable Storage units, multiple choice physical interfaces and, most importantly, a user configurable API. This would be a Unit of Technology that provides the most benefits for the $$$. The reality is still short of that.

Some interesting references are:
Adam Leventhal’s Weblog
“Adam and Brendan refer to each other in their Weblog articles.”

Brendan Gregg’s Weblog
“A previous ZFS feature (the ZIL) allowed you to add SSD disks as log devices to improve write performance. This means ZFS provides two dimensions for adding flash memory to the file system stack: the L2ARC for random reads, and the ZIL for writes.”

Adam Leventhal’s ACM “Flash Storage Memory” article
“Can flash memory become the foundation for a new tier in the storage hierarchy?”

http://www.fusionio.com

This way favors PCIe 8x.

Vendors are all over the place, but many of them are starting to do this sort of thing, we’re individual cards in their systems act, more or less, like a second level of memory or, if you will, high speed swap.

Joe.

take a look at zfs Level 2 ARC (L2ARC) cache. It has already happened.

And Robin, thanks for insightful post.

As to the so-called “silent corruption,” one could argue that this is nothing but FUD> If it can be detected and corrected by ZFS or the operating system of a server, then too it can be detected and corrected by the storage device itself - as does the ZeusIOPS drive already today. In addition, it takes but a few addition guard bits to verify that what was written to a disk is in fact returned - not necessary to implement complex journals and commit logs. Today most enterprise-class storage arrays (including both Symmetrix and CLARiiON) already incorporate such Data Integrity Bit protection today (you can’t trust disks to return good data either, as Robin reported CERN discovered this last year). And with T10-DIF, we may soon have an end-to-end protection against undetected data corruption. Add simple RAID across multiple flash drives, and it is easy to correct from drive-detected errors through a simple block rebuild and without host operating system, database or file system involvement. I note that today FusionIO has added additional flash to their flash PCI card specifically to provide RAID for the main capacity.

So it’s not the issues of volatility or silent corruption that’s driving this “better in the server” mentality - or at least, it shouldn’t be.

I don’t think it’s correct to call the “Server Side” of Sun’s business as not in the Enterprise business. I know it may seem that way from the outside, but frankly that’s just not how Sun works internally at the moment.

Anyway, yes. This is not an either-or. You will certainly start seeing improved flash in the nonvolatile parts of highly available storage (i.e., the journals, write commits, intent logs, whatever you want to call them) that make failover “safe” in large numbers very, very soon.

As for permanent storage, there are kinks, like higher incident rates of silent corruption and the like than with spinning media. But talk to the Sun guys about this and they will say “ZFS can fix that, and is an ideal file system for flash because of that.” They’re probably right.

Joe

This was the panel moderated by Ryan Floyd (Storm Ventures) and made up of Robin, Mike Cornwell (Sun - server side), Jim Porter (Disk/Trend), Steffan Hellmold (Seagate) and Joel Hagberg (Fujitsu).

Given that none of the participants (save Seagate) were in any way connected to “enterprise storage,” I guess I’m not surprised at your results - from everyone else’s perspective, flash-as-the-disruptor is probably be more interesting than the boring old flash-as-the-next-generation-of-storage vision that EMC (and soon IBM and Hitachi) are delivering.

From what I’ve been told by attendees, the “unanimous” allies were Sun (flash-belongs-in-the-server), the VC/moderator, the Disk/Trend guy and yourself.

Seagate, on the other hand, positioned a more evolutionary approach, leveraging that which is available today (disk drive form factor), while Fujitsu’s VP of Business Development claimed that Fujitsu would “be there” once Flash was ready for prime time (which it must not yet be, since Fujitsu doesn’t make any flash drives today).

Hardly a balanced view of “enterprise storage” in the opinions of several people in the audience that I’ve heard from…

But what I don’t understand is why this has to be an either-or discussion in the first place?

The more probable outcome is that we’ll have persistent solid-state storage appearing in lots of places up and down the I/O stack - just as we do DRAM today. The original model was all RAM in the server and none in the peripherals; today even cheap disk drives have more RAM cache than did most computers back in the 1980’s.

I mean, why can’t we have our flash as both cache and permanent storage?