Another StorageMojo Best paper, The Bleak Future of NAND Flash Memory, presented at this year’s FAST ’12 conference, quantifies flash’s declining reliability, endurance, and performance as density increases.
Researchers Laura M. Grupp and Steven Swanson from the UCSD Non-volatile Systems Lab and John D. Davis of Microsoft Research collected data from 45 flash chips from 6 manufacturers. Using that empirical data they predict the performance and cost characteristics of future SSDs.
Faster better cheaper or slower worse cheaper?
While NAND flash is produced with semiconductor processes, smaller feature sizes don’t lead to faster performance or greater reliability. As NAND features shrink, so do the number of trapped electrons that store information.
Figures of merit
The research found that performance, program/erase endurance, energy efficiency, and data retention time all got worse with feature shrink.
Based on past performance, the team derived equations to describe how changes in feature size have affected key specs. They looked at SLC, MLC and TLC and feature sizes scaled from 72 nm to 6.5 nm (the consensus smallest feature size published in the International Technology Roadmap for Semiconductors (ITRS0), and assumed a fixed silicon budget for flash storage.
Key results
- Latency. MLC write latency will double over time. Triple-level cell writes will grow to over 2.5MS, noticably reducing its performance advantage over disk writes.
- Bandwidth. Small – 512B – read bandwidth and all writes decline by up to 50% over time. The impact is greatest on high-performance SLC flash.
- IOPS. MLC flash I/O rates will drop almost in half.
Flash may be the new disk in a few years.
The StorageMojo take
One important qualifier is that for the purposes of their modeling the team constrained the number of chips in the hypothetical future devices whose performance they predicted. While fine for isolating the impact of future chip shrinks, it ignores the potential of much greater parallelism for managing these changes.
Bandwidth drops by half? Double the number of chips.
But if something can’t go on forever, it won’t. NAND flash will soon enter an end-of-life crisis for computer applications that need performance. That’s why ReRAM (resistance RAM) looks to be a good bet for replacing computer flash – not mobile device flash – over the next decade.
Courteous comments welcome, of course. A version of this post was published on ZDNet last week.
Robin, I read this paper with interest. I agree that SSD doesn’t look good for backup or large cost sensitive applications that requires increasing storage density. I’ve had complaints from people who leave an SSD on the shelf for a year and come back to find nothing on it. The other common complaint is the “sudden failure” syndrome. While hard drives often give an indication or warning that they are failing, some claim their SSDs fail suddenly with total data loss.
Despite all this, it seems a day doesn’t go by without another large vendor announcing a hybrid drive, cached NAS or SAN, or blazing fast PCI-e storage board. I think people buying SSD and paying the 8-12 times price over similar sized drives are rationalizing it on performance above all.
I’m very curious about the background of the computer scientists that did this story (students? PHDs?) and whether they got the benefit of talking to Intel’s best and brightest for ideas and/or access to the lastest SSDs. My reading tells me Intel is regarded above some other vendors in reliability and I know they’ve made some advances in chip architecture. I’m not sure if these advances apply to SSD or whether they will overcome the reliability problems pointed out as density increases.
There are a number of serious problems with the way that paper draws its conclusions. They discuss everything in the context of individual SSDs, the realities of flash operation in the context of large arrays can be quite different. For example certain modern flash storage arrays can hide all the write latency and its effects on read latency. Flash in an SSD operates at whatever temperature the server it is installed in is running at, flash in an array with a flash controller that can monitor temperature can talk to its array controller and ask it to change the fan speed when it gets too warm. I could go on, but the point is that at best the papers conclusions can support the view that the SSD is not the optimal vehicle for NAND flash, they don’t support the conclusion that NAND flash has a dim future.
As far as bandwidth and IOPs, they use a 4K/8K write size for MLC/TLC, but MLC already exists with 8K pages, as well as having the ability to write more than one plane at once, which doubles the write bandwidth. Double the page size again and you double the BW.
Now bigger page sizes only help on the reads if you can use more than a single user read worth of data in the page, which might be possible depending on what the system knows about access patterns. But without making any assumptions about the ability to store data together that’s likely to be read together, garbage collection, which, if you are in a high write rate environment, can wind up reading more bytes than the user does, can use most of the data in a page.
So there are factors of 2X, 4X maybe 8X in performance that the paper misses out on.
As far as density, it is not necessary to go to smaller features to get more bits per chip by using 3D techniques such as Toshiba’s P-BiCS (Pipe-shaped Bit Cost Scalable) MLC NAND which allow vertical stacking which increases density without using smaller features with their worse performance and lifetime.
The paper makes the mistake of treating flash chips as a physical phenomena that can be extrapolated out to the future based on a handful of parameters without recognizing that not only can those basic parameters change, but parameters they don’t know about can change, and parameters that aren’t currently relevant can be added.
If you read the paper on the recent Toshiba / SanDisk TLC announcement ( http://www.theregister.co.uk/2012/02/24/tosh_sandisk_tlc/ ) you will see all the new advances that were used to increase the write BW even as they shrank the size.
Just to give one example, in the white paper on the TLC I mentioned above they discuss how adding a digital thermometer to the chip to allow for correction for temperature effects led to a 75% reduction in the BER, there is a half an order of magnitude improvement just from asking the chip to tell its temperature.
The group at UCSD that authored this has done some nice work so I don’t mean to be too negative, but they are trying to predict too far from a limited and artificially fixed set of assumptions which unfortunately negates much of the validity of this paper.
Go check the LSI Warp Drive technology out it acts like a mini storage aray on a PCIe card gives great performance and has a 3 year warranty, performance is amazing and LSI has been putting data on and off disk for a long time so their number one concern is reliability. The rest I think the article is probably right on! SSD’s seem to be way over hyped nad do seem to slow down as they fill up even if they do house keeping in the background. I would say the best is server based flash, lots of memory and a well designed SAN !
Hi Robin,
Thank you for sharing this report, which seems to have gone “viral” in the solid state world.
To be fair, many smart people discounted the ability for conventional hard drive technology to last as long as it has. Just when we thought areal density could not get any better, a new innovation appeared to take the technology forward another 3-5 years. While the science behind the UCSD paper is accurate, it observes NAND technology where it stands today. Many companies with extremely deep pockets are already at work on the issue on NAND durability loss at smaller geometries, such as Micron’s ClearNAND and similar efforts from Toshiba and others.
Also, just because NAND geometries shrink does not necessarily mean that enterprise storage vendors need to jump to the latest geometry reduction. Witness the HDD industry itself: SAS and FC 15K rpm drives have long lagged SATA drives in density, and this is accepted.
Finally, keep in mind that UCSD is among the largest sponsors of efforts around phase change memory (PCM), a potential successor to NAND. There is at least some ax to grind here, though their work in the PCM field is extremely interesting.
Best regards,
Thomas Isakovich
CEO
Nimbus Data Systems, Inc.
I think this is a great discussion, and I’m glad to see that people are reading our paper and that it’s fostering discussion. If anyone would like to discuss it further please drop me a line at swanson@cs.ucsd.edu.
I need to respond to Thomas’ suggestion that, since we have done work on PCM, we have an “ax to grind” with respect to NAND.
It’s true that we have done work on PCM, but it has not colored our perspective on NAND. Nor does my lab stand to gain anything if flash’s bleak future materializes. As researchers, our responsibility is to make unbiased measurements, collect the best data we can and “call it like we see it.” That’s what we do in all our papers.
-Dr. Steven Swanson
Director, Non-Volatile Systems Laboratory
Assistant Professor
Computer Science and Engineering
University of California, San Diego
Some of us have been pointing out these trends for years now. Sun’s Michael Cornwell told us this back in ’09.
http://storagemojo.com/2009/09/10/flash-bash-disks-and-dimms/#comment-205302
So the Flash “lithography death march” is not new, but the future of flash is even bleaker than that painted by the researchers.
Quietly, over the last few years, applications that formerly needed to do lots of random IO have been moving their data structures into DRAM, which exorcises the disk-thrashing demons that created the need for SSD in the first place.
This “in-memory-database” (IMDB) trend was pointed out by Jim Gray in 2006, when he said “Tape is dead, disk is tape, flash is disk, and RAM locality is King”.
So, it’s location, location, location…but until recently, the cost of Real Estate in DRAMville has been too high.
Following Intel’s introduction of the Nehalem EX (now ‘E7’) processors, x86 DRAM sub-systems can now be designed large enough to hold TBytes of DB, and (most notably in the case of IBM’s “EX5” implementations) are more reliable in BER’s than either Flash or Disk. Examples of this trend in operation include SAP HANA, IBM PureScale, Oracle TimesTen IMDB and Microsoft SQL Server.
Even more mundane applications like MS Exchange have already been re-architected to move I/O intensive datastructures into DRAM, all but eliminating random IO. In two short product generations, Exchange has gone from among the most spindle-hungry, IOPS gobbling applications in IT to an IO profile that is almost 100% large-block sequential. As such Exchange 2010 works perfectly well with 7,200 RPM disks, thank you, has little use for 15K disks, and has no use whatsoever for SSDs. The cost of the incremental memory required to tame the beast was nominal, a tiny fraction of the costs of fixing the problem with Flash.
With scalable and reliable main-memory becoming widely available, applications architects have some elbow room, and are now free to leverage Gene Amdahl’s famous advice — “the best IO is the one you don’t have to do”.
Of course spinning disk, like tape, is extremely cost effective in terms of MB/Sec/$ as long as you keep it streaming, e.g. stop doing random IO. IBMD technologies are here now, and they virtualy eliminate random IO.
http://blogs.technet.com/b/dataplatforminsider/archive/2012/04/09/the-coming-in-memory-database-tipping-point.aspx
Without random IO, what is the value proposition for Flash-based SSD?
You might find the following graph of interest: https://docs.google.com/open?id=0B5QDPUNHLpKMcF9CbTc2LVJ6WVE
This is a plot of the history of storage capacity per inflation adjusted dollar for magnetic and flash storage from 1980 to April 2012. There is a clear logistic growth curve for magnetic media showing its terminal decline in density per dollar, whereas flash media seems to show a logarithmically constant growth. Figures are for consumer devices in the US consumer market. Flash media is from Intel only.
Some day flash storage will go linear growth, then exponentially declining growth just as any form of logistic growth. Until we pass the point of inflection no one can predict for sure when growth will end.
Niall
@ Niall,
The graph you cite above is incorrectly constructed. First, the Flash trend has been forced to a linear plot, while the magnetic trend is polylogarithmic. The resulting intersection in $/GB showing at about 2018 is illusory.
If you look closely at the data points for Flash between 2007 and 2012, you can see that the slope of that line is much shallower than the linear trend. Finally, to the best of my knowledge, there was no such thing as a NAND-Flash based SSD prior to 2003, but the data points go back all the way to 1985.
???
I don’t think any meaningful conclusions can be drawn from that chart. Do you know the source of the data?