Digital coinage can’t do everything a physical coin can do, but that’s not stopping people from signing up – or going to conferences. There’s one in Silicon Valley next week and the elite StorageMojo analyst crew will be there in force.

Digital currency as a store of value?
Today Bitcoin and other digital currencies look more like stocks than bonds because of their volatility. But with Amazon and eBay looking at accepting them, they could become more like money. If that seems unlikely, recall that much of what you use now as “money” is simply electronic transfers: credit cards; debit cards; PayPal.

These non-national currencies have much in common with how the US currency system used to work. Each local bank issued its own currency supported – in theory – by the deposits of customers.

To redeem the currency you’d go to the bank and exchange the notes for coin. Since 19th century travel was often difficult, the bank notes would depreciate with distance from the issuing bank.

With the frequent business crashes of those years, people were alert to the possibility that the issuing bank could fail, leaving the notes worthless. Thus if business tanked people would “run” to the bank to exchange their notes for specie. Since banks borrow short term and loan long term, they would often run out of coinage and close.

That’s why the US has a national currency, a Federal Reserve Bank and insures bank deposits (FDIC). But digital currencies have none of these protections.

The StorageMojo take
A recurring strand of American thought is that we should go back on the gold standard rather than letting the dollar to “float” against other currencies. After all, advocates contend, without gold the dollar isn’t backed by anything at all.

And yet the dollar remains the worldwide currency of choice, not only for B2B but as a store of value and convertibility as hard cash. Most US currency circulates outside the US – us locals would rather use credit cards.

Since the US dollar isn’t backed by gold, and since the Fed can loan as much money as it wants to banks that can use it as reserves against loans – if only they were making loans! – why exactly do we ascribe value to the dollar? Global acceptance and ready convertibility are two major reasons.

Which is where the value proposition for digital currencies makes the most sense. So can a digital currency replace – or at least supplement – national currencies? Yes.

It isn’t that different from what we used not so long ago – or what we use today. Digital currency is the new frontier in more ways than one.

Courteous comments welcome, of course. Any StorageMojo readers going? Look me up!

The brains behind the ZFS filesystem – including Jeff Bonwick and Bill Moore – have been hard at work for several years at start up DSSD. What are they doing with Andy Bechtolsheim’s money?

Bill’s recent Usenix bio says that “. . . DSSD, Inc., [is] a stealth startup focused on creating the fastest and most reliable storage possible.”

That puts them in good company.

They also have a trademark on the term “Cubic RAID” which describes “Computer software for use in electronic storage of data.” That narrows it right down.

Patent search
The good news is that while they are still in stealth mode their 6 patents are out there for all to see. What looks interesting?

The “Method And System For Multidimensional RAID” uses something called “RAID grids” to enable reliable and robust data storage and access.

The “Storage System With Self Describing Data” adds to the multidimensional RAID idea and adds object storage-like constructs into the storage system.

“Storage System With Incremental Multidimensional RAID” expands on this idea. By distributing the data and parity across the grid the design intends to enable data recovery despite multiple instances of data corruption or media failure.

The grid, of course, is three-dimensional, turning it into something they call a RAID cube. Given the focus on data recoverability perhaps the D in DSSD stands for durable.

The StorageMojo take
Looking at the technology it seems that the folks at DSSD are focused on a very particular set of requirements.

Self-describing data sounds like object storage. The extreme durability and high performance suggests a system optimized for transaction processing.

The use of solid-state storage implied in the name also suggests a focus on high performance. But SSD costs mean a use case very different from the video storage that objects are commonly used for now.

So what are they building? They are taking a radically different approach to the problem of high-performance transaction processing storage. The use of flash is a given in TP, and the extra durability, scalability and guaranteed read latency would be very attractive in large TP applications.

The most surprising piece is the object storage-like characteristics suggested by the patents. But handling billions of small objects at high-speed in a flat namespace would make it easy to distribute object indexes among hundreds of users, reducing file system I/O latency. The 3D RAID could eliminate the encoding overhead inherent in advanced erasure codes while providing similar robustness, enabling way-beyond-RAID6 availability.

Going after the last stronghold of big iron RAID arrays may seem bold. But as Willy Sutton said of banks, that’s where the money is.

The key will be whether their technology enables them to come to market with a clear economic advantage, be it cost or performance. With some of the most brilliant and original minds in storage and Silicon Valley working the problem, whatever they come up with is bound to be innovative.

Courteous comments welcome, of course. A launch this year feels about right.

StorageMojo is focussed on new storage technologies, products, companies and markets. And where do new technologies come from? From people researching at the limits of the known.

That’s why StorageMojo attends the Usenix File And Storage Technology (FAST) conference every year. Top academics – many with corporate ties – and grad students present the latest research.

Many papers are submitted and reviewed before some are chosen for presentation at the conference over two and a half days. Here are StorageMojo’s favorites from FAST 13, in no particular order:

SD Codes: Erasure Codes Designed for How Storage Systems Really Fail by James S. Plank, U of Tennessee, and Mario Blaum and James L. Hafner of IBM Research. RAID systems are vulnerable to a disk failures and unrecoverable read errors, but RAID 6 is overkill for UREs. The paper investigates lighter-weight erasure codes – disk plus sector, instead of 2 disks – to reclaim capacity for user data.

The StorageMojo take: high update costs make this most attractive for active archives, not primary storage. The capacity savings could extend the economic life of current RAID strategies vs newer erasure codes.

Gecko: Contention-Oblivious Disk Arrays for Cloud Storage by Ji-Yong Shin and Hakim Weatherspoon of Cornell, Mahesh Balakrishnan of Microsoft Research and Tudor Marian of Google. The limited I/O performance of disks makes contention a persistant problem on shared systems. The authors propose a novel log structured disk/SSD configuration and show that it virtually eliminates contention between writes, reads and garbage collection.

The StorageMojo take: SSDs help with contention, but they aren’t affordable for large-scale deployments. Gecko offers a way to leverage SSDs for log-structured block storage that significantly improves performance at a reasonable hardware cost.

Write Policies for Host-side Flash Caches by Leonardo Marmol, Raju Rangaswami and Ming Zhao of Florida International U., Swaminathan Sundararaman and Nisha Talagala of Fusion-io and Ricardo Koller of FIU and VMware. Write-through caching is safe but expensive. NAND’s non-volatile nature enables novel write-back cache strategies that preserve data integrity while improving performance. Thanks to large DRAM caches, read-only flash caches aren’t the performance booster they would have been even 5 years ago.

The StorageMojo take: Using flash only for reads mean ignoring half – or more – of the I/O problem. This needs to be fixed and this paper points the way.

Understanding the Robustness of SSDs under Power Fault by Mai Zheng and Feng Qin of Ohio State and Joseph Tucek and Mark Lillibridge of HP Labs. The authors tested 15 SSDs from 5 vendors by injecting power faults. 13 of the 15 lost data that should have been written and 2 of the 13 suffered massive corruption.

The StorageMojo take: We may be trusting SSDs more than they deserve. This research points out problems with still immature SSD technology.

A Study of Linux File System Evolution by Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau and Shan Lu of the University of Wisconsin. The authors analyzed 8 years of Linux file system patches – 5079 of them – and discovered, for instance, that

. . . semantic bugs, which require an understanding of file-system semantics to find or fix, are the dominant bug category (over 50% of all bugs). These types of bugs are vexing, as most of them are hard to detect via generic bug detection tools; more complex model checking or formal specification may be needed.

The StorageMojo take: Anyone building or maintaining a file system should read this paper to get a handle on how and why file systems fail. Tool builders will find some likely projects as well.

Courteous comments welcome, of course. There were some great posters and WIP presentations as well that I hope to write about soon.

The StorageMojo team has been hankering to see some bright lights and – just maybe – avoid some more mountain snow and cold. So its off to the fleshpots of Silicon Valley to see what’s been cooked up at the File And Storage Technology ’13.

If you see someone looking like the fellow to the right, don’t be shy, come over and say “howdy!” Always looking to meet new folks.

The StorageMojo take
FAST isn’t as flashy as more commercial events, but that’s OK. There is a lot of out-of-the-box thinking by very smart people. That’s a Very Good Thing.

Courteous comments welcome, of course.

I was asked at the SNIA nonvolatile memory conference why I did not include virtualization as a major driver for the use of nonvolatile memory. Flash helps with the multiple virtual machine I/O blender problem.

But we also had that problem when we were running multiple applications on a single machine. Yes, performance requirements were lower, but we managed.

I consider virtualization a feature and not a market. Why is virtualization a feature rather than a long-term product as many keen observers believe?

VM history
In the 1970s we had the virtual memory operating system wars. A number of companies, including DEC, Prime and IBM, developed virtual memory operating systems.

The key enabler of the virtual memory operating systems was the advent of 32-bit processors with true 32-bit address spaces. This is back when people programmed on minicomputers with a quarter of a megabyte to a maximum 4 MB of physical memory.

The VAX/VMS (Virtual Address eXtension/Virtual Memory System) OS, in contrast, offered a massive 4GB address space, 2 of which were reserved for the system and 2 for users.

Virtual memory operating systems enabled important changes for the industry. Software developers could focus on making their software as functional as possible without worrying about the underlying memory architecture.

The virtual memory wars continued into the PC era, when MS-DOS was limited to a 640KB address space and Quarterdeck systems came in with QEMM. But eventually Intel and Microsoft got their act together and solved the problem of virtual memory for the masses.

No one pays extra for a virtual memory operating system today. It is expected functionality that most don’t even know exists.

Like 32-bit processors the advent of microprocessors powerful enough to run multiple applications led to the virtual machine opportunity. If Microsoft had not been intent on selling as many server licenses as possible, they could have improved their multitasking so that the problem of server sprawl might never have occurred.

Be that as it may, there is nothing in today’s virtualization technology that could not and should not be incorporated into server operating systems. In 20 years new CompSci grads won’t know that virtual machines weren’t always built into the OS.

Feature vs product
Now just because something is at bottom a feature rather than a product doesn’t mean that you can’t make gobs of money before it becomes common. VMware is one example.

Data deduplication, for example, is clearly a feature. But the founders of Data Domain were able to make a lot of money by exploiting that feature with a high-quality application-focused implementation and being first to market.

Now deduplication is being built into new products. While debates over implementation details will continue among engineers, in a few years most users will see deduplication as a checkbox item.

The transient and the permanent
How do we distinguish between a feature and a product? It is the difference between the transient and the permanent.

Transient problems can be resolved. Permanent problems can only be managed.

Processor virtual memory management is a problem that has been solved for the majority of the world’s computers. Data de-duplication can be added over time to storage systems as computing power increases and the cost of bandwidth and random I/Os – thanks NVM! – drops.

But some problems can only be managed, not solved. The issues of scale, aggregation and metadata – among others – will always be with us.

Like gas-rich regions of galactic star formation, these manageable-but-not-solvable issues will continue to be areas rich in startup formation.

The StorageMojo take
Applying this theory to current markets yields some predictions:

• VMware, despite its feature-rich ecosystem and early lead, will lose to vendors, such as Microsoft and Red Hat, who can incorporate the most important virtualization features into their OS. VMware has no OS to fall back on and thus has no long-term future.
• Data Domain is a wasting asset. As others add dedup to their products, DD’s differentiation will decline along with its market value.
• Scale-out storage, like Isilon, will remain a lively market segment as the economics of disk, NVM, software, aggregation and metadata keep changing the calculus of efficient and durable storage.
• The improving economics of erasure coding will enable more efficient approaches to backup and archiving – as long as Moore’s Law continues to hold.
• System management is a permanent problem. When we get autonomic management at one level, the problem just kicks up a level with increasing scale.

Just as no one remembers the critical register and segment management skills required for 16-bit minicomputers, in a decade or so all the painfully acquired knowledge required to manage VMs will lose value as it gets built into the OS infrastructure. But there will always be new worlds to conquer.

Courteous comments welcome, of course. How would you define a feature vs a product?

No CES for me this year, but did attend Storage Visions 2013. Some cool stuff there.

Panasonic’s Blu-ray RAID archive robot
Panasonic has a Blu-ray-based storage system that looks interesting. Imagine 12 Blu-ray discs in the striped RAID configuration. The RAID gives the Blu-ray discs speed and competitive capacity.

Specs:

• 200MB/sec throughput
• 1.2TB per cartridge of 12 discs
• 108TB per archive of 90 cartridge
• 60 second access time
• 6 Watts standby, 100 Watts R/W
• 6U rackmountable box

The 12 Blu-ray discs are in a cartridge about the size of a tape. The disc unloader places them into 12 different Blu-ray drives for reading and writing.

But can Panny’s Blu-rays with a claimed 50 year life be trusted? I’d like something better.

The 1000 year DVD
That something better may be here. It is a DVD, and soon to be a Blu-ray disc, disc, with a claimed life of 1000 years.

The key is a writable layer that, unlike existing writable DVDs, uses an inorganic mineral layer to store the data. Doug Hansen, CTO, told me that they use a stone-like material.

Most rocks are silicates, oxides and metalloids. M-DISC’s material at nanoscale looks like igneous rock. Extremely thin layer and fine-grained – 100nm thick – horizontal scale 1/2 micron and smaller for bluray.

The laser burn heats the mineral layer so there’s a bit of melting and surface tension creates a hole. The material migrates to create a berm.

Because it is physically burnt into that inorganic mineral layer of the DVD media, it cannot and will not shift or change over time. The layer will last as long as the disc’s tough polycarbonate plastic. Most credible optical story I’ve heard.

Ridata is producing the discs, with Blu-ray versions expected this summer. All current LG DVD burners are warranteed to properly burn M-DISCs, which means you take your chances with other brands today.

Quotium
Quotium’s StorSentry software analyzes the quality of tape storage on a real-time runtime basis. With LTO tapes warranteed for only 200 complete read/write cycles, and costing almost as much as a disk drive, anything that can extend the useful life of a tape while protecting your data is a Good Thing.

Media Entertainment & Scientific Storage
MESS is a new group looking at long-term storage stack from media all the way to Digital Asset Management. Paul Evans, a long-time big systems guy, is heading it up.

The StorageMojo take
For some reason my attention was on the long-term storage issues that media and entertainment folks must grapple with. Given the massive carelessness of the movie industry over the last 100 years – and the loss of thousands of movies – I don’t believe that republishing all data every 5 years is a viable long-term strategy.

Long-term readability is a major problem with any digital medium. Assuming the media is good, can you count on a machine able to read it?

Optical has an important advantage because optical technology is everywhere and easy to replicate. Compare that to the specialized head/media engineering of tape – typically promising compatibility over 2-3 generations – and you’ll have a much more difficult time reading an LTO-5 tape in 20 years.

Panasonic should get together with M-DISC to produce a true long-term, high-performance storage system. But in the meantime, tape is still a major piece of the long-term storage puzzle, and Quotium’s software seems like a step in the right direction.

But MESS is correct to focus on the entire stack. Even if you can read the media, can you make sense of it?

One of these days the relentless growth in the density and speed of digital media will slow. When that happens it will become practically impossible to maintain existing data without a long-term strategy.

While we can hope that slowdown is decades away, the earlier we start thinking about it the better prepared we will be. Our digital civilization depends on it.

Courteous comments welcome, of course. Come to think of it, maybe the fact that I moderated a panel on long-term storage put me in that frame of mind. Go figure.

Interesting companies at this year’s Flash Memory Summit, but the winner of the StorageMojo BuzzGen award is Skyera. That there’s little detail on their system may have helped: attendees get to fantasize about the putative magic under the covers.

Founded by PhD SandForce alumni with no VC funding – but with investments by a large storage vendor and a large flash vendor – this team has an impressive track record and hundreds of patents. But the lack of detail is troubling: how much is real and how much is planned?

44TB in 1U
The claims are impressive:

• Up to 44TB in a 1U half-depth, 750 watt box
• Up to 1 million IOPS or 3.6GB/sec
• Built-in network switch: 40 x 1GbE & 3 x 10 GbE
• Software including in-line Compression/De-dup, AES Encryptio to protect data at rest, Snapshots, Writeable clones, Configurable QoS levels, Consistency groups, Thin provisioning, Dynamic re-sizing & more.
• Flash Controller: “Proprietary algorithms dynamically tune the partitions during the lifetime of the flash minimizing damage to the flash oxide layer typically experienced during write cycles”
• $3/GB raw; 99¢/GB with dedup & compression turned on

This all translates to the economic bottom-line:

. . . an all-Flash enterprise solid-state storage system that delivers next-generation performance and density through solid-state technology at a price point equivalent to spinning disk. For the first time, enterprise customers will be able to utilize solid-state technology as a direct replacement for traditional hard disk drives. . . .

[bolding added]

Where have we heard that before?

The StorageMojo take
At a private briefing off the show floor, Skyera’s CEO and CTO wouldn’t go into any detail about the how, focusing on the what. They wouldn’t even open up the pizza box labeled “44TB” for inspection, pleading foreign patent filing issues.

Really?

From the comments they did make, it appears the secret sauce includes a variable-block size flash translation layer and a wear-aware ECC function that increases ECC power as flash ages. And much more, no doubt.

Skyera’s rapid emergence is one more sign of the vibrancy of the storage market. The paucity of supporting information about their box underlines the difficulty customers have in evaluating competing claims.

But the larger trend is clear: enterprise flash array vendors are targeting the cost advantage of disk arrays. And one of these days they will catch up. Then the sea change from disk to flash arrays will start in earnest.

Courteous comments welcome, of course.

StorageMojo’s crack analyst team is hitting the road: Flash Memory Summit; Intel Developer Forum; and SNIA’s Storage Developer Conference.

Flash Memory Summit
I’ll be chairing a tutorial at the Flash Memory Summit on Thursday morning, August 23rd. Topics include:

• Architectural approaches to storage of relational and other structured databases
• Actual results in database applications such as computer graphics, IC design, and data warehousing

IDF
At the IDF the topic is Object Storage, though the title may be something like Revolutionary Storage. Panel members will include Amplidata, Aspera software, Quanta and an Intel cloud architect.

Storage Developer Conference
Not sure of my role at the Storage Developer’s Conference, but I’ll do my best to be interesting. If you develop storage this is the conference for you.

The StorageMojo take
The breadth of these events reflects the incredible ferment in storage today. I’ve been following the industry for 30 years and this is the most exciting time I’ve seen.

Given the continuing explosion of data – and we’re just scratching the surface – the demand for fast, reliable and economical storage capacity will also grow explosively. What’s also clear is that the old paradigms won’t scale into the future.

If you make it to any of these events I’d be pleased if you’d introduce yourself. Warning: I don’t usually interact with many people, so if I appear a bit glazed it isn’t you, it’s me.

Courteous comments welcome, of course. Want to make it to VMworld too, but haven’t figured out the logistics.

The post-RAID (noRAID) era has begun. While RAID arrays aren’t going away, the growth is elsewhere, and corporate investment follows growth.

Why now?
There are now architecturally superior alternatives to RAID that are lower cost. But you could argue that the post-RAID milestone was passed years ago.

• The authors of the 1988 original RAID paper (Patterson, Gibson and Katz) all moved on a decade or more ago: Patterson to OceanStore; Gibson to Panasas, a scale-out object storage company he co-founded; and Katz has been working on Hadoop among other projects.
• What are probably the fastest growing large storage infrastructures in the world – Google’s and Amazon’s – aren’t based on RAID.
• Major storage vendors including NetApp, HP, EMC and Hitachi have all invested in – and are selling – noRAID systems.
• But the biggest reason? The math behind erasure codes improved after the RAID paper was written.

The math?
RAID uses a form of Reed-Solomon erasure coding to create parity information that protects a RAID array from 1 (RAID5) or 2 (RAID6) uncorrectable read errors (URE). But RAID 5 stopped working 3 years ago if you use SATA drives.

Erasure coding’s key advantage is that you can break up your data into n fragments, add m additional fragments, store the fragments across n+m devices, and then recover the original data from any n of the devices. Thus in a RAID5 8 drive stripe, the original data is divided into 7 fragments, an 8th fragment is calculated – the parity data – and then any one of the 8 drives can fail without losing (theoretically) any data.

RAID5′s problems are that as disks get larger, rebuild times get dangerously long – increasing the chance that another disk will fail before rebuild completes while reducing performance all the while – and that an URE will be found on another disk, killing the rebuild. Surviving 2 failures is the minimal reasonable protection today.

In the ’90s a new form of erasure coding was developed that enabled developers to create codes with an arbitrary level of redundancy – survive 4 failures? 10? Pick a number! – called fountain or rateless erasure codes. Startups including Digital Fountain, Cleversafe and Amplidata have sprung up to take advantage of these new codes.

A new StorageMojo video explores the advantages of rateless codes, using the Amplidata example. One key advantage: the redundancy needed to survive 4 failures is, they tell me, down to 50-60% of the data. Much better than the 3x replication that Amazon and Google use in their infrastructures, and competitive with RAID6.

The StorageMojo take
Redundant Arrays of Inexpensive Disks shook up a complacent industry almost 25 years ago. But time and technology move on.

Despite the huge investment the industry has in RAID controller code, we now have better solutions. Properly priced and marketed, these solutions will drive the next big round of storage growth.

Courteous comments welcome, of course. I’ve been doing work for Amplidata. For a quick intro to erasure coding for storage developers, check out Prof. Jim Plank’s Erasure Codes for Storage Applications (pdf) presentation.

Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

StorageMojo offered its soapbox to any vendors willing to weigh in on the question of whether enterprise arrays should be built from flash SSDs or not. Ed Lee, architect at Tintri, formerly of Data Domain and a Berkeley Ph.D, elected to respond. It is a long piece but rich in insight.

Tintri produces hybrid disk/flash SSD appliances optimized for virtual environments, not Symm-killers. They use SSDs in their products, as do other folks like Nimble Storage.

No money changed hands between Tintri and StorageMojo or related entities. My accountant is weeping in the next room.

Begin Tintri’s response:

Outside the SSD Box: More than Faster Disk
Robin Harris of Storage Mojo in his recent article, “Are SSD-based arrays a bad idea? and Matt Kixmoeller of Pure in his response, The SSD is Key to Economic Flash Arrays, present interesting perspectives on whether or not SSDs are the best technology for building flash-based arrays. Robin argues that by rethinking how flash can be packaged outside the SSD box, you can achieve better performance, reliability, cost and flexibility. And these observations are supported by the experience of existing flash-based storage vendors who have developed their own custom flash modules and packaging. Matt argues that SSDs provide an industry-standard product that requires less investment to leverage, better economies of scale, and rapid improvement in technology. These are also very valid points, especially for startups with limited time and capital.

Latency
Taking latency as a point for comparison, flash-based storage vendors using custom packaging often quote IO latencies in the tens of microseconds versus SSD latencies of low hundreds of microseconds. While this is a notable difference, software and interfaces can also add overhead and the final latency seen at the subsystem level may differ by only a factor of two to four. Server-side flash products can avoid more of the software and interface overhead and provide better latencies – but may require rewriting applications to capitalize on this advantage. Keep in mind that hard disk latencies can easily reach tens of milliseconds under even moderate load. ALL of these flash-based products have latencies that are hundreds of times faster than disk.

In short, most of the performance improvement comes from simply replacing hard disk with some form of flash. This immediately shifts the performance bottleneck from storage to some other component in your system. As a result, you won’t be able to take full advantage of flash performance without also optimizing the performance of the rest of your infrastructure, and ultimately rewriting your applications as well.

The above phenomenon explains why replacing your hard disk with flash often speeds up your applications by only a factor of two to three rather than ten or a hundred. Congratulations! You’ve just moved the bottleneck from storage to some other component of your system. By Amdahl’s Law, further improving only storage performance has diminishing returns. So while custom packaging does provide significant advantages in latency, most applications are unlikely to benefit until the rest of the computing ecosystem is optimized to take full advantage of flash.

To take a closer look at SSD latencies, I ran the following simple experiment:
1) Erase an MLC SSD so that no logical blocks were actually mapped to flash, and then issue small random reads.
2) Overwrite the entire SSD so that all logical blocks are mapped, and issue the same small random reads in step 1.

The idea here is to measure the software and protocol overheads of accessing flash packaged as SSD separately from accessing the data on the SSD. Reads with no blocks mapped had latencies of around 70us, while the reads with all blocks mapped had latencies of 250us. In this case only a fraction of the overall IO latency was due to SW and protocol overhead, indicating that SSDs may still have significant room for improving latency.

Form factor
Another important issue discussed by both Robin and Matt is the relative cost of flash packaged in SSD versus non-SSD form factors. Robin argues that an SSD costs significantly more $/GB than the underlying flash while Matt argues that non-SSD packaging is expensive to develop, and SSDs provide useful flash management functions as well as hot-swap capability. It’s certainly true that developing custom packaging has a high up front cost, although this is likely balanced by lower unit costs. But as Robin points out, there are also standard packaging options available for non-SSD form factor flash, which may make custom packaging for non-SSD flash unnecessary.

A very important point to keep in mind when thinking about commercially available SSD vs. non-SSD form factors is that SSDs are designed as a substitute for disk, while non-SSD form factors are often designed as substitutes for memory. This means that SSDs focus primarily on reducing $/GB (its greatest weakness vs. disk), while non-SSDs focus on reducing $/IOPS (its greatest weakness vs. DRAM). This explains why SSD is currently much cheaper on a $/GB basis than PCIe flash, while PCIe flash designed as memory expansion is cheaper on a $/IOPS basis than SSD. This is not to say that you can’t build a non-SSD form factor that has lower $/GB than SSD, just that the primary applications for these non-SSD form factors today is usually not as a replacement for disk.

Whether flash in SSD versus non-SSD form factors is better for use in storage subsystems in the long run primarily depends on the relative volumes of these products, and the feature and price sensitivity of the applications these products serve. At this point the ‘winning’ form-factor seems hard to predict. So as a flash subsystem vendor, it seems desirable to keep your options open and ensure that your technology will work well with a variety of packaging options.

More than just a faster disk
But flash is about more than just performance and packing. Flash enables much more than just a faster, denser replacement for disk. With flash, we can finally remove a key mechanical barrier to scaling not only storage systems, but computing systems in general. Going forward, CPU, network and storage can now all scale with improvements in semiconductor technology. When transistors replaced vacuum tubes, we got more than just compact radios; we got simpler, more powerful computing systems. Similarly, flash is a catalyst that will enable far greater levels of automation and functionality for storage and computing systems than is possible today.

I tend to think of the value of new technology as the product of its simplicity times the functionality it offers. It’s clear why functionality is important, but why is simplicity so important? Technology that is simple to use will be used more often, to solve more problems, in less time. As a result, simplicity has a compounding effect on value:

Value = Simplicity * Functionality

How does one measure simplicity? One way is to list the basic steps it takes to perform a task and how long each step takes. One to three is good, four to six is manageable, and anything resembling a twelve step program will likely require written directions and a significant amount of focus. Note that in assessing the simplicity and functionality of a technology, one must do it in the context of the job that needs to be done. For example, a chainsaw has great features for cutting down trees but not for giving haircuts.

A common problem with many general purpose storage products when applied to applications such as virtualization is that they require executing long lists of steps to get anything done – and most of the features are not directly applicable to virtualization. Paradoxically, many of the features that try to make these products better suited to the application end up making the products more complex – resulting in little improvement in overall value. Kind of like adding too many tools to a Swiss army knife until you have so many that the attachments start to stick and rub against each other.

Flash as a catalyst
Flash eliminates a key mechanical barrier to scaling computing systems and is 400 times faster than disk. To keep things in perspective, the speed of sound is “only” 250 times faster than walking! If I could get to work at supersonic speeds, I would no doubt save a lot of time each year. But would I do no more which such an ability? Similarly, is flash just a faster replacement for disk? Will it make no significant difference in the way storage is managed and used? We obviously don’t think so. Flash will greatly increase the value of storage by improving both the simplicity and functionality of enterprise storage products. But these gains will not come easily or without their own set of problems.

An obvious way flash promotes simplicity is by eliminating performance bottlenecks, but as flash enables more dense storage systems many of those gains will be converted to problems in quality-of-service. A more significant way flash promotes value is by providing a better building block for constructing storage systems: flash promotes simplicity by enabling higher levels of automation and allows the implementation of more powerful functionality.

Flash will fragment the enterprise storage market. The general purpose storage systems of today will be supplanted by new flash-based products that are far simpler and more powerful for the specific application areas that they target. This will amplify the simplicity and power that flash already makes possible, and further accelerate the fragmentation of the storage market. This is precisely what happened in the 1980’s when advances in networking technology caused a shift from centralized computing to networked computing – and in the process fragmented the direct attached storage market into ones based on networked storage technology. Over time, the networked storage markets consolidated into the current general purpose storage market dominated by a few major vendors. And so the cycle is repeating itself.

We are at the start of a new technological shift. A shift that is made possible by flash and one that will disrupt the existing enterprise storage market. Just as transistors enabled new products such as personal computers and smart phones, flash will enable simple, intelligent and fast enterprise storage systems. In turn, this will lead to much higher value for end users, but only if we think outside the storage box and treat flash as more than just a faster, denser disk.

The StorageMojo take
For the record the original post wasn’t looking at hybrid solutions, although it is obvious that SSDs can help legacy designs stay competitive without replacing all disks for a few years. For folks like Tintri and Nimble who want to speed up disk storage to stay affordable SSDs make sense. Why engineer a small part of your system when an off-the-shelf solution will suffice?

But for high end transactional SAN storage I still don’t see how SSDs are the right way to go. But I’m expecting more responses, so stay tuned.

Courteous comments welcome, of course. I’m working on a post that reflects directly on Ed’s comment about SSD latency. You’ll like it.

Another StorageMojo Best paper, The Bleak Future of NAND Flash Memory, presented at this year’s FAST ’12 conference, quantifies flash’s declining reliability, endurance, and performance as density increases.

Researchers Laura M. Grupp and Steven Swanson from the UCSD Non-volatile Systems Lab and John D. Davis of Microsoft Research collected data from 45 flash chips from 6 manufacturers. Using that empirical data they predict the performance and cost characteristics of future SSDs.

Faster better cheaper or slower worse cheaper?
While NAND flash is produced with semiconductor processes, smaller feature sizes don’t lead to faster performance or greater reliability. As NAND features shrink, so do the number of trapped electrons that store information.

Figures of merit
The research found that performance, program/erase endurance, energy efficiency, and data retention time all got worse with feature shrink.

Based on past performance, the team derived equations to describe how changes in feature size have affected key specs. They looked at SLC, MLC and TLC and feature sizes scaled from 72 nm to 6.5 nm (the consensus smallest feature size published in the International Technology Roadmap for Semiconductors (ITRS0), and assumed a fixed silicon budget for flash storage.

Key results

• Latency. MLC write latency will double over time. Triple-level cell writes will grow to over 2.5MS, noticably reducing its performance advantage over disk writes.
• Bandwidth. Small – 512B – read bandwidth and all writes decline by up to 50% over time. The impact is greatest on high-performance SLC flash.
• IOPS. MLC flash I/O rates will drop almost in half.

Flash may be the new disk in a few years.

The StorageMojo take
One important qualifier is that for the purposes of their modeling the team constrained the number of chips in the hypothetical future devices whose performance they predicted. While fine for isolating the impact of future chip shrinks, it ignores the potential of much greater parallelism for managing these changes.

Bandwidth drops by half? Double the number of chips.

But if something can’t go on forever, it won’t. NAND flash will soon enter an end-of-life crisis for computer applications that need performance. That’s why ReRAM (resistance RAM) looks to be a good bet for replacing computer flash – not mobile device flash – over the next decade.

Courteous comments welcome, of course. A version of this post was published on ZDNet last week.

A hardware storage controller is an expensive guarantee that you’re using old technology to handle your most important data. Hardware specs are frozen early in the typical 18-24 month development cycle so by the time you get your “new” controller it is already 2 years old.

But it may not have to be that way. In Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization researchers Muli Ben-Yehuda, Michael Factor, Eran Rom, Avishay Traeger, Eran Borovik and Ben-Ami Yassour of IBM Research–Haifa wanted to find out if virtualized storage controller features are feasible.

Short answer: with some tweaking, yes.

The big question is overhead. Storage controllers are typically in the data path, so latency, as well as compute efficiency on out-of-date processors, are real concerns.

Unlike the gateway approach of virtual storage appliances (VSA), the team ran the VMs directly on storage controllers using the Linux KVM hypervisor.

Overhead
The team identified 3 sources of performance overhead:

• Base. System work such as virtual memory managment or process switching.
• External communication. Important if a new function is layered on top of the storage system, such as a file server.
• Internal communication. Virtual machine coordination and communication with the hardware controller.

Reducing overhead
Different techniques are used to limit each type of overhead.

Base They statically allocate CPU cores to the guest to ensure sufficient resources. Memory is also statically allocated to the VM to reduce translation overheads.

External Device assignment is the highest-performing approach as it eliminates hypervisor intervention for physical events. This requires assigning the network device directly to the guest using an SR-IOV (single root I/O virtualization) enabled adapter which allows the guest to send requests directly to the device.

Internal communications To reduce internal communication overhead, they modified KVM’s block driver to poll instead of interrupt. This gives a fast, exit-less, zero-copy transport.

Results

By using these techniques, we show no measurable difference in network latency between bare metal and virtualized I/O and under 5% difference in throughput. For internal communication, micro-benchmarks show 6.6μs latency overhead, read throughput of 357K IOPS, and write throughput of 284K IOPS; roughly seven times better than a base KVM implementation. In addition, an I/O intensive filer workload running in KVM incurs less than 0.4% runtime performance overhead compared to bare metal integration.

That sounds pretty good.

The StorageMojo take
While the static assignments may reduce flexibility, the win is updating storage functionality on the fly. But are there viable use cases? The arc of controller history suggests there are.

The earliest disk drives were directly controlled by the host CPU. Over the decades that and much other functionality migrated to controllers and to disks. Lately that trend has slowed because of large investments in existing standards.

This paper shows that it is possible to migrate more functionality to controllers without lengthy development cycles, enabling architects to make different tradeoffs.

For example, big data requires big pipes, and big pipes are expensive. If volume-reducing preprocessors could be added to file servers, existing bandwidth could be optimized.

More importantly, it suggests that by virtualizing the controller’s applications, the underlying hardware can be updated more frequently. To be fair, that’s not what the authors suggested, but it certainly seems possible based on their work.

Courteous comments welcome, of course. Jeff Darcy of Red Hat has his own list of favorite papers from FAST ’12 here.

FAST – File and Storage Technology – is a must-see conference for StorageMojo, and I’ll be reviewing several Best Papers from FAST ’12 . While most emerging technology is developed in private company labs, FAST is where much of the first publicly available research is published.

Case in point, a StorageMojo Best Paper of FAST ’12: Optimizing NAND Flash-Based SSDs via Retention Relaxation by Ren-Shuo Liu and Chia-Lin Yang of National Taiwan University, and Wei Wu of Intel. NAND engineers have known for years that it is possible to speed up writes by allowing for shorter retention, but this paper quantifies the process.

Data retention was a theme of several papers. Disk drives don’t care if an update needs to last a minute or a year, but flash does.

NAND retention
NAND flash writes are spec’d – by JEDEC – for one year of retention. But relaxing that retention requirement can be beneficial.

• Speed. Writes can be 1.8 to 5.7x faster, depending on how long the data is to be kept.
• SSD architecture. The need for overprovisioning and other choices is a direct result of incoming data rates and flash write speeds. Faster writes might also mean allow less aggressive garbage collection.
• ECC. As feature sizes shrink and NAND cells get flakier, the ECC overhead required to achieve a year’s retention grows. Single error correcting codes used to suffice. Now we need 24-error correcting codes and the arms race continues.

These advantages are meaningless if most writes need to be retained for more than, say, 2 weeks. The authors looked at a number workload traces and found that for all but one of them, at least 50% of the writes were retained for 1 week or less. For active enterprise workloads the percentage is likely to over 75%.

What happens when the time is up?
The authors propose that the Flash Translation Layer keep track of how long each block remains unchanged. When – and if – it reaches the threshold, a background process rewrites the data for the standard 1 year retention.

It is feasible to differentiate between host writes and background writes – garbage collection, for example – and to write them differently. Long-term writes would get improved ECC, while host writes would avoid the costly ECC encoding required.

Yes, there is overhead in managing the fast blocks and rewriting long-term data. But the added performance appears to make that a small price to pay.

The StorageMojo take
The paper presents a strong case for relaxing retention requirements to improve performance. As future generations of flash become less reliable and slower we’ll need this and other techniques to improve – or at least maintain – performance.

Many performance enhancement schemes require unrealistic levels of intelligence about application or system behavior to be effective. But this is within the realm of practical implementation.

The retention issue is a fair example of being handed a lemon and making lemonade. Or offering another degree of freedom to system architects.

In fact, some vendors are already exploring this possibility. If it extends the useful life of flash for a few years it will be well worth the engineering effort.

Courteous comments welcome, of course. A somewhat analogous process for disks is the concept of shingle writes, an area UCSC has been working in. Will disk vendors pick it up?