Over 3 years ago StorageMojo saw that Violin Memory was “. . . on the winning architectural track.” Well, it took a lot of time and money, but Violin is making good on that early promise.

StorageMojo’s enthusiasm was kindled by Violin’s unique architecture. Here’s a short video that shows how Violin’s architecture addresses key problems with flash:

Full screen mode recommended.

The StorageMojo take
The industry is still in the early days of digesting the implications of fast persistent solid state storage. We’ve built up 50 years of cruft to deal with disk’s many issues. It will take a few more years for flash’s new options to ripple through the entire storage, server and application stack.

Take, for example, failover. If all apps and monitoring software could declare a failure in 10 seconds rather than, say, a minute, how much smoother would major apps run? How much better would be the perception of system uptime and response times be?

There are many other possibilities – what about metadata? – that flash and its successor technologies will affect. I’ll be offering more detail in my keynote at the Solid State Storage Symposium on Wednesday, April 25 in Silicon Valley. S4 is free and you can register here.

Courteous comments welcome, of course. The other flash company I liked in 2009 was Fusion-io, and they’ve done OK. And yes, Violin paid StorageMojo to produce the video white paper, but the opinions are my own.

StorageMojo offered its soapbox to any vendors willing to weigh in on the question of whether enterprise arrays should be built from flash SSDs or not. Ed Lee, architect at Tintri, formerly of Data Domain and a Berkeley Ph.D, elected to respond. It is a long piece but rich in insight.

Tintri produces hybrid disk/flash SSD appliances optimized for virtual environments, not Symm-killers. They use SSDs in their products, as do other folks like Nimble Storage.

No money changed hands between Tintri and StorageMojo or related entities. My accountant is weeping in the next room.

Begin Tintri’s response:

Outside the SSD Box: More than Faster Disk
Robin Harris of Storage Mojo in his recent article, “Are SSD-based arrays a bad idea? and Matt Kixmoeller of Pure in his response, The SSD is Key to Economic Flash Arrays, present interesting perspectives on whether or not SSDs are the best technology for building flash-based arrays. Robin argues that by rethinking how flash can be packaged outside the SSD box, you can achieve better performance, reliability, cost and flexibility. And these observations are supported by the experience of existing flash-based storage vendors who have developed their own custom flash modules and packaging. Matt argues that SSDs provide an industry-standard product that requires less investment to leverage, better economies of scale, and rapid improvement in technology. These are also very valid points, especially for startups with limited time and capital.

Latency
Taking latency as a point for comparison, flash-based storage vendors using custom packaging often quote IO latencies in the tens of microseconds versus SSD latencies of low hundreds of microseconds. While this is a notable difference, software and interfaces can also add overhead and the final latency seen at the subsystem level may differ by only a factor of two to four. Server-side flash products can avoid more of the software and interface overhead and provide better latencies – but may require rewriting applications to capitalize on this advantage. Keep in mind that hard disk latencies can easily reach tens of milliseconds under even moderate load. ALL of these flash-based products have latencies that are hundreds of times faster than disk.

In short, most of the performance improvement comes from simply replacing hard disk with some form of flash. This immediately shifts the performance bottleneck from storage to some other component in your system. As a result, you won’t be able to take full advantage of flash performance without also optimizing the performance of the rest of your infrastructure, and ultimately rewriting your applications as well.

The above phenomenon explains why replacing your hard disk with flash often speeds up your applications by only a factor of two to three rather than ten or a hundred. Congratulations! You’ve just moved the bottleneck from storage to some other component of your system. By Amdahl’s Law, further improving only storage performance has diminishing returns. So while custom packaging does provide significant advantages in latency, most applications are unlikely to benefit until the rest of the computing ecosystem is optimized to take full advantage of flash.

To take a closer look at SSD latencies, I ran the following simple experiment:
1) Erase an MLC SSD so that no logical blocks were actually mapped to flash, and then issue small random reads.
2) Overwrite the entire SSD so that all logical blocks are mapped, and issue the same small random reads in step 1.

The idea here is to measure the software and protocol overheads of accessing flash packaged as SSD separately from accessing the data on the SSD. Reads with no blocks mapped had latencies of around 70us, while the reads with all blocks mapped had latencies of 250us. In this case only a fraction of the overall IO latency was due to SW and protocol overhead, indicating that SSDs may still have significant room for improving latency.

Form factor
Another important issue discussed by both Robin and Matt is the relative cost of flash packaged in SSD versus non-SSD form factors. Robin argues that an SSD costs significantly more $/GB than the underlying flash while Matt argues that non-SSD packaging is expensive to develop, and SSDs provide useful flash management functions as well as hot-swap capability. It’s certainly true that developing custom packaging has a high up front cost, although this is likely balanced by lower unit costs. But as Robin points out, there are also standard packaging options available for non-SSD form factor flash, which may make custom packaging for non-SSD flash unnecessary.

A very important point to keep in mind when thinking about commercially available SSD vs. non-SSD form factors is that SSDs are designed as a substitute for disk, while non-SSD form factors are often designed as substitutes for memory. This means that SSDs focus primarily on reducing $/GB (its greatest weakness vs. disk), while non-SSDs focus on reducing $/IOPS (its greatest weakness vs. DRAM). This explains why SSD is currently much cheaper on a $/GB basis than PCIe flash, while PCIe flash designed as memory expansion is cheaper on a $/IOPS basis than SSD. This is not to say that you can’t build a non-SSD form factor that has lower $/GB than SSD, just that the primary applications for these non-SSD form factors today is usually not as a replacement for disk.

Whether flash in SSD versus non-SSD form factors is better for use in storage subsystems in the long run primarily depends on the relative volumes of these products, and the feature and price sensitivity of the applications these products serve. At this point the ‘winning’ form-factor seems hard to predict. So as a flash subsystem vendor, it seems desirable to keep your options open and ensure that your technology will work well with a variety of packaging options.

More than just a faster disk
But flash is about more than just performance and packing. Flash enables much more than just a faster, denser replacement for disk. With flash, we can finally remove a key mechanical barrier to scaling not only storage systems, but computing systems in general. Going forward, CPU, network and storage can now all scale with improvements in semiconductor technology. When transistors replaced vacuum tubes, we got more than just compact radios; we got simpler, more powerful computing systems. Similarly, flash is a catalyst that will enable far greater levels of automation and functionality for storage and computing systems than is possible today.

I tend to think of the value of new technology as the product of its simplicity times the functionality it offers. It’s clear why functionality is important, but why is simplicity so important? Technology that is simple to use will be used more often, to solve more problems, in less time. As a result, simplicity has a compounding effect on value:

Value = Simplicity * Functionality

How does one measure simplicity? One way is to list the basic steps it takes to perform a task and how long each step takes. One to three is good, four to six is manageable, and anything resembling a twelve step program will likely require written directions and a significant amount of focus. Note that in assessing the simplicity and functionality of a technology, one must do it in the context of the job that needs to be done. For example, a chainsaw has great features for cutting down trees but not for giving haircuts.

A common problem with many general purpose storage products when applied to applications such as virtualization is that they require executing long lists of steps to get anything done – and most of the features are not directly applicable to virtualization. Paradoxically, many of the features that try to make these products better suited to the application end up making the products more complex – resulting in little improvement in overall value. Kind of like adding too many tools to a Swiss army knife until you have so many that the attachments start to stick and rub against each other.

Flash as a catalyst
Flash eliminates a key mechanical barrier to scaling computing systems and is 400 times faster than disk. To keep things in perspective, the speed of sound is “only” 250 times faster than walking! If I could get to work at supersonic speeds, I would no doubt save a lot of time each year. But would I do no more which such an ability? Similarly, is flash just a faster replacement for disk? Will it make no significant difference in the way storage is managed and used? We obviously don’t think so. Flash will greatly increase the value of storage by improving both the simplicity and functionality of enterprise storage products. But these gains will not come easily or without their own set of problems.

An obvious way flash promotes simplicity is by eliminating performance bottlenecks, but as flash enables more dense storage systems many of those gains will be converted to problems in quality-of-service. A more significant way flash promotes value is by providing a better building block for constructing storage systems: flash promotes simplicity by enabling higher levels of automation and allows the implementation of more powerful functionality.

Flash will fragment the enterprise storage market. The general purpose storage systems of today will be supplanted by new flash-based products that are far simpler and more powerful for the specific application areas that they target. This will amplify the simplicity and power that flash already makes possible, and further accelerate the fragmentation of the storage market. This is precisely what happened in the 1980’s when advances in networking technology caused a shift from centralized computing to networked computing – and in the process fragmented the direct attached storage market into ones based on networked storage technology. Over time, the networked storage markets consolidated into the current general purpose storage market dominated by a few major vendors. And so the cycle is repeating itself.

We are at the start of a new technological shift. A shift that is made possible by flash and one that will disrupt the existing enterprise storage market. Just as transistors enabled new products such as personal computers and smart phones, flash will enable simple, intelligent and fast enterprise storage systems. In turn, this will lead to much higher value for end users, but only if we think outside the storage box and treat flash as more than just a faster, denser disk.

The StorageMojo take
For the record the original post wasn’t looking at hybrid solutions, although it is obvious that SSDs can help legacy designs stay competitive without replacing all disks for a few years. For folks like Tintri and Nimble who want to speed up disk storage to stay affordable SSDs make sense. Why engineer a small part of your system when an off-the-shelf solution will suffice?

But for high end transactional SAN storage I still don’t see how SSDs are the right way to go. But I’m expecting more responses, so stay tuned.

Courteous comments welcome, of course. I’m working on a post that reflects directly on Ed’s comment about SSD latency. You’ll like it.

Another StorageMojo Best paper, The Bleak Future of NAND Flash Memory, presented at this year’s FAST ’12 conference, quantifies flash’s declining reliability, endurance, and performance as density increases.

Researchers Laura M. Grupp and Steven Swanson from the UCSD Non-volatile Systems Lab and John D. Davis of Microsoft Research collected data from 45 flash chips from 6 manufacturers. Using that empirical data they predict the performance and cost characteristics of future SSDs.

Faster better cheaper or slower worse cheaper?
While NAND flash is produced with semiconductor processes, smaller feature sizes don’t lead to faster performance or greater reliability. As NAND features shrink, so do the number of trapped electrons that store information.

Figures of merit
The research found that performance, program/erase endurance, energy efficiency, and data retention time all got worse with feature shrink.

Based on past performance, the team derived equations to describe how changes in feature size have affected key specs. They looked at SLC, MLC and TLC and feature sizes scaled from 72 nm to 6.5 nm (the consensus smallest feature size published in the International Technology Roadmap for Semiconductors (ITRS0), and assumed a fixed silicon budget for flash storage.

Key results

• Latency. MLC write latency will double over time. Triple-level cell writes will grow to over 2.5MS, noticably reducing its performance advantage over disk writes.
• Bandwidth. Small – 512B – read bandwidth and all writes decline by up to 50% over time. The impact is greatest on high-performance SLC flash.
• IOPS. MLC flash I/O rates will drop almost in half.

Flash may be the new disk in a few years.

The StorageMojo take
One important qualifier is that for the purposes of their modeling the team constrained the number of chips in the hypothetical future devices whose performance they predicted. While fine for isolating the impact of future chip shrinks, it ignores the potential of much greater parallelism for managing these changes.

Bandwidth drops by half? Double the number of chips.

But if something can’t go on forever, it won’t. NAND flash will soon enter an end-of-life crisis for computer applications that need performance. That’s why ReRAM (resistance RAM) looks to be a good bet for replacing computer flash – not mobile device flash – over the next decade.

Courteous comments welcome, of course. A version of this post was published on ZDNet last week.

A hardware storage controller is an expensive guarantee that you’re using old technology to handle your most important data. Hardware specs are frozen early in the typical 18-24 month development cycle so by the time you get your “new” controller it is already 2 years old.

But it may not have to be that way. In Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization researchers Muli Ben-Yehuda, Michael Factor, Eran Rom, Avishay Traeger, Eran Borovik and Ben-Ami Yassour of IBM Research–Haifa wanted to find out if virtualized storage controller features are feasible.

Short answer: with some tweaking, yes.

The big question is overhead. Storage controllers are typically in the data path, so latency, as well as compute efficiency on out-of-date processors, are real concerns.

Unlike the gateway approach of virtual storage appliances (VSA), the team ran the VMs directly on storage controllers using the Linux KVM hypervisor.

Overhead
The team identified 3 sources of performance overhead:

• Base. System work such as virtual memory managment or process switching.
• External communication. Important if a new function is layered on top of the storage system, such as a file server.
• Internal communication. Virtual machine coordination and communication with the hardware controller.

Reducing overhead
Different techniques are used to limit each type of overhead.

Base They statically allocate CPU cores to the guest to ensure sufficient resources. Memory is also statically allocated to the VM to reduce translation overheads.

External Device assignment is the highest-performing approach as it eliminates hypervisor intervention for physical events. This requires assigning the network device directly to the guest using an SR-IOV (single root I/O virtualization) enabled adapter which allows the guest to send requests directly to the device.

Internal communications To reduce internal communication overhead, they modified KVM’s block driver to poll instead of interrupt. This gives a fast, exit-less, zero-copy transport.

Results

By using these techniques, we show no measurable difference in network latency between bare metal and virtualized I/O and under 5% difference in throughput. For internal communication, micro-benchmarks show 6.6μs latency overhead, read throughput of 357K IOPS, and write throughput of 284K IOPS; roughly seven times better than a base KVM implementation. In addition, an I/O intensive filer workload running in KVM incurs less than 0.4% runtime performance overhead compared to bare metal integration.

That sounds pretty good.

The StorageMojo take
While the static assignments may reduce flexibility, the win is updating storage functionality on the fly. But are there viable use cases? The arc of controller history suggests there are.

The earliest disk drives were directly controlled by the host CPU. Over the decades that and much other functionality migrated to controllers and to disks. Lately that trend has slowed because of large investments in existing standards.

This paper shows that it is possible to migrate more functionality to controllers without lengthy development cycles, enabling architects to make different tradeoffs.

For example, big data requires big pipes, and big pipes are expensive. If volume-reducing preprocessors could be added to file servers, existing bandwidth could be optimized.

More importantly, it suggests that by virtualizing the controller’s applications, the underlying hardware can be updated more frequently. To be fair, that’s not what the authors suggested, but it certainly seems possible based on their work.

Courteous comments welcome, of course. Jeff Darcy of Red Hat has his own list of favorite papers from FAST ’12 here.

FAST – File and Storage Technology – is a must-see conference for StorageMojo, and I’ll be reviewing several Best Papers from FAST ’12 . While most emerging technology is developed in private company labs, FAST is where much of the first publicly available research is published.

Case in point, a StorageMojo Best Paper of FAST ’12: Optimizing NAND Flash-Based SSDs via Retention Relaxation by Ren-Shuo Liu and Chia-Lin Yang of National Taiwan University, and Wei Wu of Intel. NAND engineers have known for years that it is possible to speed up writes by allowing for shorter retention, but this paper quantifies the process.

Data retention was a theme of several papers. Disk drives don’t care if an update needs to last a minute or a year, but flash does.

NAND retention
NAND flash writes are spec’d – by JEDEC – for one year of retention. But relaxing that retention requirement can be beneficial.

• Speed. Writes can be 1.8 to 5.7x faster, depending on how long the data is to be kept.
• SSD architecture. The need for overprovisioning and other choices is a direct result of incoming data rates and flash write speeds. Faster writes might also mean allow less aggressive garbage collection.
• ECC. As feature sizes shrink and NAND cells get flakier, the ECC overhead required to achieve a year’s retention grows. Single error correcting codes used to suffice. Now we need 24-error correcting codes and the arms race continues.

These advantages are meaningless if most writes need to be retained for more than, say, 2 weeks. The authors looked at a number workload traces and found that for all but one of them, at least 50% of the writes were retained for 1 week or less. For active enterprise workloads the percentage is likely to over 75%.

What happens when the time is up?
The authors propose that the Flash Translation Layer keep track of how long each block remains unchanged. When – and if – it reaches the threshold, a background process rewrites the data for the standard 1 year retention.

It is feasible to differentiate between host writes and background writes – garbage collection, for example – and to write them differently. Long-term writes would get improved ECC, while host writes would avoid the costly ECC encoding required.

Yes, there is overhead in managing the fast blocks and rewriting long-term data. But the added performance appears to make that a small price to pay.

The StorageMojo take
The paper presents a strong case for relaxing retention requirements to improve performance. As future generations of flash become less reliable and slower we’ll need this and other techniques to improve – or at least maintain – performance.

Many performance enhancement schemes require unrealistic levels of intelligence about application or system behavior to be effective. But this is within the realm of practical implementation.

The retention issue is a fair example of being handed a lemon and making lemonade. Or offering another degree of freedom to system architects.

In fact, some vendors are already exploring this possibility. If it extends the useful life of flash for a few years it will be well worth the engineering effort.

Courteous comments welcome, of course. A somewhat analogous process for disks is the concept of shingle writes, an area UCSC has been working in. Will disk vendors pick it up?

Amazon Web Services architect James Hamilton has been posting on network issues for over a year and researching them much longer. As Ethernet becomes the de facto SAN technology, his views become more relevant to the larger storage market.

Critique
Part of Mr. Hamilton’s concern is the structure of the networking industry: the high margins; the dominance of a single player, Cisco; the closed technology; and the heavy vertical integration. All antithetical to the dynamics that have driven server costs down so successfully in the last 20 years.

These are issues the storage industry knows too well. But Mr. Hamilton is more concerned about the waste the current high-cost industry structure causes.

Waste?

Workload placement
The cost of network bandwidth leads to network over-subscription. Networks are configured as tree topologies: the further you move from end nodes the worse the over subscription.

As described in the 2009 Microsoft Research paper VL2: A Scalable and Flexible Data Center Network:

. . . the capacity between different branches of the tree is typically over- subscribed by factors of 1:5 or more, with paths through the highest levels of the tree oversubscribed by factors of 1:80 to 1:240. This limits communication between servers to the point that it fragments the server pool — congestion and computation hot-spots are prevalent even when spare capacity is available elsewhere.

This throttles data center performance by limiting server-to-server bandwidth, fragmenting resources and reducing network utilization. The latter reflects the redundant paths needed in case of switch failure: ≈50% or more of costly data center bandwidth goes unused.

As might be expected, big Internet data centers like Amazon’s have complex and unpredictable workloads. They need lots of bandwidth between all servers all the time.

A solution
The VL2 paper describes an experimental solution to these problems that includes location-specific and application-specific addressing, multi-path traffic load balancing and a novel directory design that efficiently handles lookups and updates to network mappings.

In an 75-node test cluster the design moved 2.75TB of data in 395 seconds – 94% of maximum network bandwidth – at a fraction of the cost of current enterprise networks. The paper calculates that a cloud-service scale network with no over-subscription could be built with commodity switches at 1/14th the cost of a traditional data center Ethernet.

Whoa!

The StorageMojo take
VC and engineering dollars follow high-growth markets. What Google, Amazon and Microsoft want, they get. With the rapid growth of public cloud services the network over-subscription problem will get solved.

Merchant silicon from Broadcom, Intel and Marvell is making a tried-and-true Moore’s Law attack on hardware cost. The protocol stack is tougher, but several open-source industry initiatives are under way with support from major companies. Progress will be slower than hoped, but within 3 years we’ll have a viable stack to build on.

Where does this leave the networking industry? That depends on where you sit.

Cisco will be the biggest loser, because they’ve been the biggest winner with the current model. They may need to pull an IBM and move big into services if they want to stick around. Ironically, Cisco’s UCS product line – which bakes in the tree-structured network – has further motivated broader industry action.

The rest of the industry can go after this emerging market with a lower-GM business model. Not all of them will, but it will be a critical success factor.

The big winner will be storage. Scale-out storage relies on spraying data across multiple racks for maximum availability, utilization and performance. Cheaper, faster, better scale-out networks will only drive storage demand.

For most of us this is an academic problem today. Lightly used systems – such as for backup and archiving – don’t see Amazon’s problems. But in 5 years this will be common even outside the public cloud providers.

Just as IT users have benefited from Google’s push on energy efficiency and much more, they will also benefit from much lower cost and more scalable networks.

Courteous comments welcome, of course. I can’t help but continue to marvel at how dumb Cisco’s UCS has turned out to be. It’s a gift that keeps on giving.

One more datapoint and we’ll have a trend: NoSQL databases managing metadata. It’s obvious in retrospect: use a scalable big data tool to handle scale-out metadata. Maybe not a requirement today, but surely will be with even bigger data tomorrow.

Metadata is a fraction of the user data set, but it gets hammered much more. As more metadata is found useful the hammering will get more insistent.

Nutanix
Nutanix, whose CTO and co-founder, Mohit Aron, was a developer of the Google File System, uses MapReduce. Nutanix achieves it scale due to its distributed metadata, masterless architecture – powered by MapReduce jobs that run in the background.

Druva
Druva, a backup company for mobile devices, also uses a NoSQL database to manage storage metadata. They say they’ve found that NoSQL scales over an order of magnitude better than relational in similar applications.

Somebody else
A company that shall remain nameless is porting Hadoop to their backend. The customer won’t be able to access Hadoop for their work – it is strictly for the system’s internal use.

It is a proof of concept so it isn’t a 3rd data point, but they see the potential advantages. Call it data point 2½.

The StorageMojo take
Small advances are the building blocks of disruption. RAID made it possible to build available storage using cheap disks. Consumer adoption of PCs made disks even cheaper. Moore’s Law made RAID controllers cheaper and faster, or faster and more capable.

A virtuous circle of disruption.

The basic architecture of scale-out storage systems – purpose-built software on clustered commodity hardware – has been stable. But this is the beginning of scale-out storage 2.0: taking scale-out technology developed for users and incorporating it into the storage infrastructure itself.

These ideas are bubbling up among the latest startups and among the establishment players. At some point the old RAID architectures will be well and truly broken, able to compete in smaller and smaller niches until the revenue can’t justify more investment.

Of course vendors have been making RAID controllers out of servers for years now, and those servers can run any software they want. But at some point the explicit and implicit assumptions in the old architecture crash into current realities – either in cost, development time, feature completeness or management overhead – and then we move on.

Courteous comments welcome, of course. I learned about Nutanix at the last Tech Field Day “The Independent IT Influencer Event” which paid for my travel expenses to Silicon Valley.

CES is a lot of fun, but my favorite toy trade show is the National Association of Broadcasters (NAB) convention. I arrive tomorrow and return Wednesday and hope – on the way back – to walk across the new bridge that is 900 feet above the Colorado River at Hoover Dam.

I do video work today and did FM broadcasting decades ago. Today we’re all narrowcasting, but digital has made the tech now so much better – and cheaper! – than what broadcasters had 10 years ago that the possibilities are endless.

Pre-show expectations
Rumor has it that Apple will announce the newest version of Final Cut Studio – my preferred editing platform – on Tuesday. I’m also looking for any and all Thunderbolt peripherals, although the pre-show PR hasn’t mentioned it. I hope Promise and LaCie have something to show, and perhaps Sony as well.

Object storage should be more visible this year as well. And where will USB 3.0 show up – if it shows up anywhere – in pro gear?

The StorageMojo take
There’s something about pro gear – even though I can’t afford 98% of it and wouldn’t use it to best advantage if I could – that fascinates. Built into all the features and specs is deep knowledge of the technology and its limitations.

There are people who spend $5k for a microphone to record a single instrument. Their ears can discern the differences between equally high-end mics and they know how to mix them to get the sound they want.

Then we listen to all that painstaking work through crummy little earbuds. Oh well!

If you have something to share please contact me through the comments. I’m looking for cool stuff.

Courteous comments welcome, of course.

The younger StorageMojo analysts have heard that New York is a den of sin and depravity and can’t wait to try some. With new Wrangler jeans and polished silver bolos they are ready to par-tay with some big city gals. Yee-ha!

Should I tell them that it will be 95% male? Nah.

The event is GigaOm’s Structure Big Data 2011 conference on Wednesday the 23rd. Haven’t been to it before, but I was overdue to see what Mr. Malik has cooked up.

The StorageMojo take
With the gradual slowing of improvement in hardware and big plans for massive data collection, the world of storage is looking at accelerating change. Drive and CPU vendors won’t be doing all the heavy lifting.

That means that architecture will become even more critical to successful products. Lots of room for creativity and innovation because there is a ready audience that needs something better than what they have today.

Looking forward to meeting new people and learning about new technologies and markets. I’ll have some time Thursday morning too, if anyone is eager to bend my ear. Leave a comment with your preferred means of contact.

Courteous comments welcome, of course.

Update: The survey is now closed. Thanks to everyone who responded!

I’ll have more later on the results. End update.

StorageMojo would like your help.

TechnoQWAN LLC, StorageMojo’s publisher, is a research and analysis firm. An IT supplier has retained us to help plan their internal storage strategy .

You’ll be anonymous, unless you choose not to be. This is a research project, not a marketing Trojan horse.

How you can help
Please donate 5 to 8 minutes to complete a survey. You’ll help keep StorageMojo the independent source of storage coolness it has always strived to be.

We’d like to get a couple of hundred respondents RSN. Can you do it now?

Here’s the link to the Internal Storage Survey.

And please pass the link on to likely friends and colleagues. The more the merrier!

The StorageMojo take
After we’ve looked at the data I plan to write about what I’ve found interesting in the results. You’ll get to learn something about the rest of the StorageMojo community.

And I’ll get to learn some more about you.

If, perchance, you’re passionate about the topic, you’ll be able to volunteer for a more in-depth discussion. Trust me: the sponsor can make a real difference in the servers you buy in a couple of years.

Courteous comments welcome, of course. Completed surveys even more so! BTW, QWAN = Quality Without A Name.

After a quick scan of the paper titles I wasn’t impressed. But after seeing presentations and posters I am.

Here’s some I found interesting. I’ll be posting longer pieces on some of these.

• A Study of Practical Deduplication Full paper *Best Paper Winner*
• Tradeoffs in Scalable Data Routing for Deduplication Clusters Full paper
• Exploiting Half-Wits: Smarter Storage for Low-Power Devices Full paper
• Reliably Erasing Data from Flash-Based Solid State Drives Full paper
• Scale and Concurrency of GIGA+: File System Directories with Millions of Files Full paper
• Emulating Goliath Storage Systems with David Full paper *Best Paper Winner*

An excellent conference. NetApp, EMC, Microsoft and IBM were recruiting.

The StorageMojo take
We’re still learning about flash, and the research presented here is a substantial addition to our meager knowledge.

Microsoft tells me they’re delivering major improvements to NTFS and Windows Server later this year. I’m looking forward to that briefing.

And it’s always a pleasure catching up with the people who, for some reason, never come to Sedona.

Courteous comments welcome, as always.

It’s that time of the year again: the Usenix File And Storage Technology conference in San Jose next Tuesday thru Thursday.

The elite StorageMojo analyst SWAT unit will be there, in color coordinated Kevlar and Spandex, rappelling into the Marriott ballroom. So come by and say hello.

The StorageMojo take
No obvious must-reads among the papers this year, so we’ll have to wait and see if lightning strikes.

Courteous comments welcome, of course.

The performance increase in individual CPUs is slowing to a crawl. All the easy wins – higher clock speeds, wider datapaths, more DRAM, larger registers and caches, 2-4 cores – have been exploited.

Doctor, is there any hope?
In the recent PCAST report on Federal technological initiatives (see Fed funding for our digital future) (pdf) one sidebar suggested “Progress in Algorithms Beats Moore’s Law.”

. . . in many areas, performance gains due to improvements in algorithms have vastly exceeded even the dramatic performance gains due to increased processor speed.

The algorithms that we use today for speech recognition, for natural language translation, for chess playing, for logistics planning, have evolved remarkably in the past decade. It’s difficult to quantify the improvement, though, because it is as much in the realm of quality as of execution time.

In the field of numerical algorithms, however, the improvement can be quantified. Here is just one example . . . a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later – in 2003 – this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms!

The StorageMojo take
Let’s file this one under “Wishful thinking” along with “US housing prices will never decline.” Piecemeal enhancements of specific application areas cannot replace the generalized performance improvements we’ve seen for decades.

No doubt there are important algorithmic improvements to be made. And that in certain problem spaces those speedups will far exceed Moore’s Law – even though the Law is about transistor count, not performance.

That doesn’t change the fact of computation today: the era of predictable and rapid performance improvement is over. Like a vein of rich ore that thins out, our computers will still improve, but the effort needed to do so is rising fast.

Cheap(er) SSDs, larger memories and caches are helping mask the performance plateau by increasing system performance, but reduced I/O latency and increased bandwidth will only take us so far. The way forward is a game of wringing out single-digit percent improvements, not the 2-3 year doubling of the last 60 years.

Courteous comments welcome, of course. The professor whose work the PCAST quote refers to is Martin Grötschel of Konrad-Zuse-Zentrum in Berlin. He’s been doing brilliant work on optimization problems- including the traveling salesman problem and data network design – for decades.

Talked to a company last week whose cloud app handles several billion transactions per month on a cluster. Sounds like SSDs could help them but how?

In a paper from the latest 5th Biennial Conference on Innovative Data Systems Research (CIDR ’11) researchers Philip A. Bernstein and Colin W. Reid of Microsoft and Sudipto Das of UC Santa Barbara have a suggestion: Hyder – A Transactional Record Manager for Shared Flash (pdf).

As underlying hardware changes – faster networks, large memories, multi-core CPUs and SSDs – database software architectures may change too. Hyder architecture supports

. . . reads and writes on indexed records within classical multi-step transactions. It is designed to run on a cluster of servers that have shared access to a large pool of network-addressable raw flash chips. . . . Hyder uses a data-sharing architecture that scales out without partitioning the database or application.

No partition scale-out
Today, most popular database clusters partition the database across multiple servers. Done well this works, but at some cost. The database design is non-trivial – cross-partition transactions, cache coherence, load balancing, scaling and multi-server debugging – are knotty issues which translate into higher design and operation costs.

Hyder eliminates partitioning, distributed programming, layers of cache, remote procedure calls and load balancing. All servers can read and write the entire database – so any server can execute any transaction. Load-balancing is simple: direct new transactions to lightly-loaded servers.

Each update transaction runs on one machine and writes to a shared log – so there’s no 2-phase commit. And no 2-phase commit locking, which can force performance off a cliff when workloads spike.

The 3 main components of Hyder are the log, the index and the roll-forward algorithm.

Log
The log runs on multiple flash devices – chips, DIMMs or ??? – and writes multi-page log records across multiple devices with parity to enable log recovery after device failures.

Hyder uses a multi-versioned database – old record versions aren’t updated-in-place, only the most recent version of a record is used – which has a couple of useful properties:

• Server caches are inherently coherent since only the most recent versions of records are used.
• Data can be read while writes are in progress.
• Queries that can be decomposed can be run across multiple servers concurrently for a faster response time.

[This may seem like voodoo to ACIDheads. A good technical intro to multi-versioning concurrency control (MVCC) is Multi-core software: to gain speed, eliminate resource contention.]

Servers run a cache update process that keeps them current with updated records. Server caches don’t have to be identical and the cache invalidate messages that most clusters use for cache coherency aren’t needed.

All log writes are idempotent appends, so if a write fails the server can simply reissue the write. The authors describe several error modes and how Hyder handles them.

Index
The index stores the database as a search tree with each node a [key, payload] pair. The tree can store, for example, a relational database. The index tree is also represented in the log.

Tree nodes are not updated in place. When node n is updated, a new copy – n’is created. Then, of course, the parent node must be updated and so on up the tree.

A binary tree minimizes the number of node updates, but can be processor intensive. The optimal tree structure for Hyder is not yet resolved.

Garbage collection is an issue. Each node pointer includes the ID of the oldest reachable data element. An element older than any that is pointed to by a node is garbage.

Roll-forward algorithm
This is the key process of Hyder.

When a record update begins, one server executes the transaction. The server is given a copy of the latest database root, a static snapshot of the entire database.

The updates are stored in a local cache and after execution the after-images are gathered into an intention record, which is broadcast to all servers and appended to the log. The update’s readset is included in the intention record, to insure all intentions are properly ordered, none are lost, and the offset is made known to all servers.

Each server can assemble a local copy of the tail of the log, which is used to determine if there are conflicting updates. The meld procedure manages conflicting updates.

Appending the intention to the database log doesn’t commit the transaction. The intention references the static snapshot of the latest database root. The meld procedure determines if any committed transactions since the snapshot conflict with the intention.

If they don’t, all is good. If they do, the transaction is aborted.

All servers roll forward using meld and don’t message each other about committed and failed transactions. Therefore there is no lock manager and no 2-phase commit.

Contention
Losing the lock manager and 2-phase commit should help performance unless other points of contention throttle the system. Hyder’s points of contention include appending intentions to the log, melding the log at each server, and aborting transactions.

Intention appends are serial. The lower the write latency the more appends can be written. A 10us write latency means a 100k TPS.

Network latency adds to write latency. Faster switches improve append performance.

The abort rate depends on the number of concurrent transactions that conflict. Fast transactions reduce the probability of aborts by reducing the number of concurrent transactions.

The worst case is a record subject to multiple updates from different servers. Detecting high-conflict transactions and serializing them by forcing them onto 1 server would reduce the hot data performance hit.

Performance
The authors model Hyder’s performance with a focus on the high-contention corner cases. In general, the tests show linear scaling as servers are added.

The problems come when the underlying hardware limits are exceeded. Increasing execution times mean more aborts and performance falls off a cliff. From the paper:

The StorageMojo take
We’ve been building disk workarounds for for decades. We now tend to assume those workarounds are fundamental architectural requirements rather than hacks.

The Hyder paper asks us to imagine a world where non-volatile mass storage is fast and cheap – and how we could re-architect basic systems to be faster and cheaper too.

The authors conclusion is a fair assessment:

Many variations of the Hyder architecture and algorithms would be worth exploring. There may also be opportunities to use Hyder’s logging and meld algorithms with some modification in other contexts, such as file systems and middleware. We suggested a number of directions for future work throughout the paper. No doubt there are many other directions as well.

Courteous comments welcome, of course. I hope to get to some of the other CIDR papers before FAST ’11 snows me under. Update: Phil Bernstein was kind enough to scan the post and I’ve updated 1 minor error. He also mentioned that it won the Best Paper award at the conference. Those CIDR folks have great taste in papers, don’t they?