IDC says HP is selling 17% less storage last quarter than a year ago. Was that because the high-end EVA and XP businesses were contracting faster than the new 3PAR converged storage business was growing?

IDC definitions
IDC defines a disk storage system broadly, including anything with a controller, cables and 3 or more drives as external storage. That covers a lot of ground. They also have another segment, called open networked disk storage (NAS Combined with non-mainframe SAN) that kicks out direct-attach storage.

In either case IDC counts OEM revenue for the branding company, not the supplier. So their numbers overstate, say, IBM’s storage presence – which includes the NetApp FAS arrays branded as IBM’s N-series – while understating NetApp’s actual shipments. Hitachi, who also OEMs to HP and Sun, also suffers in the IDC rankings.

The HP answer
At HP Discover I asked David Scott, formerly 3PAR’s CEO and now head of HP storage, reporting to Dave Donatelli, formerly of EMC, about this. He answered that the high-end HP storage business was growing and that the contraction was in the low-end MSA and JBOD business.

According to Mr. Scott, due to HP’s current troubles, the server group has been moving away from low-margin business, as well as building servers that have up to 62 drives internally (HP ProLiant SL4540 Gen8 Server). Double whammy: server sales are down, and with them, the easy add-on sales of low-end storage; and the servers they do sell have more storage internally and don’t count as external storage sales.

As the largest seller of servers and disk drives, it doesn’t take much margin growth for both to drop off. It’s an intensely competitive market with lots of low-margin business that can juice the top line while eviscerating the bottom line.

The StorageMojo take
The high-end storage market is consolidating. EMC is winning big, and if traditional storage arrays are what you want, they should.

HP is the only major player with a tightly integrated next-gen product strategy that is a stark contrast to EMC’s product stovepipes. Any enterprise storage product/architecture eval needs to include HP.

Now that they’ve added the HP 3PAR StoreServ 7450 flash array HP has covered most of the waterfront in enterprise storage. Despite the inevitable headwinds during the largest product transition of any storage vendor, HP is well-positioned for the future.

Market dynamics are just that – dynamic – and quarterly numbers are only a snapshot. So while I’m having trouble making sense of the numbers, I know that big transitions are rarely smooth. All should become clear in the fullness of time.

HP storage has great technology. Their real challenge is getting the HP sales force revved up to compete better against EMC and NetApp. HP’s tech culture doesn’t appreciate what salesmen need to win, so I’d expect that’s where Scott and Donatelli are spending a lot of their time.

Courteous comments welcome, of course. HP is putting me up here in Las Vegas – a favorite drive of mine from NoAZ – but they aren’t paying for my attendance. Bummer.

Have you ever noticed that it is difficult to get good information about how flash works? The vendors know but they’ve never been terribly forthcoming.

For example, how does flash wear out? When most things break you lose their contents. But once flash stops working your data is still there. Huh?

And the fact that flash is a wearing medium spooks many people. How should we think about flash? Can we live with a wearing medium?

Or write amplification? How does that work? What can be done to reduce it?

That’s why it was a pleasure to sit down with Rob Ober of LSI. Rob is an LSI Fellow and system architect with deep technical knowledge of flash and how it interacts with systems and applications.

Rob holds dozens of patents and is articulate and open. Plus he’s a very nice guy.

I distilled down what I learned and some of Rob’s key points into a StorageMojo video white paper that LSI commissioned. If you are curious about flash, how it works, how it fails and how it can be turned into an enterprise class storage medium, you’ll find the video informative.

At least I did my level best to make it so, including video from Wilson Canyon, one of my favorite local hikes. Here’s the video:

The StorageMojo take
As a thought experiment I sometimes wonder about how storage would be different if IBM had invented flash back in 1956 instead of the RAMAC disk drive. What it reads were fast and free while writes were expensive?

That’s essentially the problem we’re trying to solve today. Except today we have an installed base of a couple billion disk drives and decades of driver, OS and application development all predicated on disk performance.

We’re still in the early days of flash integration, even though forward-leaning architects have been working on it for 6 years or more. Thanks to flash – and cloud – storage has never been more vibrant or exciting.

Courteous comments welcome, of course. Feel free to ask about anything in the video that wasn’t clear or didn’t go deep enough. Your questions help me understand what you find valuable.

Note: This post got so long it needed to be posted in 2 parts. Part 1 is here. And while I promised this 2nd part “tomorrow” the editing took much longer than expected. End note.

HP has made the most dramatic bet with their 3PAR-based converged storage line. While the rapid growth of the new products is overwhelmed by the even quicker decline of older products like EVA, they’re off to a good start, claiming over 1200 new customeres.

The challenge for former EMC’er Dave Donatelli and 3PAR’s David Scott isn’t technology but sales. Do you have field storage sales specialists and, if so, how do you compensate everyone so account managers are comfortable bringing them into large deals? Or do you flog marketing to make selling converged storage so simple and remunerative that server guys do it themselves?

Ideally both, but the HP go-to marketing strategy is mind-numbing detail and then tossing it over the wall for sales to figure out. That’s not a winning strategy these days.

Bottom line: HP’s lead in physical servers makes them a threat for many add-on storage sales. But EMC’s storage-focused sales force keeps winning deals.

IBM‘s storage business is at risk. While they have great technology, none of their hardware products are even a strong #2. Two thirds of IBM’s Systems and Technology group business is servers, and it is clear that IBM management isn’t happy with how some product lines are performing.

IBM top management is not sentimental. Given the disposal of once-core assets, such as disk drives, and IBM’s shrinking storage market share, it is clear that IBM will likely sell off or shutter some hardware product lines to focus on higher-margin storage. The Texas Memory Systems solid state arrays, storage software and mainframe storage are likely safe.

But at some point – and that point will come sooner rather than later due to the cloud’s competitive pressure – IBM’s management will decide that the overhead and investment required to support 3rd place and worse products isn’t worth it. That’s likely behind the **BILLION DOLLAR ALL FLASH** data center initiative: a desperate attempt to stay relevant in a rapidly changing storage world.

Expect to see less-than-competitive hardware lines put on life-support with cheap-to-implement HW upgrades – new, bigger disks! now with SSD added! – and new OS quals, until most of the base has migrated or the profits are gone. It is a sad comedown for the company that created and dominated the storage business for decades and then lost it to an upstart named EMC.

Who is the final dwarf?
Well, that’s 6 companies. Who’s the final dwarf? Fujitsu? NEC? Cisco? Huawei? Yes, Huawei is in the storage business.

I like Fujitsu and their Eternus system for the final spot, although NEC’s HYDRAstor has some great scale-out technology. But is NEC getting traction? If they are it’s a well-kept secret.

Eternus is probably doing best in Japan, a large enough market to keep them going for years. But don’t expect any game-changers.

The StorageMojo take
The most important impact of cloud infrastructure is that it gives CFOs a yardstick to beat their CIO with measure their internal IT against. And they aren’t happy with what they’re seeing.

For decades IT has been the wayward child in the corporate family. Cost overruns, service delays, unfathomable mumbo-jumbo, security lapses – IT is a never ending string of expensive problems – as the short life of CIOs attests.

But the need for IT infrastructure is only growing. It’s the means that are changing. That’s where the aggressive, new architecture storage companies like Fusion-io/NexGen, Tegile, Nimble, Avere, Kaminario, DDN, Tintri, Nutanix and now Exablox come in.

Since the advent of RAID systems in the early 90s we haven’t seen much turnover in the major players. That’s about to change.

While it will be wrenching for those involved, it will be good for the industry and for the myriad organizations that need reliable, cost-effective storage. People will buy large arrays for decades to come – just as they still buy mainframes – but they’ll have many more options for non-transactional storage.

Courteous comments welcome, of course. Texas Memory Systems was a long-time advertiser on StorageMojo until recently. HP flew me to their corporate trade show in Europe last year to get the converged storage story direct from David Scott and others. I’ve done work for some of the other firms mentioned.

In response to the post on Avere’s architecture for fronting backend NAS filers – where StorageMojo said that no front-end to NAS boxes has succeeded – alert reader Jacob Marley asked “What about F5′s ARX to stitch/balance storage across multiple filers?”

Good question! What can we deduce from publicly available sources?

The F5 ARX product line is billed as an “intelligent file virtualization solution” that
“. . .preserves the logical access to files regardless of their current location on storage.” Like earlier file switches

The ARX device does not introduce a new file system; rather it acts as a proxy to the file systems that are already there.

ARX is not a storage device itself but a load-balancer for NAS filers. Then, per Mr Marley’s question, is ARX not a success?

Competitive analysis
First up, let’s take a look at the latest quarterly 10-Q report, courtesy of the SEC’s EDGAR database.

In “Management’s Discussion and Analysis of Financial Condition and Results of Operations” they describe their product revenues as

The majority of our revenues are derived from sales of our application delivery networking (ADN) products including our high end VIPRION chassis and related software modules; BIG-IP Local Traffic Manager, BIG-IP Global Traffic Manager, BIG-IP Link Controller, BIG-IP Application Security Manager, BIG-IP Edge Gateway, BIG-IP WAN Optimization module, BIG-IP Access Policy Manager, WebAccelerator; FirePass SSL VPN appliance; Traffix diameter signaling products; and ARX file virtualization products.

Unless this is a last-but-not-least ordering, it looks like management is not leading with the ARX products. But let’s look for more evidence of management’s priorities.

Combing through the F5 newsroom, for instance, we find that the last press release on ARX is almost 2 years old. Titled “F5’s New ARX Platforms Help Organizations Reap the Benefits of File Virtualization” it is surprising that later press releases don’t call out other success stories.

The most recent ARX white papers, “Reducing Storage Costs with F5 ARX” and “Enabling Flexibility with Intelligent File Virtualization” are both dated 2011. The ARX data sheet is from 2013 though.

The StorageMojo take
It’s clear that F5 has backed away from the ARX technology – which they acquired with Acopia in 2007 – in favor of the Application Delivery Controller market. But does that mean that the Acopia/F5 ARX didn’t succeed?

Clearly, ARX succeeded for a while: F5 bought them after all. And the F5 PR archives have several success stories from 2009.

But within the current F5 context – where they have several high-growth segments – ARX is getting little investment. At another company, perhaps, ARX would be a success, but at F5 it clearly is not.

If I were a customer I would certainly look at ARX if I wanted to virtualize disparate NAS filers. But I’d be sure to have some contingency plans in place if F5 decided to end-of-life the product in the next 2 years.

Courteous comments welcome, of course. Fun fact: F5′s name was inspired by one of my favorite movies – and an AFI top 100 selection – Twister, that popularized the Fujita scale (now the Enhanced Fujita scale) for tornado intensity.

The traditional model of NAS filers is handy if you only have a few. But once you get to 8 or 10 NAs filers your life gets complicated.

Your oldest data is on the oldest filer and your active data is on the newest. If that new filer bottlenecks your entire system slows down.

Hence the old saw “you’ll love your first filer and hate your tenth.” System administrators will load balance by moving data back and forth, an inherently wasteful and error-prone exercise.

A history of failure
A number of start ups – such as Z-force and Zambeel – have attempted a fix. The general idea is a switch that virtualizes the backend filers to create a single pool.

While the concept sounds good, results have been dismal. No storage system whose primary function was to front-end existing NAS boxes has succeeded.

Once more unto the breach
Now another entrant enters the fray. Avere Systems has raised $50 million and is on v3 of their tin-wrapped software.

At NAB 2013 they announced the FXT 3800 Edge Filer. The 3800 tiers across RAM, SSD, SAS, backend NAS and cloud across one namespace.

They’re understandably proud of their new SPECsfs2008 NFS result with a FXT 3800 32 Node Cluster that reached 1,592,334 Ops/Sec. That beats NetApp, Isilon, Hitachi/BlueArc and everybody else, except Huawei’s OceanStor 8500, which used 24 file systems and more than twice the number of SSDs get over 3 million Ops/Sec.

Oh, and they included a transcontinental latency in the network. As you might see using a cloud provider like Amazon Web Services, which was showing an Avere proto version in their NAB booth.

The hard question
After the briefing by Avere I asked Ron Bianchini, the CEO and cofounder, why Avere would escape the fate of their erstwhile predecessors.

I boiled his answer down to 4 points:

• Avere’s appliance is a read and write cache, so hot data I/O is handled directly and not routed to the backend filers. Typically, he says, only 1 out of 50 I/Os leave Avere for backend NAS, and for some workloads it is as little as 1 out of 200.
• Their file system is the client of the backend filers, so they know exactly where the data is at all times. Furthermore, they’ve certified vendors like NetApp, so they handle the inevitable corner cases.
• The system moves data across 4 tiers – DRAM, SSD, SAS, SATA and the backend filers so it is capable of extremely high performance, unlike products that relied upon backend performance.
• They also manage blocks within files, so a change in a file doesn’t require rewriting the entire file, a popular feature in large file applications.

The StorageMojo take
Rip and replace has never been popular. With today’s data volumes it is ever more unwieldy.

Avere’s performance and cost-effectiveness make it more than a simple pooling of NAS capacity: by reducing the load on current filers it extends their economic life while eliminating hot-spots and bottlenecks. You keep what you’ve got and make it faster and easier to manage.

Since most disk-based systems are way over-configured on capacity, this also means reduced CapEx and OpEx as fewer new filers are bought and less floor space, power and maintenance is needed. Given their scale-out architecture – minimum config is 3 nodes – you can add performance without adding more filers.

Bottom line: Avere, using 21st century technology, has built a new way to utilize existing resources while improving performance and reducing costs. That’s something no other NAS front-end ever managed.

They’ll do well.

Courteous comments welcome, of course. Any Avere users want to comment on their experience? I haven’t done any work for Avere, but that could change.

Choice is a great thing, unless there’s too much of it. And choice is what we have a lot of in today’s data storage market.

A longtime StorageMojo reader has an interesting problem: architect a 3PB data storage facility. Can you help?

Here’s what he wrote to StorageMojo. His email has been slightly edited for clarity and length.

One of my current problems is to design one of the nodes for a large research data storage facility. I’ve had to do this stuff in varying degrees, varying modalities and varying tech in times gone by.

I’ve been given a number and “capacity” to look into – somewhere near or around 3PB to begin with. We won’t even go down the path of discussing workloads or disk technology fit for purpose at this stage, but, something has struck me as interesting.

There is this clear divergence in disk technologies at the moment and I’m finding it hard to resolve what is the “right” one of the task.

Currently, I see:

• Heavy-end storage virtualisation frames [VSP, Symmetrix et al]
• Big grid-ish things [IBM XIV etc]
• Weird “stacked” commodity LSI Silicon [NetApp E5400/5500, SGI IS5500/IS5600, Dell MD3660F etc – all the same silicon I think?!]
• Quasi virtualisation arrays with modular form factors (Hitachi’s HUS-VM?)
• High performance dense trays in modular form factors [DDN's SFA-12K Exa and Grid scaler tech?]
• Bog-standard performance dense trays in modular form factors [Hitachi HUS, EMC VNX, HP EVA, Dell compellent etc etc]
• That wild crazy pure flash/RAM/SSD/NAND world that guys like Violin inhabit.

Currently I’m trying to rationalise what I should be using for a storage platform that needs to scale big, but do it in a sensible economic standpoint, with density, performance of interconnect and throughput with gross mixed workloads being all big factors.

Some folks suggest to me that I should be happy enough with the LSI horizontally stacked 60-drive trays, but I am not sure the technology is tracking too well in terms of performance or density (Hitachi, DDN and maybe some others can now do 84 drives in as little as 4-RU!).

I guess my question to you is – where do you see that dense high performance market heading? I know the guys at the LLNL over your way were crowing about the NetApp E5400 LSI stuff where they managed their “1TB/sec” file system (I think it was Lustre based?), but I have to wonder if that could have been more efficiently carried out using a DDN GridScaler/SFA-12K-E etc.

The StorageMojo take
Two issues here: is the segmentation our correspondent offers realistic and helpful? And what are the core architectural issues he needs to think about?

For the first issue an object store or a highly parallel NFS – like Panasas – seems to be indicated.

Given that this is a general purpose high-performance system, the critical problem seems to be how the system – however architected – handles file creation/update/deletion metadata. String enough disks together – 1,000 to 2,000 – and you can get a reasonable # of IOPS and, if you need more, put some SSDs in front.

There are a number of scale-out storage systems that will credibly and economically grow to 3PB. Metadata is often the bottleneck, as Isilon buyers have found when creating many small files.

A maximum performance spec – including file creation etc. rates – will probably help eliminate likely laggards, while a budget $ per usable TB/PB will eliminate the uneconomic products.

Vendors are welcome to offer their perspectives. Please just identify your company so we know where you’re coming from.

Practitioners who’ve done this, or something similar, are encouraged to share their hard-earned wisdom. 3PB is non-trivial today.

Courteous comments welcome, of course. I’m going to start offering almost-free consulting for end-users. Stay tuned!

This came in this morning’s email from a reader I’ll call Perplexed. How would you advise Perplexed?

I’m looking at a new iSCSI storage system for two sites with ~ 20 servers each – 10TB each should do it. Picture two fairly usual manufacturing/mining sites, 200-500 users, email, file/Print, finance and production database services, MS Domain etc.

Looking at IOPS – we would be serviced by 24 x 2.5″ SAS 10K disks in a RAID6 array.So – the thought occurs – that SSD would easily match that performance with far less devices.

Say 15 VM’s and 5 Servers per location. Requirement for about 5TB of data with limited growth – lets say 10TB storage and under 1000 IOPS.

Throughput is not an issue except for backup and DR. If we can saturate 2 or 3 Gigabit Ethernet links that is adequate.

This would be served comfortably by 24 x 10K 2.5″ RAID6 arrays at each location. Two of them for redundancy.

But – a single Intel 710 SSD could meet that IOPS rate and probably throughput as well. One SSD disk replacing an entire 24 disk array!

I would then ask, why have RAID at all? RAID is based on spindles being the smallest block for failure. With SSD, that block could be much smaller. The controller is already doing some ECC for wear management with overprovisioning.

Is there a new paradigm the granularity is no longer a “spindle”? Should we simply over-provision by 50%? SSD generally comes with provisioned spare capacity – starting to sound like redundancy and error correction is built into the controllers to some degree already.

What would be ideal is a 1RU box full of 10TB solid state storage with 10G iSCSI – no separate disks.

Has SSD let us start to move beyond RAID? With the death of spindles and the huge IOPS available, is the entire R1, R5, R6, R10 debate finished? Does RAID have it’s place in a box full of chips, and if yes, does it look the same as what we know?

Has the world started to change in storage, or is SSD still just non-moving spindles?

10TB, 1000IOPS, 10G iSCSI – how would you buy it?

Readers, what say you?
What suggestions do you have for Perplexed? The IOPS are low and he doesn’t suggest heavy bandwidth requirements either. But he does seem very interested in reliability.

Update: Vendors are welcome to comment. I only ask that you identify yourself as such. End update.

The StorageMojo take
Aside from cost – I’d expect a minimum of $4-$5 per gigabyte or ≈$100k+ for the storage – the low IOPS requirement means SSD could be overkill. Perhaps a hybrid SSD/disk solution? SSDs can and do fail, so relying on a single SSD is as dangerous as relying on a single HDD.

A number of companies might be appropriate, including Nexsan, Nimble, TwinStrata, Nexenta, Nutanix, Tintri, Violin, Pure, Nimbus, Tegile and Avere among others. Some have features, such as WAN replication or cloud backup, that might prove useful. Others have VM support, but not with iSCSI.

Performance isn’t likely to be an issue with any of these vendors, so I’d focus on availability, management, support and then look at cost.

Courteous comments welcome, of course. I’ve done work for some of the companies mentioned.

The brains behind the ZFS filesystem – including Jeff Bonwick and Bill Moore – have been hard at work for several years at start up DSSD. What are they doing with Andy Bechtolsheim’s money?

Bill’s recent Usenix bio says that “. . . DSSD, Inc., [is] a stealth startup focused on creating the fastest and most reliable storage possible.”

That puts them in good company.

They also have a trademark on the term “Cubic RAID” which describes “Computer software for use in electronic storage of data.” That narrows it right down.

Patent search
The good news is that while they are still in stealth mode their 6 patents are out there for all to see. What looks interesting?

The “Method And System For Multidimensional RAID” uses something called “RAID grids” to enable reliable and robust data storage and access.

The “Storage System With Self Describing Data” adds to the multidimensional RAID idea and adds object storage-like constructs into the storage system.

“Storage System With Incremental Multidimensional RAID” expands on this idea. By distributing the data and parity across the grid the design intends to enable data recovery despite multiple instances of data corruption or media failure.

The grid, of course, is three-dimensional, turning it into something they call a RAID cube. Given the focus on data recoverability perhaps the D in DSSD stands for durable.

The StorageMojo take
Looking at the technology it seems that the folks at DSSD are focused on a very particular set of requirements.

Self-describing data sounds like object storage. The extreme durability and high performance suggests a system optimized for transaction processing.

The use of solid-state storage implied in the name also suggests a focus on high performance. But SSD costs mean a use case very different from the video storage that objects are commonly used for now.

So what are they building? They are taking a radically different approach to the problem of high-performance transaction processing storage. The use of flash is a given in TP, and the extra durability, scalability and guaranteed read latency would be very attractive in large TP applications.

The most surprising piece is the object storage-like characteristics suggested by the patents. But handling billions of small objects at high-speed in a flat namespace would make it easy to distribute object indexes among hundreds of users, reducing file system I/O latency. The 3D RAID could eliminate the encoding overhead inherent in advanced erasure codes while providing similar robustness, enabling way-beyond-RAID6 availability.

Going after the last stronghold of big iron RAID arrays may seem bold. But as Willy Sutton said of banks, that’s where the money is.

The key will be whether their technology enables them to come to market with a clear economic advantage, be it cost or performance. With some of the most brilliant and original minds in storage and Silicon Valley working the problem, whatever they come up with is bound to be innovative.

Courteous comments welcome, of course. A launch this year feels about right.

StorageMojo is focussed on new storage technologies, products, companies and markets. And where do new technologies come from? From people researching at the limits of the known.

That’s why StorageMojo attends the Usenix File And Storage Technology (FAST) conference every year. Top academics – many with corporate ties – and grad students present the latest research.

Many papers are submitted and reviewed before some are chosen for presentation at the conference over two and a half days. Here are StorageMojo’s favorites from FAST 13, in no particular order:

SD Codes: Erasure Codes Designed for How Storage Systems Really Fail by James S. Plank, U of Tennessee, and Mario Blaum and James L. Hafner of IBM Research. RAID systems are vulnerable to a disk failures and unrecoverable read errors, but RAID 6 is overkill for UREs. The paper investigates lighter-weight erasure codes – disk plus sector, instead of 2 disks – to reclaim capacity for user data.

The StorageMojo take: high update costs make this most attractive for active archives, not primary storage. The capacity savings could extend the economic life of current RAID strategies vs newer erasure codes.

Gecko: Contention-Oblivious Disk Arrays for Cloud Storage by Ji-Yong Shin and Hakim Weatherspoon of Cornell, Mahesh Balakrishnan of Microsoft Research and Tudor Marian of Google. The limited I/O performance of disks makes contention a persistant problem on shared systems. The authors propose a novel log structured disk/SSD configuration and show that it virtually eliminates contention between writes, reads and garbage collection.

The StorageMojo take: SSDs help with contention, but they aren’t affordable for large-scale deployments. Gecko offers a way to leverage SSDs for log-structured block storage that significantly improves performance at a reasonable hardware cost.

Write Policies for Host-side Flash Caches by Leonardo Marmol, Raju Rangaswami and Ming Zhao of Florida International U., Swaminathan Sundararaman and Nisha Talagala of Fusion-io and Ricardo Koller of FIU and VMware. Write-through caching is safe but expensive. NAND’s non-volatile nature enables novel write-back cache strategies that preserve data integrity while improving performance. Thanks to large DRAM caches, read-only flash caches aren’t the performance booster they would have been even 5 years ago.

The StorageMojo take: Using flash only for reads mean ignoring half – or more – of the I/O problem. This needs to be fixed and this paper points the way.

Understanding the Robustness of SSDs under Power Fault by Mai Zheng and Feng Qin of Ohio State and Joseph Tucek and Mark Lillibridge of HP Labs. The authors tested 15 SSDs from 5 vendors by injecting power faults. 13 of the 15 lost data that should have been written and 2 of the 13 suffered massive corruption.

The StorageMojo take: We may be trusting SSDs more than they deserve. This research points out problems with still immature SSD technology.

A Study of Linux File System Evolution by Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau and Shan Lu of the University of Wisconsin. The authors analyzed 8 years of Linux file system patches – 5079 of them – and discovered, for instance, that

. . . semantic bugs, which require an understanding of file-system semantics to find or fix, are the dominant bug category (over 50% of all bugs). These types of bugs are vexing, as most of them are hard to detect via generic bug detection tools; more complex model checking or formal specification may be needed.

The StorageMojo take: Anyone building or maintaining a file system should read this paper to get a handle on how and why file systems fail. Tool builders will find some likely projects as well.

Courteous comments welcome, of course. There were some great posters and WIP presentations as well that I hope to write about soon.

I was asked at the SNIA nonvolatile memory conference why I did not include virtualization as a major driver for the use of nonvolatile memory. Flash helps with the multiple virtual machine I/O blender problem.

But we also had that problem when we were running multiple applications on a single machine. Yes, performance requirements were lower, but we managed.

I consider virtualization a feature and not a market. Why is virtualization a feature rather than a long-term product as many keen observers believe?

VM history
In the 1970s we had the virtual memory operating system wars. A number of companies, including DEC, Prime and IBM, developed virtual memory operating systems.

The key enabler of the virtual memory operating systems was the advent of 32-bit processors with true 32-bit address spaces. This is back when people programmed on minicomputers with a quarter of a megabyte to a maximum 4 MB of physical memory.

The VAX/VMS (Virtual Address eXtension/Virtual Memory System) OS, in contrast, offered a massive 4GB address space, 2 of which were reserved for the system and 2 for users.

Virtual memory operating systems enabled important changes for the industry. Software developers could focus on making their software as functional as possible without worrying about the underlying memory architecture.

The virtual memory wars continued into the PC era, when MS-DOS was limited to a 640KB address space and Quarterdeck systems came in with QEMM. But eventually Intel and Microsoft got their act together and solved the problem of virtual memory for the masses.

No one pays extra for a virtual memory operating system today. It is expected functionality that most don’t even know exists.

Like 32-bit processors the advent of microprocessors powerful enough to run multiple applications led to the virtual machine opportunity. If Microsoft had not been intent on selling as many server licenses as possible, they could have improved their multitasking so that the problem of server sprawl might never have occurred.

Be that as it may, there is nothing in today’s virtualization technology that could not and should not be incorporated into server operating systems. In 20 years new CompSci grads won’t know that virtual machines weren’t always built into the OS.

Feature vs product
Now just because something is at bottom a feature rather than a product doesn’t mean that you can’t make gobs of money before it becomes common. VMware is one example.

Data deduplication, for example, is clearly a feature. But the founders of Data Domain were able to make a lot of money by exploiting that feature with a high-quality application-focused implementation and being first to market.

Now deduplication is being built into new products. While debates over implementation details will continue among engineers, in a few years most users will see deduplication as a checkbox item.

The transient and the permanent
How do we distinguish between a feature and a product? It is the difference between the transient and the permanent.

Transient problems can be resolved. Permanent problems can only be managed.

Processor virtual memory management is a problem that has been solved for the majority of the world’s computers. Data de-duplication can be added over time to storage systems as computing power increases and the cost of bandwidth and random I/Os – thanks NVM! – drops.

But some problems can only be managed, not solved. The issues of scale, aggregation and metadata – among others – will always be with us.

Like gas-rich regions of galactic star formation, these manageable-but-not-solvable issues will continue to be areas rich in startup formation.

The StorageMojo take
Applying this theory to current markets yields some predictions:

• VMware, despite its feature-rich ecosystem and early lead, will lose to vendors, such as Microsoft and Red Hat, who can incorporate the most important virtualization features into their OS. VMware has no OS to fall back on and thus has no long-term future.
• Data Domain is a wasting asset. As others add dedup to their products, DD’s differentiation will decline along with its market value.
• Scale-out storage, like Isilon, will remain a lively market segment as the economics of disk, NVM, software, aggregation and metadata keep changing the calculus of efficient and durable storage.
• The improving economics of erasure coding will enable more efficient approaches to backup and archiving – as long as Moore’s Law continues to hold.
• System management is a permanent problem. When we get autonomic management at one level, the problem just kicks up a level with increasing scale.

Just as no one remembers the critical register and segment management skills required for 16-bit minicomputers, in a decade or so all the painfully acquired knowledge required to manage VMs will lose value as it gets built into the OS infrastructure. But there will always be new worlds to conquer.

Courteous comments welcome, of course. How would you define a feature vs a product?

Hewlett-Packard made a major announcement in December: their grid-based StoreServ 7000 based on 3PAR software. 3PAR has been selling enterprise-class storage for almost a decade and HP is building on that legacy with a radical take on enterprise storage.

HP calls it converged storage. The concept is simple: enterprise-class scale-out storage that is configurable to meet a wide range of application requirements. Flexible commodity hardware to mix-and-match to meet requirements, while configuring SSDs, hard drives or tape makes it easy to meet a wide range of price-performance ratios.

It is the enterprise analog of Google’s and Amazon’s cloud infrastructures whose scale, flexibility and cost are up-ending traditional IT. I’ve been looking for this from a major vendor since I wrote about the Google File System 7 years ago.

GFS breaks that model and shows us what can be done when the entire storage paradigm is rethought. Build the availability around the devices, not in them, treat the storage infrastructure as a single system, not a collection of parts, extend the file system paradigm to include much of what we now consider storage management, including virtualization, continuous data protection, load balancing and capacity management.

GFS is not the future. But it shows us what the future can be.

A choice, not an echo
The prevailing vendor model is the stovepipe: for every function within the enterprise there is a specialized product or set of products. These stovepipes are then glued together with a management layer that is supposed to manage everything from a single pane of glass – but never does.

Competition is stovepipe vs. stovepipe. The big iron arrays from EMC, Hewlett-Packard and Hitachi all compete, in theory, on availability, performance, support and cost.

The reality? Account control is often the deciding factor. Competition has stagnated over the last 10 years.

EMC is the most successful stovepipe peddler. Their technology publishing model – buy young companies that have crossed the chasm and sell the hell out of them – reduces their engineering risk at the cost of grievous incompatibility.

Storage as a layer, not a pipe
But as long as EMC can afford to buy the best – and they can – competitors are always at a disadvantage. But HP’s converged storage model changes that – just as Amazon has forever changed enterprise IT.

HP says they’ve already won 1200 customers for their converged storage line.

The StorageMojo take
No one is going to out-EMC EMC. They have the largest collection of best-of-breed point products in the industry.

Dave Donatelli, formerly of EMC, knows this well. Thus HP’s flanking move with converged storage.

Assuming HP can deliver a substantial fraction of Amazon’s benefits inside the data center, they will do very well. But the real magic will come when customers start interconnecting islands of converged storage to create a storage layer – like the network layer – in their infrastructure.

There are substantial customer political obstacles, but Amazon’s success has changed how CFOs look at IT. Smart CIOs are listening and responding.

Congratulations to HP and the 3PAR team for coming up with the only credible challenge to EMC in the last 15 years. I wish them every success.

Courteous comments welcome, of course. The friendly folks at HP flew me to Europe last year to get the announcement first hand. But I’d like a technical briefing from a 3PAR architect.

There are years where new ideas and concepts explode. And there are years of consolidation. 2013 will be the latter.

The storage industry has a lot to digest. Here are some of the issues.

ReRAM
Would-be vendors of the enterprise NAND flash replacement technology, Resistance RAM or ReRAM, will be hunting for launch customers. While their new product won’t be as cheap as NAND flash, it doesn’t need to be because enterprise customers will pay for greater speed, reliability and durability. The medium isn’t a big cost driver.

If ReRAM launches this year, who will lead the charge? I’d bet on either Intel, Violin Memory, Kaminario or Samsung. Intel and Samsung would incorporate ReRAM into their SSDs, while Kaminario and Violin would build it into their high-end solid-state arrays.

Disks
Disk drive vendors are trying to figure out how to survive the deflation of their largest market due to tablets and smart phones. They’re counting on hybrid drives that blunt the performance advantages of pure SSDs. But they’ll need a higher level of commitment to architecting the right products than I’ve seen so far.

We’ll also see if Hitachi’s helium drives can make it in the marketplace. Hitachi/WD must have a major data center launch customer – Google, Facebook or Amazon – since no major server vendor would offer them without a 2nd source.

Next-gen systems
Architects from next-gen storage companies such as Avere, Nimble Storage, Amplidata, Nimbus Data, Fusion-io, Starboard, Nutanix, Violin Memory, Tegile and many others will be tuning their systems and software – mostly software these days – based on real customer workloads instead of guesstimates.

HP’s converged storage is a special case. Will Donatelli be able to motivate HP’s enterprise sales force to move the product?

Cloud
Cloud storage and computing will continue to bend the arc of enterprise new technology adoption. CFOs will become tougher in their questioning of CIOs given the obvious CapEx advantages of cloud infrastructure.

Big Data
Multiple slogs for Big Data. Privacy concerns and regulation. The plethora of infrastructure options slows decision making and PO writing. Dawning awareness that the output is only as good as the questions you ask – technology is secondary.

But we’re only at the beginning of the Big Data revolution.

The StorageMojo take
The last decade has seen an explosion of new storage architectures and technologies. That’s as it should be, since storage is the hardest part of information infrastructure – and the most critical.

But while innovation won’t halt this year, it’s time for the market to sort things out. CIOs who value their jobs will be looking beyond their usual suppliers to better compete with cloud IaaS vendors.

It will be an interesting year.

Courteous comments welcome, of course. I’ve done work for several of the mentioned vendors, but for most I haven’t.

EMC’s Pravda, Chuck Hollis, chats in an interview about XtremIO’s problems. This quote from a Storage Newsletter interview may have some transcription errors, since SN is a French publication, but let’s not split hairs. Read it all to learn more.

When will you have a dedicated all-flash array?
We announced our XtremIO acquisition last year. We demonstrated it at VMworld and EMC World. It’s an all-flash design. Made of de-dupe, scale-out architecture, all flash. It fits very precise parts of the market.

Do you sell it?
No, right now we’re in what’s called ‘early access’. Test with the customers, make sure it works right. You’ll probably see it in the first half of 2013 as a general product. It’s got a very specific profile. Random IO/s, extreme performance and tiering do not work. We have a data set where everything has to run fast.

Signal intelligence
Mr. Hollis is paid well to present the EMC party line, and he’s good at it. There are reasons for everything he says.

1. Pre-announcing the new flash array is intended to freeze the market, especially Violin Memory, Kaminario, Nimbus Data and Pure Storage among others.
2. “Precise parts of the market” means that EMC doesn’t want customers to see it as a Symm replacement – which given their likely Xtreme pricing and need to keep VMAX billions rolling in – is simple self-preservation.
3. It won’t ship until the 2nd half of 2013 at the earliest and may slip further. But an H1/2013 announcement is likely.

Unpacking the ship date
EMC is very good at keeping secrets when they want to. In this case they don’t want to.

Why?

By telling us that they’re intending to enter the market for pure flash arrays they are hoping to slow the competitors kicking VMAX to the curb. EMC sales hates losing deals, but since they don’t have a competitive flash array they are stymied.

If the product were shippable in Q1/2013, EMC would deploy evangelists, including Mr. Hollis, to give NDA presentations, offer test lab visits, early delivery units and so on. But the “early access” program – “test with the customers, make sure it works right” – is a beta test. Normally that would come late in the development process when announce and ship dates are reasonably firm.

The dates aren’t firm if “probably” H1/2013 is the best EMC can offer. When XtremIO was bought, EMC predicted a Q1/2013 product launch.

The StorageMojo take
XtremIO hadn’t shipped a beta when EMC bought them. It’s clear that XtremIO’s “great technology” is further from a “great product” than they’d hoped.

Thus the damage control effort.

To limit embarrassment EMC may announce at EMC world in May with “planned availability” for Q3. But given the dynamics – flash arrays have a lot of interesting wrinkles – Q4 is my forecast for shipments of a de-featured product.

What went wrong? Under-estimating market demand for all-flash arrays plus over-estimating VMAX with flash cache appeal, resulting in a reluctance to pay $2B+ for a less-mature-than-they’d-like, but installed product company like Kaminario or Violin.

The year’s delay will be costly for EMC as flash array competitors become firmly entrenched in major accounts. Fixing that will take more than spin.

Courteous comments welcome, of course. I’ve done work for Kaminario and Violin, and I’m still a little peeved over EMC threatening StorageMojo.

I’ve had the pleasure of moderating a half-dozen panel discussions on Big Data and object storage in the last few months. It’s been a learning experience.

Big Data has always been as big as we could afford, be it block, file or object. Google’s MapReduce, running on massive commodity-based scale-out GFS object storage, inspired the current Big Data craze.

If storage were the only problem we could stop with a discussion of Amplidata, the object storage company that sponsored StorageMojo’s participation (check out the StorageMojo object storage video to get a quick overview). But there’s much more to it than that.

Defining Big Data
Ever notice that the number of disk drives in most infrastructures stays fairly constant? The growth in drive capacity parallels the growth in data, so disk drive populations should be stable.

Thus a definition of Big Data based on the number of drives is simple and flexible over time. Defining the lower bound of Big Data as an application that uses 100 large capacity SATA drives means that with 4TB drives you aren’t doing Big Data unless you have 400TB of raw storage.

Smaller data sets can be vital to business or research; this isn’t a “mine is bigger than yours” competition. But as a rule of thumb you can probably live with filers if you’re certain your application won’t need more than 100 drives ever and, if you will, you should consider object storage.

Technology
There is no doubt that advanced erasure codes – variously referred to as rateless or fountain codes – are technically and financially superior to the older and much more common Reed-Solomon erasure codes for Big Data.

They are lower overhead, offer higher redundancy and better security than RS codes. Want to survive 4 failures with ≈50% overhead? Rateless codes are the answer.

Moving Big Data
Shipping boxes of big SATA drives overnight is still the high bandwidth solution – assuming the drives survive – but the physical hassle is prohibitive for most operations. Assuming you need to move gigabytes regularly, you need 2 things: a fast transport; and a storage system that can manage sustained write performance – which not all can.

The friendly folks at Aspera software have figured out how to achieve line speed data rates across long distances, a trick TCP has never learned. They can feed the data directly to an object storage system.

Integration
Object storage isn’t a drop-in replacement for NAS: the common REST interface is much simpler than NFS 4.1 or SMB 3.0. Many new apps are now written to use REST, but what about legacy apps that expect NAS?

At least one vendor,Panzura offers a NAS front end to object storage. Their Global Cloud Storage System serves up NAS protocols while using object storage on the back end.

But they only support a few back-end options, so you better like their choices. There are options beside Panzura though.

The business problem
The technology is there, but as I asked at one of the panels “if this stuff is so freakin’ great, why isn’t everyone using it?” It isn’t only enterprise IT inertia.

There’s another problem: business units and enterprises aren’t used to seeing storage as horizontal infrastructure, where scale-out storage wins. Even major adopters of object storage typically use a single app to justify an investment.

Networks are a horizontal infrastructure. We buy them anyway. It is a solvable problem that I’ll be writing more about more later.

The StorageMojo take
A few years ago there was debate about public vs private clouds. I continue to think there are business cases to be made for both.

But they aren’t the same cases.

As the commoditization of scale-out architectures grows and the cost of public cloud storage becomes an issue, the advantage that public providers offer today will decline. AWS is driving app development to the S3 API, and those apps will migrate easily to private clouds.

The growth of private clouds won’t crater AWS. Yet AWS will enable private clouds.

Courteous comments welcome, of course. How do you define Big Data?