In response to the post on Avere’s architecture for fronting backend NAS filers – where StorageMojo said that no front-end to NAS boxes has succeeded – alert reader Jacob Marley asked “What about F5′s ARX to stitch/balance storage across multiple filers?”

Good question! What can we deduce from publicly available sources?

The F5 ARX product line is billed as an “intelligent file virtualization solution” that
“. . .preserves the logical access to files regardless of their current location on storage.” Like earlier file switches

The ARX device does not introduce a new file system; rather it acts as a proxy to the file systems that are already there.

ARX is not a storage device itself but a load-balancer for NAS filers. Then, per Mr Marley’s question, is ARX not a success?

Competitive analysis
First up, let’s take a look at the latest quarterly 10-Q report, courtesy of the SEC’s EDGAR database.

In “Management’s Discussion and Analysis of Financial Condition and Results of Operations” they describe their product revenues as

The majority of our revenues are derived from sales of our application delivery networking (ADN) products including our high end VIPRION chassis and related software modules; BIG-IP Local Traffic Manager, BIG-IP Global Traffic Manager, BIG-IP Link Controller, BIG-IP Application Security Manager, BIG-IP Edge Gateway, BIG-IP WAN Optimization module, BIG-IP Access Policy Manager, WebAccelerator; FirePass SSL VPN appliance; Traffix diameter signaling products; and ARX file virtualization products.

Unless this is a last-but-not-least ordering, it looks like management is not leading with the ARX products. But let’s look for more evidence of management’s priorities.

Combing through the F5 newsroom, for instance, we find that the last press release on ARX is almost 2 years old. Titled “F5’s New ARX Platforms Help Organizations Reap the Benefits of File Virtualization” it is surprising that later press releases don’t call out other success stories.

The most recent ARX white papers, “Reducing Storage Costs with F5 ARX” and “Enabling Flexibility with Intelligent File Virtualization” are both dated 2011. The ARX data sheet is from 2013 though.

The StorageMojo take
It’s clear that F5 has backed away from the ARX technology – which they acquired with Acopia in 2007 – in favor of the Application Delivery Controller market. But does that mean that the Acopia/F5 ARX didn’t succeed?

Clearly, ARX succeeded for a while: F5 bought them after all. And the F5 PR archives have several success stories from 2009.

But within the current F5 context – where they have several high-growth segments – ARX is getting little investment. At another company, perhaps, ARX would be a success, but at F5 it clearly is not.

If I were a customer I would certainly look at ARX if I wanted to virtualize disparate NAS filers. But I’d be sure to have some contingency plans in place if F5 decided to end-of-life the product in the next 2 years.

Courteous comments welcome, of course. Fun fact: F5′s name was inspired by one of my favorite movies – and an AFI top 100 selection – Twister, that popularized the Fujita scale (now the Enhanced Fujita scale) for tornado intensity.

The traditional model of NAS filers is handy if you only have a few. But once you get to 8 or 10 NAs filers your life gets complicated.

Your oldest data is on the oldest filer and your active data is on the newest. If that new filer bottlenecks your entire system slows down.

Hence the old saw “you’ll love your first filer and hate your tenth.” System administrators will load balance by moving data back and forth, an inherently wasteful and error-prone exercise.

A history of failure
A number of start ups – such as Z-force and Zambeel – have attempted a fix. The general idea is a switch that virtualizes the backend filers to create a single pool.

While the concept sounds good, results have been dismal. No storage system whose primary function was to front-end existing NAS boxes has succeeded.

Once more unto the breach
Now another entrant enters the fray. Avere Systems has raised $50 million and is on v3 of their tin-wrapped software.

At NAB 2013 they announced the FXT 3800 Edge Filer. The 3800 tiers across RAM, SSD, SAS, backend NAS and cloud across one namespace.

They’re understandably proud of their new SPECsfs2008 NFS result with a FXT 3800 32 Node Cluster that reached 1,592,334 Ops/Sec. That beats NetApp, Isilon, Hitachi/BlueArc and everybody else, except Huawei’s OceanStor 8500, which used 24 file systems and more than twice the number of SSDs get over 3 million Ops/Sec.

Oh, and they included a transcontinental latency in the network. As you might see using a cloud provider like Amazon Web Services, which was showing an Avere proto version in their NAB booth.

The hard question
After the briefing by Avere I asked Ron Bianchini, the CEO and cofounder, why Avere would escape the fate of their erstwhile predecessors.

I boiled his answer down to 4 points:

• Avere’s appliance is a read and write cache, so hot data I/O is handled directly and not routed to the backend filers. Typically, he says, only 1 out of 50 I/Os leave Avere for backend NAS, and for some workloads it is as little as 1 out of 200.
• Their file system is the client of the backend filers, so they know exactly where the data is at all times. Furthermore, they’ve certified vendors like NetApp, so they handle the inevitable corner cases.
• The system moves data across 4 tiers – DRAM, SSD, SAS, SATA and the backend filers so it is capable of extremely high performance, unlike products that relied upon backend performance.
• They also manage blocks within files, so a change in a file doesn’t require rewriting the entire file, a popular feature in large file applications.

The StorageMojo take
Rip and replace has never been popular. With today’s data volumes it is ever more unwieldy.

Avere’s performance and cost-effectiveness make it more than a simple pooling of NAS capacity: by reducing the load on current filers it extends their economic life while eliminating hot-spots and bottlenecks. You keep what you’ve got and make it faster and easier to manage.

Since most disk-based systems are way over-configured on capacity, this also means reduced CapEx and OpEx as fewer new filers are bought and less floor space, power and maintenance is needed. Given their scale-out architecture – minimum config is 3 nodes – you can add performance without adding more filers.

Bottom line: Avere, using 21st century technology, has built a new way to utilize existing resources while improving performance and reducing costs. That’s something no other NAS front-end ever managed.

They’ll do well.

Courteous comments welcome, of course. Any Avere users want to comment on their experience? I haven’t done any work for Avere, but that could change.

Choice is a great thing, unless there’s too much of it. And choice is what we have a lot of in today’s data storage market.

A longtime StorageMojo reader has an interesting problem: architect a 3PB data storage facility. Can you help?

Here’s what he wrote to StorageMojo. His email has been slightly edited for clarity and length.

One of my current problems is to design one of the nodes for a large research data storage facility. I’ve had to do this stuff in varying degrees, varying modalities and varying tech in times gone by.

I’ve been given a number and “capacity” to look into – somewhere near or around 3PB to begin with. We won’t even go down the path of discussing workloads or disk technology fit for purpose at this stage, but, something has struck me as interesting.

There is this clear divergence in disk technologies at the moment and I’m finding it hard to resolve what is the “right” one of the task.

Currently, I see:

• Heavy-end storage virtualisation frames [VSP, Symmetrix et al]
• Big grid-ish things [IBM XIV etc]
• Weird “stacked” commodity LSI Silicon [NetApp E5400/5500, SGI IS5500/IS5600, Dell MD3660F etc – all the same silicon I think?!]
• Quasi virtualisation arrays with modular form factors (Hitachi’s HUS-VM?)
• High performance dense trays in modular form factors [DDN's SFA-12K Exa and Grid scaler tech?]
• Bog-standard performance dense trays in modular form factors [Hitachi HUS, EMC VNX, HP EVA, Dell compellent etc etc]
• That wild crazy pure flash/RAM/SSD/NAND world that guys like Violin inhabit.

Currently I’m trying to rationalise what I should be using for a storage platform that needs to scale big, but do it in a sensible economic standpoint, with density, performance of interconnect and throughput with gross mixed workloads being all big factors.

Some folks suggest to me that I should be happy enough with the LSI horizontally stacked 60-drive trays, but I am not sure the technology is tracking too well in terms of performance or density (Hitachi, DDN and maybe some others can now do 84 drives in as little as 4-RU!).

I guess my question to you is – where do you see that dense high performance market heading? I know the guys at the LLNL over your way were crowing about the NetApp E5400 LSI stuff where they managed their “1TB/sec” file system (I think it was Lustre based?), but I have to wonder if that could have been more efficiently carried out using a DDN GridScaler/SFA-12K-E etc.

The StorageMojo take
Two issues here: is the segmentation our correspondent offers realistic and helpful? And what are the core architectural issues he needs to think about?

For the first issue an object store or a highly parallel NFS – like Panasas – seems to be indicated.

Given that this is a general purpose high-performance system, the critical problem seems to be how the system – however architected – handles file creation/update/deletion metadata. String enough disks together – 1,000 to 2,000 – and you can get a reasonable # of IOPS and, if you need more, put some SSDs in front.

There are a number of scale-out storage systems that will credibly and economically grow to 3PB. Metadata is often the bottleneck, as Isilon buyers have found when creating many small files.

A maximum performance spec – including file creation etc. rates – will probably help eliminate likely laggards, while a budget $ per usable TB/PB will eliminate the uneconomic products.

Vendors are welcome to offer their perspectives. Please just identify your company so we know where you’re coming from.

Practitioners who’ve done this, or something similar, are encouraged to share their hard-earned wisdom. 3PB is non-trivial today.

Courteous comments welcome, of course. I’m going to start offering almost-free consulting for end-users. Stay tuned!

This came in this morning’s email from a reader I’ll call Perplexed. How would you advise Perplexed?

I’m looking at a new iSCSI storage system for two sites with ~ 20 servers each – 10TB each should do it. Picture two fairly usual manufacturing/mining sites, 200-500 users, email, file/Print, finance and production database services, MS Domain etc.

Looking at IOPS – we would be serviced by 24 x 2.5″ SAS 10K disks in a RAID6 array.So – the thought occurs – that SSD would easily match that performance with far less devices.

Say 15 VM’s and 5 Servers per location. Requirement for about 5TB of data with limited growth – lets say 10TB storage and under 1000 IOPS.

Throughput is not an issue except for backup and DR. If we can saturate 2 or 3 Gigabit Ethernet links that is adequate.

This would be served comfortably by 24 x 10K 2.5″ RAID6 arrays at each location. Two of them for redundancy.

But – a single Intel 710 SSD could meet that IOPS rate and probably throughput as well. One SSD disk replacing an entire 24 disk array!

I would then ask, why have RAID at all? RAID is based on spindles being the smallest block for failure. With SSD, that block could be much smaller. The controller is already doing some ECC for wear management with overprovisioning.

Is there a new paradigm the granularity is no longer a “spindle”? Should we simply over-provision by 50%? SSD generally comes with provisioned spare capacity – starting to sound like redundancy and error correction is built into the controllers to some degree already.

What would be ideal is a 1RU box full of 10TB solid state storage with 10G iSCSI – no separate disks.

Has SSD let us start to move beyond RAID? With the death of spindles and the huge IOPS available, is the entire R1, R5, R6, R10 debate finished? Does RAID have it’s place in a box full of chips, and if yes, does it look the same as what we know?

Has the world started to change in storage, or is SSD still just non-moving spindles?

10TB, 1000IOPS, 10G iSCSI – how would you buy it?

Readers, what say you?
What suggestions do you have for Perplexed? The IOPS are low and he doesn’t suggest heavy bandwidth requirements either. But he does seem very interested in reliability.

Update: Vendors are welcome to comment. I only ask that you identify yourself as such. End update.

The StorageMojo take
Aside from cost – I’d expect a minimum of $4-$5 per gigabyte or ≈$100k+ for the storage – the low IOPS requirement means SSD could be overkill. Perhaps a hybrid SSD/disk solution? SSDs can and do fail, so relying on a single SSD is as dangerous as relying on a single HDD.

A number of companies might be appropriate, including Nexsan, Nimble, TwinStrata, Nexenta, Nutanix, Tintri, Violin, Pure, Nimbus, Tegile and Avere among others. Some have features, such as WAN replication or cloud backup, that might prove useful. Others have VM support, but not with iSCSI.

Performance isn’t likely to be an issue with any of these vendors, so I’d focus on availability, management, support and then look at cost.

Courteous comments welcome, of course. I’ve done work for some of the companies mentioned.

Our digital civilization requires data integrity and long-term preservation, and neither is assured by our current storage infrastructure. But progress continues.

Latest case in point: Amplidata. This 4 year old company, based in Belgium with a growing US footprint, brings a new level of erasure code goodness to the both problems with a cluster-based object store.

What erasure code goodness, you ask? The first – AFAIK – rateless erasure code, AKA fountain code, storage system in production use.

And that is a good thing because?
Robustness and efficiency.

Amplidata claims storage durability well beyond RAID 6: 10 9′s (spread across 16 drives with up to 4 failures) durability – though the spreads can be much larger logically and geographically. They do this by breaking the data object into segments and adding redundancy data.

The redundancy data adds about 50% to the object size – more efficient than mirroring or triple replication. The benefit is that the system can lose hundreds of segments and still reconstruct the data.

Each object is protected by checksums that can protect against more than 1000 simultaneous bit errors per object. And each write goes to at least to controllers before it is committed.

What kind of monster controller is able to perform all this magic? The minimum configuration is 3 Xeon-based commodity controller nodes with as many 10-drive Atom-based storage nodes as you need.

Amplidata is optimized for bandwidth, not IOPS. With their latest software update they now spec each controller at 750MB/sec, and you can have as many controllers as you can afford.

Sounds like Cleversafe
Cleversafe thought so too, and they’ve sued Amplidata for patent infringement. But Intel – who knows about patents and due diligence – invested after the suit.

Like NetApp’s suit against ZFS, this seems like a vanity project. Surely Cleversafe has more important things to invest in. If they don’t they’re in bigger trouble than we know.

The StorageMojo take
The need for robust, inexpensive and massive storage has been a theme of StorageMojo’s for years. Object storage is the best solution to the problem of scale, while the kind of redundancy and end-to-end checksumming that Amplidata uses seems as robust as anything on the market today.

As for inexpensive, that is in the eye of the beholder, but Amplidata tells me that their newest storage node lists for less than $0.60/GB while consuming only 60 watts. That should be attractive to people running tape silos who want faster access and better redundancy.

Courteous comments welcome, of course. I’m working with Amplidata to produce a video white paper on their technology, so stay tuned for more info on a promising company.

BlueArc and Drobo founder Geoff Barrall has a new perch: Gridstore, one of the companies I’ve been following for almost 3 years. Geoff is the new executive chairman. Formal announcement is expected this week.

Gridstore’s concept is a low-cost scale-out NAS appliance designed for office environments. Each box is a small, low-power node with a couple of TB. Stack ‘em for as much redundancy, capacity and performance you want.

Think of it as the consumerization of hyper-scale technology. Nutanix writ small.

Gridstore details
Gridstore is offering a low-cost, scale-out network file server for $500 a node. That is too cheap for the enterprise storage companies to sell directly.

Founded 5 years ago, Gridstore got a beta out in 2010, and have been shipping for well over a year. They are a Microsoft CIFS protocol file server, using Microsoft’s storage server software. Running on small, 25 watt Atom-based boxes, a 6 node configuration is the size of a bread box.


Like other scale-out NAS systems, the Gridstore NAS has no single point of failure and can survive multiple node failures without going down or losing data.

They call their redundancy scheme RAIDg. When you set up a volume you dial in how many faults you want to survive and the software handles the rest.

Today the number of faults they can handle is limited to half the number of nodes minus one. If you have a 6 node configuration it can handle the loss of 2 nodes. They expect to relax that requirement in the future.

The StorageMojo take
Haven’t spoken to Geoff about this, but Gridstore seems like a natural for him. If there’s a theme to his many endeavors, its making advanced NAS technology more accessible.

Gridstore fits the bill nicely. If there’s one complaint about Drobo, its the lack of box-level redundancy. Gridstore answers this objection, at a higher price point.

Drobo – over 200,000 units sold – has blazed a trail for bringing advanced storage technology to the masses at affordable prices. They may be the first, but as Gridstore and others demonstrate, they won’t be the last.

Courteous comments welcome, of course. Hoping to make it to CES later this week. Readers: anyone I should make a point to see?

A StorageMojo reader has a problem. Can you help?

Our mail hub (80,000+ mailboxes) is virtualized with vSphere 4.1 with Red Hat Enterprise Linux 5 x64 and Dovecot 2.0 [an open source IMAP/POP3 email server for Linux/UNIX-like systems]. We are using HP LeftHand Networks P4300 iSCSI storage in a “network RAID10 setup of RAID10 storage” for Dovecot indexes and multiple “networks RAID1 of RAID5 storage” for actual mailboxes.

This is my take: our Dovecot indexes are getting hammered with lots of small I/O requests, about 8,000 IOPS continuous during 8-working-hour days, 75% write. Indexes are fairly small (50 GB) and expected to grow to 100-150 GB, but need a lot of random I/O. We need real-time replication in storage (LeftHand is ok for us) and we think that SSD should shine in this situation. Bandwidth is not a problem (200-300 megabits of indexes traffic, but we need more IOPs).

The problem is the indexes, but our total mailbox capacity is expected to grow to 6 TB compressed using zlib compression in Dovecot.

We want to buy a storage appliance with the following requirements:

• Vsphere 4.1 & 5 certified storage, VAAI enabled (if possible)
• iSCSI (1 gbps)
• High number of IOPS (at least 12,000+, most of them writes)
• Small size (200 GB)
• Fault tolerant (RAID, battery-backed write cache, power supply, fans, multiple gigabit uplinks, synchronous replication)
• Cheap (less than $30k the full setup)

We want to buy at the beginning of 2012. Any product that fits?

The StorageMojo take
Suspect price will be the most significant limiter. But the respondent only needs index storage not the whole shooting match. He’s pretty happy with LeftHand for mailbox storage.

But if we can solve both problems for him, why not? If he should relax some constraint, feel free to suggest it.

He’ll be watching the comments, so if you have questions please ask them. I’ll be following the comments as well.

Courteous comments welcome, of course. His email was edited for clarity.

Most of us know what it is like when a relationship goes bad: the sinking feeling that this just isn’t going to work.

Can this configuration be saved?
Dear StorageMojo:

I joined a company last year that is running Oracle 10g on a NetApp NAS/SAN.

Immediately I asked why they were not using Clustering, Oracle RAC, Oracle ASM or Fiber Channel. No answer.

Fast fwd to a year later and they are asking me to deploy this to an I/O bound customer with hundreds of connections and lots of transactions to their DB over NFS.

Long story short it’s slow-w-w-w. They tried trunking multiple network connections. They tried tuning. They tried a bunch of stuff. And it’s still a dog.

How slow?

I have a screaming Dell r710 running a 7TB database attached over SAS to a set of MD3000 storage arrays. I am getting 450MBs…..and this barely suffices…..

The “new” system they showed me gets 50MBsec…the same screaming Dellr710 but connected over NFS (instead of SAS) to the NetApp NAS.

Do you have any suggestions?

Thank you for reading this nightmare.
Bob

Poor Bob! He’ll be getting grief from the client for months, maybe years to come, unless this gets fixed.

The StorageMojo take
Maybe Bob could have been better about developing a relationship with the guys configuring the systems. More questions, fewer conclusions, at first.

Suggestions to the customer for acceptance testing might be in order.

But there are 2 problems here:

1. What to do now.
2. How to keep this from happening again.

What would you suggest to Bob on either or both topics? I’ve asked him to watch the comments, so if more info would be useful, I hope he’ll provide it.

Courteous comments welcome needed. When a multi-billion dollar near-sighted telescope can get sent into orbit, it is surprising more IT projects don’t go wrong.

Talked to Nimble Storage a few months ago. The 1st time they sounded cool and now I know why.

What they do
Nimble builds a converged storage appliance out of commodity hard drives and SSDs that offers high performance – is there any other kind? – and iSCSI, backup, a form of dedup and WAN replication. The pitch is EqualLogic & Data Domain merged into a single low-cost appliance. Only better.

• iSCSI + dedup
• Capacity-optimized snapshots
• SATA + flash instead of high-rpm drives
• Can run off a remote snapshot

EL & DD sell a lot of kit, so this could work.

Claim to fame
Cache Accelerated Sequential Layout is what Nimble calls their secret sauce.

CASL combines a variable block size, in-line compression, application-specific block sizes and checksum and compression data kept in the block header. They coalesce the blocks and only write in full stripes to disk.

The box has a large flash-based cache where the full stripe writes are also written, overcoming the small write performance hit that flash shares with parity raid. This also insures a high percentage of cache hits on the first read.

The system maintains an index of where all the blocks are written. Typically, this index is also held in flash for maximum lookup performance.

App-specific block sizes
Nimble uses of variable block sizes to improve performance. For example, the last three versions of exchange have all used different block sizes. CASL recognizes the different versions of Exchange and dynamically adjusts its block size to the best fit.

They claim a 2x performance advantage on Exchange databases.

Coalesce
They take the variable size blocks then coalesce those blocks into big chunks and write to flash. They write in large blocks – full block writes to flash and in full stripe writes to disk. Result: fast reads & writes across both media

Their page sizes are variable but small, ranging from 4KB to 64KB. The greater granularity means that frequent snapshots are much smaller than large page size systems like EqualLogic.

The StorageMojo take
There’s no reason that data protection should be separate from data storage. We’ve been moving towards integration since the CDP craze.

The average business wants to store and protect their data and they don’t want to spend much time or money on it. Nor should they.

With powerful commodity processors and nickel-per-GB storage there’s a huge market for a box that – or 2 or 3 boxes – that

• Stores terabytes of data
• Protects that data with local replication and frequent snapshots
• Auto-connects to cloud storage for DR and archiving
• Doesn’t confuse users with LUNs and stripes
• Offers Time Machine like data recovery to end users

It will look like magic – as any sophisticated technology should – and you’ll buy it at Office Max. As with any volume product the key will be architecting to maximize the user experience at an affordable price point.

Nimble certainly has the right idea.

Courteous comments welcome, of course. I don’t know which analyst the Nimble guys are blowing their money on, but it isn’t me.

These requests came in over the transom in the last couple of days. Maybe some StorageMojo readers have wisdom to share.

I have a question I hope you can help me with. My boss asked me . . . to research HP Left-hand SANs and Dell Equallogic SANs. Do you have any special knowledge of these products and, if so, would you make an informal recommendation?

What say you, StorageMojo readers? If you evaluated both, why did you make the choice you did? Vendors welcome to comment, but please identify yourself as such.

The StorageMojo take
AFAIK, both products are good iSCSI systems. Both are backed by major corporations. EqualLogic may be stronger in the channel today, but HP has channel chops as well. HP’s blade servers may be a more expandable platform, but EqualLogic’s software portfolio may be more affordable.

Translation: you could do worse than either of these.

Part II
Another customer perplexity: service.

We have a pair of HP disk arrays, EVA 8000 and 6000 and I am looking for a consultant to help up with storage planning. Do you do such work or could you recommend someone to me. I am looking for someone who goes beyond just being a seller, I have plenty of potential sellers already.

The writer is in a small city in the Mountain West, so you should be used to working remotely with clients. No, not in Arizona.

The StorageMojo take
HP folks may be wondering: why doesn’t he call HP? My guess: not big enough for a direct engagement.

Courteous comments welcome, of course.

Imagine cloud storage that didn’t cost much more than bare drives. High density storage with RAID 6 protection, reasonable bandwidth and web-friendly HTTPS access.

And really, really cheap.

Raw disk cost is only 5-10% of a RAID systems cost. The rest goes for corporate jets, sales commissions, 3 martini lunches, tradeshows, sheetmetal, 2 Intel x86 mobos, obscene profits and a few pale and blinking engineers in a windowless lab who make the whole thing work.

Storage for ascetics
But let’s say you didn’t want the 3 martini lunch or the barely-clad booth babes. All you want is really cheap economical, reasonably reliable storage.

You aren’t running the global financial system – what’s left of it anyway – and you don’t have a 2500 person call center hammering on a few dozen Oracle databases 7 x 24. No, you’re thinking a quiet cloud storage business for SMB’s, maybe backup and some light file sharing, that will give you a nifty little revenue stream with annual renewals so you can see trouble coming 12 months in advance.

Enough redundancy so when something breaks you can wait until morning to fix it instead of an 0300 pajama run to the data center. Easy connectivity so you aren’t blowing the savings on Cisco switches.

Bliss
Well, you aren’t the only one. Backblaze, a new online backup provider, designed the Storage Pod for their own use and are sharing it with everyone. They aren’t in the hardware business and I think they figured sharing it would be a nice little attention-getting device.

It worked for me. Here’s the box – which they are using in production.

Here’s an exploded diagram with a simplified BOM:

And then there’s the (free) software. 64-bit Debian Linux, IBM’s open source JFS file system and HTTPS access. Put a stateless webserving front end on it and you’re good to go. Scale out the webserver and add storagepods to grow the system.

The StorageMojo take
This isn’t general purpose or high-performance storage. Nor is it backed by global network of 7 x 24 service professionals. But there are a lot of applications out there that just need a big bit bucket. This is it.

No one is manufacturing this for you either – which is a good thing. If you don’t know what you’re doing you can get in a lot of trouble with a lot of data real fast. Want to be the Bernie Madoff of cloud storage?

But the density is good, the performance is reasonable, the availability is decent and the price is right. This is a DC-3, not a 747. It is all you need for the right application.

Courteous comments welcome, of course. See the plans and get the box model in the paper Backblaze put together.

The video folks have an interesting set of problems: large needs; major bandwidth; time-critical collaboration; lots of metadata; and more. Like budgets. I do some video production myself and empathize.

They are today where most of us will be in 10 years: lots of large files; local and remote sharing; processor and bandwidth intensive operations; large archives of wanted and rarely accessed files. Today high-end video folks are working at 2k, 4k and, sometimes, 8k video resolutions – and 10 years from now I wouldn’t be surprised if home users weren’t too.

What prompts this is a note I received from, well, I’ll let him introduce himself.

I have a boutique post-production company and I’m a filmmaker. We are small, under a dozen, but swell to a few times that size with freelancers on a project-by-project basis. Because we work with very high resolution media, we need a lot of space, and very high throughput to each user. . . . [W]e’re all working with 2K and 4K media (300 and 1200MBps respectively to EACH user) and 3D animation rendering. . . . We use a mix of Linux, Windows, and OS X clients. In total, we could easily make use of 100TB+ right now, and prefer to stop archiving everything to tape and deleting it, but rather migrate to another tier of storage but keep in one global namespace with the tape just for disaster recovery. We also need security administration.

I can’t find a storage system that does all this. DataDirect Networks seems to be the du jour high-end storage for my industry, and supposing I’m willing to finance that big-ticket brand, they still don’t have a filing system answer. They’re suggesting StorNext or CXFS, and I know the multi-user scalability and expansion limitations well (can anybody say “forklift”?).

The closest I’ve come is Lustre. It seems like it would fit the bill nicely, especially since we’re savvy to integrate in-house, except that it is Linux only, and NFS/CIFS gateways don’t seem like a great idea. I keep hearing they’re working on at least a Windows client, but who knows when it will be ready?

Can you help at all? What have I overlooked? Doesn’t anyone make what I’m looking for?

Short answer to last question:
No.

Longer answer:
No. But there are workarounds.

For those new to video, here’s an abbreviated chart of some video rates in megabytes per second:
[Adapted from Integrity Data Systems which offers the whole chart. Aspect ratios and frame rates left out.]
Update: Larry Jordan, a writer and trainer in video editing, graciously wrote to let me know that the above data rates are uncompressed – and that most production houses would use compressed data. The amount of compression varies based on the codec as Larry explains in this informative post. End update.

Issue 1: Interconnects
GigE won’t even handle 32-bit RGB standard def video. And when you get into HD video it gets hairier fast. Trunk multiple GigE’s? 10GbE? 4x Infiniband? FC? eSATA or PCI-e direct attached storage?

Issue 2: Virtualization
A single address space is a wonderful thing. You’ll need a software layer that clusters multiple boxes. You’ll also probably want to build an archive infrastructure that is distinct from your higher performance working set storage, but some vendors will disagree.

Likely software suspects include IBRIX, Parascale, Caringo, MatrixStore, Bycast and Permabit.

On the combined HW/SW side there’s Panasas and Isilon. Something tells me there are some other options, like HP’s Extreme Data Storage 9100, that are also applicable.

Lustre is not a product I would recommend since it was designed for HPC, a market where PhDs work as sysadmins. Sun may have tamed it since they bought it, but it is a non-trivial piece of software.

Come one, come all
StorageMojo readers are invited to offer their 2¢ worth. Architecting is non-trivial, especially if money is an object.

Update:
Our interlocutor wrote in to add some detail:

thanks for the response. Here’s some answers:

– We can manage expensive interfaces like 10GigE and Infiniband QDR. We’ve been paying for dual-channel 4Gb FC for the past few years, after all. I just want to also allow standard Gigabit connections to the cheap seats without a lot of complexity. So I guess the jargon for that would be “multiprotocol” switching?

– The large naming space might be a luxury. The fact is that jobs come in one of three general sizes, and we could have volumes of that size waiting to take on new jobs as they come in, so at least there is one namespace per job. As you said, capacity is cheap…

– Truth is I am pretty savvy, but other than that we have a lot of power desktop users but not sysadmin types. I contract some people with steady part-time work, but it has been our business model to try to keep as many of our full-time people on the creative and producing side as possible, and not in support/administration.

The one thing I don’t understand is what you say about Infiniband not being so great when there’s lots of node churn?

I know what you mean about DAS, but I think I’ve ruled out distributing the data through push/pull from a central repository. The fact is jobs just move to fast through here for that, and we often have about two seconds notice that we need to bring a certain job’s data to System X, Y or Z to do work on it. It’s very dynamic.

I see some brands in your blog post I haven’t checked on yet.

What turned me onto Lustre is that Frantic Films in London has deployed it. They’re the only ones AFAIK.
End update.

The StorageMojo take
Some thoughts on the infrastructure issues.

Capacity is cheap, network bandwidth is expensive. Raw SATA disk is less than $0.10/GB. 10GbE switch ports are over a grand apiece. Infiniband is better from a price/performance perspective, but not as friendly for networks where there is much node churn – unless that’s been fixed in the last few years.

Direct attached storage will give you the best performance – especially with 4k. The new PCI-e attached arrays from JMR and others can offer up to 4,000 MB/sec bandwidth. Stripe across 4 of those and you’ll be able to handle 8k.

Transaction processing is well on its way to niche status, like mainframes and hierarchical databases that once ruled the earth. It is a big file world out there and the files are getting bigger every year.

Courteous comments welcome, of course. I’ve done work for many of these folks – but not all – at one time or another.

Hewlett-Packard’s acquisition of the LeftHand Networks shows how cluster storage is going mainstream – and how HP plans to be right in the middle of it. First PolyServe and now LeftHand.

This is about commodity-based clusters
Not iSCSI or GigE or 10 GigE as a storage interconnect. Fibre Channel’s failure to move downmarket – and Infiniband’s similar problem – means GigE is the only game in town.

Reaching the huge, not currently imploding, SMB market requires meeting people where they live. SMBs don’t live in Fibre Channel glass houses. GigE isn’t ideal, but it’s cheap and it works.

Did HP overpay?
$360 million isn’t pocket change, but it is only about 4x the $86 million investors put in. The investors get some nice coin, but it isn’t the 10-bagger they were hoping for.

Once the Lefties go through the interminable internal HP meat grinder, sales will grow rapidly. I suspect they weren’t up to Isilon’s $100M in sales – maybe $70M – but LeftHand was much closer to profitability. Net net: the price looks fair for a market leader in a high-growth market.

HP vs EMC
Battle of the competing cluster storage visions. Polyserve handles files; LeftHand blocks. EMC’s Maui is aimed at large-scale distributed file storage, a utility that ISP’s might resell to SMBs, but nothing an SMB would implement on their own.

Which will win – and there’s room for both – rests on the answer to the question Are there economies of scale in storage?. If there are, small-scale clusters sales will suffer and Maui should win.

The StorageMojo take
This is cluster storage market skirmishing, not a pitched battle. That will come but right now everyone is feeling their way, coming into the market from different directions, waiting to see what clicks.

Right now though, HP seems to have the strongest position. XIV is too new; Maui even newer; Lustre too complex; Isilon is digging out of a big hole. HP has the pole position with implementable products today and the services to back them up. Should be a powerful combination.

Courteous comments welcome, of course. Disclosure: I’ve done some work for HP, Isilon and Sun.

StorageMojo has long held the view that our storage workloads are changing: more file storage, less block storage; larger file sizes; and cooler data. While all the indicators said this was happening it’s good to find a study that confirmed this intuition.

In the Measurement And Analysis Of Large-Scale Network File System Workloads (pdf) researchers Andrew W. Leung and Ethan L. Miller from UC Santa Cruz and Shankar Pasupathy and Garth Goodson of Netapp measured 2 large file servers for 4 months. Their results are worth reviewing, since so many of the optimizations in storage infrastructures rely on workload assumptions.

Unstudied CIFS
The authors point out that there have been no major studies of the CIFS protocol, odd since it is the default on Windows systems. Furthermore, the last major study of network file loads was performed in 2001 – seven years ago – an interval in which average this drive sizes have gone from 20 GB to 500 and network speeds from 100 MB to 1 GB.

Most surprising, however is that no published study has ever analyzed large-scale enterprise file system workloads. Researchers have studied workloads closer to home: university and engineering workloads.

Enterprise workloads
One was a midrange file server with 3 TB of capacity with almost 3 TB used by over 1000 marketing sales and finance employees. The second server was a high end Netapp filer with 28 TB capacity – 19 TB used – supporting 500 engineering employees.

Yes, marketers, engineers get the good toys. You can cry about it over your next 3 martini lunch.

Some significant differences from prior studies:

• Workloads more write oriented. Read/write byte ratios and are now only 2 to 1 compared to the 4-1 or higher ratios reported earlier.
• Workloads less read-centric. Read/write workloads are now 30x more common.
• Most bytes transferred sequentially. These runs are 10x the length found in the old studies.
• Files 10x bigger.
• Files live 10x longer. Less than half are deleted within a day of creation.

Cool new findings

• Files rarely re-opened. Over 66% are re-opened once and 95% fewer than 5 times.
• Over 60% of file re-opens are within a minute of the first open.
• Less than 1% of clients account for 50% of requests.
• Infrequent file sharing. Over 76% of files are opened by just 1 client.
• Concurrent file sharing very rare. As the prior point suggests, only 5% of files are opened by multiple clients and 90% of those are read only.
• Most file types have no common access pattern.

And there’s this: over 90% of the active storage was untouched during the study. That makes it official: data is getting cooler.

Another interesting finding: 91% of VMWare Virtual Disk (vmdk) files accesses were small sequential reads – not the larger sequential accesses I’d expect.

The StorageMojo take
The writers rightly suggest that given the rarity of file reads after creation it makes sense to migrate files to cheap storage sooner than later.

Perhaps primary file storage should be thought of as a large FIFO buffer – tossing 3 month old files to an archive for long-term storage. A data flow architecture instead of a series ever-larger buckets.

Kudos to NetApp and UCSC for this work. It seems like NetApp has been doing the best job of leveraging academic researchers lately. I’d like to see them get more marketing mileage out of their good work.

Courteous comments welcome, of course.