An Open-Source SAN

by Robin Harris on Thursday, 17 August, 2006

Update Over at TechRepublic, Scott Lowe offers another view of AoE here. If I were an SMB VAR, I’d be checking AoE out.

It Is About Time
Here’s a potential game-changer – especially for the SMB market. It is low-cost SAN functionality based on local Ethernet. From a company named Coraid. Available for Windows, Linux, Solaris, FreeBSD and Mac OS X.

Wait a minute? Isn’t that iSCSI? It is a block device after all. Nope. Different. IMHO, better. There are some Don’t Gets, and a lot of Don’t Needs.

Putting Local – And Storage – Back In LAN
Coraid’s innovation is the open ATA over Ethernet (AoE) protocol. The big Don’t Get is that the protocol isn’t routable – it is strictly local – no IP involved. So the Don’t Needs include no TCP/IP overhead, no TCP/IP offload engines, no CPU-cycle sucking and latency-inducing TCP/IP stacks. AoE sits right on the data link layer – level two – of the ISO network model, so with a switched LAN – is there any other kind these days? – you get very low latency and full network bandwidth across a low-cost, industry standard LAN.

The other big Don’t Get: expensive and finicky Fibre Channel HBAs, switches and storage, along with the extra bandwidth FC offers. Like FC, AoE appears to make very effective use of available bandwidth – maxing it out with storage traffic. You’ll want a dedicated storage network to run AoE across.

Practice Makes Perfect
Even though it is cleared for use with Oracle, it probably isn’t a solution, today, for habitually late adopters. You’ll need to think through your security and system management processes to ensure that data doesn’t get munged by an inattentive sysadmin. A dedicated AoE SAN is a start, and VLAN techniques can help partition off potential damage-doers. The key: it just looks like a disk, and anything goofy you can do to a disk you can do over AoE.

Write Once, Read Never?
So far it appears that Coraid is the only company building AoE hardware. It doesn’t appear they are trying to keep anyone else from doing it, only it just hasn’t happened yet. That might be a worry for some folks. So in a smart move, Coraid has a Linux tool called srcat a tool for recovering data from the raw disks on a Coraid JBOD or array. So if the company goes belly up, controller breaks, no replacements available, you can still pull the drives and use srcat to pull the data off. Neat. Take
Congrats to Coraid for a creative way to bring the benefits of network economics to storage networks, just as some of us thought FC would 10 years ago. By creating an open platform and protocol, they’ve started the open-source equivalent of a SAN. If you require – or would like to be able to afford – a lot of storage capacity, you should certainly check these guys out.

Update: Over at Tech Republic, Scott Lowe offers some more info on AoE. The (literal) money quote:

AoE is cheap! An array capable of supporting up to 11.25 TB from Coraid starts at less than $4,000 without disks. Today’s price for a 750-GB disk at is $400 and the unit supports 15 disks. So, for less than 10 grand, you can get 11.25 TB of shared block storage. If you do the math, that runs at about $888/TB or $0.87/GB. Not bad!

{ 7 comments… read them below or add one }

Alan Berezin October 6, 2006 at 1:07 pm

Does anyone have some experience with coraid raid boxes. Specifically, any performance comparisons with, for example, low end iSCSI boxes like Promise? Also, any experience in a real clustered environment? Finally, the technology seems like it should be lower network overhead than iSCSI but has anyone actually measured any real performance benefit?

josh April 11, 2007 at 11:26 pm

I found this blog via google so this entry is a bit old – just wanted to throw my 2 cents in – i’ve installed a SR1521 from coraid this week, fully populated with 750gb drives and their nas gateway serving it up. Not bad at all for the money. You need to be very comfortable at the command line to make this thing hum, but their support has been very fast and i have not had too many huge problems. worth a look if your department is a on a tight budget for sure.

Saifi Khan July 17, 2010 at 8:40 am

Hi Robin:

In retrospect, how does this stuff look to you today ?

Looking forward to reading your thoughts on FCoE.

Have a nice weekend.


Leo Vang March 28, 2011 at 8:47 pm

I’m very interested in this, it’s been quite a few years since this post and I would love to hear and see a production setup of this. Preferring to a redhat/centos cluster or cloud hosting if possible.

Dave July 15, 2011 at 3:45 am

Leo – here’s the update … FC has been and gone; AoE never took off; iSCSI is the winner.

Coraid was a great idea at the time but ultimately it was the Beta video casette to VHS’s iSCSI 😉

There are plenty of interesting open-source SAN/NAS technologies however – Openfiler (Linux based) and Nexenta (Solaris/ZFS based) are just two to watch …

Dave July 25, 2011 at 1:31 pm

I disagree completely with the Dave before me. I have over 200 TB in 9 coraid devices of many sizes and love it!!!

Just ordered everything to upgrade my core switch to have 10GB and the new SRX SFP

Happy Coraid customer for 4 Years!!!

Mostly Fedora and a Few CentOS
The way we use coraid doesnt work great with windows servers (not architected to)

madhavi August 7, 2012 at 9:38 pm

Coarid only makes the AOE driver (initiator) available – for Linux, Solaris and FreeBSD. I dont see them publishing their server software anywhere on internet? how come they are opensource? The Linux AOE target vblade is single threaded and not updated since 5 years!!

Leave a Comment

Previous post:

Next post: