Over the years StorageMojo has seen several architectures that just seemed smart, but whose market potential was blighted by management and funding issues. Violin Memory was one. Coraid was another.
In both cases a new CEO – at Coraid, Kevin Brown – has made a world of difference. New funding, new customers, more focus. All good.
What does Coraid do?
Storage, of course. Inexpensive block-based, scale-out, network storage. Global namespace.
Define inexpensive
How does $0.40/GB sound?
How?
Let’s start at the server. The HBAs are Intel NICs that are re-programmed to present themselves as SCSI controllers. The server sees a SCSI controller, but it’s Ethernet out the other side.
Running over the network is a connectionless datagram protocol that accesses the storage on commodity x86 servers. The protocol is fast and lightweight, being based on the simple and well-understood ATA command set.
There’s a router for connecting sites, but most users use it in a single site.
Who?
People who want a lot of inexpensive block storage. Average order is ≈175 TB. Average sales cycle is 30 days.
Coraid is a horizontal play, but they get interest from the usual big data suspects: media and entertainment; science; service providers; private clouds.
The StorageMojo take
Coraid takes the commodity game seriously: commodity servers; commodity HBAs; commodity open-source protocol; commodity network; and commodity SCSI drivers. Not much custom engineering required.
It’s prices reflect that. There’s a huge unmet need for fast, cheap storage that Coraid is tapping. Their biggest problem is that they do things differently than everyone else, and it takes people a while to decide to work with those differences.
Which is a shame. The more competition for the block storage dollar the better it is for all consumers.
Courteous comments welcome, of course. I saw Coraid as part of a Tech Field Day program, where they, among other vendors, helped pay for transportation, food and lodging.
I had an interesting conversation last weekend with a person who worked at coraid for a while – who mentioned to me that while the cost is low when you go with the most basic of basic stuff, when you load up on the software features they often found themselves more expensive than the likes of Dell or even NetApp on occasion too.
He did give the system high props for being easy to use, and very simple to setup. The cost factor was really interesting to me though I wouldn’t of expected them to be in the same price band of the likes of NetApp with certain configurations, since their primary selling point seems to be low cost.
Came very close to coming to terms with Coraid, and the determining factor is of course price. They scale very well and the speed/latency meets our requirements. I would be interested in hearing more about “load up the software features”, as those type of surprises are not pleasant.
Nate : sorry but this is completely false because Coraid prices is all inclusive… No software license costs…
You pay for the box and support and you have all in once, all features and upgrade included which is far from what the legacy storage vendors you mentioned are doing 🙂
How can AoE be considered a standard when CORAID is the only one skipping AoE Kit and no one else can produce it because of CORAIDs patents?
Dude, I took a quick look at Coraid’s 6 patents and patent applications and didn’t see anything that would preclude others from developing and shipping AoE storage based on patent issues. The patents seem to be specific and narrow.
A quick search found a Taiwanese vendor who offers AoE storage.
All in all, it looks like the AoE protocol is open, not overly encumbered by patents and available from a 3rd party. Why isn’t the support broader? A combination of market demand and a reluctance by larger storage companies to embrace a much lower cost storage technology would be my guess.
I think I pretty much know why AoE didn’t get much traction: because iSCSI is available everywhere, well known, well supported and works fine. I’ve done my research on AoE and found that it had no significant advantage over iSCSI: throughput is the same, IOPS performance the same.
It’s an advantage for CoRaid because it’s their own technology, but that’s about it.
What about price? Seems like Coraid is way less than iSCSI boxes.
Does all inclusive include replication too? Maybe that was different/specific to replication since he said replication required another box it wasn’t built in(at the time (~few months ago)).
All,
I appreciate the discussion and wanted to add a couple quick thoughts from the Coraid perspective.
1) On pricing, our objective is to be a price-performance leader. We leverage the commoditization curves of off-the-shelf hardware and standard Ethernet combined with a scale out architecture to reduce the capital costs of storage without sacrificing performance. Except for very small deployments where the scale out architecture is not exercised, Coraid is typically a fraction of the cost of legacy storage vendors.
2) The protocol, AoE, is an enabler of our architecture by providing very low latency and high bandwidth access to storage (1,800 MB/s). Critically, it provides this high performance with a very simple operational model – it’s just regular Ethernet and every link is used automatically. Expanding capacity is as simple as plugging in another cable, without any special multipathing software or switch configuration. This is massively parallel, connectionless Ethernet and it simplifies cloud operations.
3) AoE is not proprietary. There are some other small players, particularly in the SMB, offering AoE solutions. The specification is open and the initiator code has been standard in the Linux kernel since 2005
4) That said, Coraid’s implementation of AoE is really an “unprotocol.†It’s the simplest way to send data over Ethernet, but is primarily internal to the Coraid solution. There are no third-party HBAs needed, no MP/IO software, no complex FCoE/DCB/CEE certifications. The solution requires Coraid, Layer 2 Ethernet, and nothing else.
The goal is to transform storage to make it simple to deploy and operate, particularly at scale. Given all the pain that customers and vendors have endured in the Fibre Channel world, it’s understandable that people are wary of new storage protocols. But Coraid has over 1,500 customers that are realizing benefits in price-performance and simplicity today.
If you have questions, please don’t hesitate to contact me.
John Gilmartin
Coraid
As an SMB-like customer embedded in a very large organization, I reviewed Coraid, iSCSI, FCoE, NFS, and other storage solutions over the past few years.
Pros and Cons for us:
– Simple and very attractive architecture. iSCSI, FCoE, FC, NFS, are all much more complicated to understand, install, operate, and troubleshoot.
– On again, off again, Mac OS X support. We really wanted Mac OS X support for several projects, but that was not a deal killer.
– However, with a single source of critical AoE parts and maintenance (Coraid), I could not risk a major investment.
I have some regrets about the storage choices we did make, and maybe the Coraid risk would have been worth it.
In response to the posts by Robin Harris and others – We are a reseller of AoE storage products for the SMB markets within the USA. The company – JW Electronics (“Taiwanese vendor”) is our supplier.
Thanks.
Keri Alleyne
Tangible Storage
AOE=Coraid hardware dongle=use FC or 10GbE instead
Let me tell you about my experiences with Coraid…
I wanted to look at some scale-out storage, so Coraid was on the list. I wasn’t so sure AOE was the way to go because like everyone says here, it’s basically a Coraid only offering and that made me nervous — my experiences below led me to totally cross Coraid off my list.
I decided to experiment with a friends Coraid unit, so I borrowed a unit from a friend. I placed an order with Coraid to buy the AOE NIC required for Windows and was told a VP refused to fill my order because I wasn’t a “new customer” i.e. someone that didn’t directly buy a Coraid unit from them.
Now under Linux the driver is open source and built in, so why under Windows would they not want to sell the card? They said due to people buying used units, they are not selling the card.
And that is why I’ll never buy anything from Coraid and will use industry standard FC HBAs or 10GbE — I don’t want to be at the mercy of Coraid or any vendor for proprietary connectivity to hosts.
Could you imagine if Dell refused to sell you a NIC because you weren’t a pre-existing customer? This decision comes from the VP level at Coraid. It’s an AOE NIC as a dongle strategy — to me it speaks volumes in their lack of confidence for their own technology.
We’ve been one of a really few AoE vendors shipping Windows AoE initiator for a long time (since 2006 or so). After some point Coraid had decided we’re getting too much money so they had kicked us away and had replaced our software initiator with their own pseudo-HBA hardware one. Nice but raises two issues: 1) Totally against common sense of wisdom and their own commercials… Why more CPU hungry iSCSI can run software and less CPU hungry AoE needs a HBA? 2) Not very good to push away one of a few early adopters who had helped to evangelise Windows community… As a results we’ve experienced a blood wash, collapsed any AoE initiator and target development (Windows kerenel mode AoE target was never released to public), put software in “free-try-and-never-buy” mode and recommend to stay away from the whole thing as far as you can. For a reason. Lock down in a vendor is always a bad thing – are you a customer or an independent software vendor (like we). There are issues with AoE protocol itself I don’t want discuss technically here (it would be natural to use SCSI and not ATA command set as no OS has native ATA storage stack, no routability so DR should run some clumsy something-over-TCP wrapper – these are flaws by design as there was no replication for AoE and no VAAI back in 2005) but it’s really other things killing the whole idea. You have John (see above) as a customer and now you have Anton (me) as a vendor. We’re telling basically the same…
Good luck to Coraid!
Anton Kolomyeytsev
CTO, StarWind Software
(ex-CEO for Rocket Division Software who’ve been supporting AoE stuff)
Really, CPU Cycles (regarding iSCSI).
I have an ASIC for that.
Wow, I hate to anonymously bash a company but we tried to purchase a coraid unit. Our purchasing department needed them to fill out a debarment form (a form certifying they don’t have pending criminal charges or have been charged or indicted in the past 3 years) and they refused to do it, putting the kibosh on the deal. Wow. I’m literally shaking my head.
We’ve been using Coraid equipment – including the SR, CLN and VS series – for the last 7 years. Their pricing was the key factor for choosing them back then. However, this spring Coraid refused to renew our Service Agreement even though some of the equipment was purchased just 2,5 years ago. We’re currently migrating data to storage from one of the “big players” we’ve also been working with for years.
From my point of view Coraid lost their single most significant competitive advantage when they decided it was no longer possible to purchase their storage product without disks. A few years back you could find the price of the array on their website and then find the least expensive Coraid-recommended disks from your local dealer. You could immediately calculate the cost of your investment and the $/GB ratio was awesome. Nowadays you’re forced to purchase disks from Coraid and as you might guess the pricing is more or less on par with the “big players”. However their support and partner organizations are not. Good thing is you’re still able to replace a failed disk/component by youself.
Coraid is trying to leave the “propellerhead” category and become a more serious player in the storage market. This, however, means that some of the flexibility and pioneer-attitude is lost along with the extremely competent pricing. Hopefully it also means something good like thorough testing of new products and quality control.
In the end I guess it’s all about keeping the investors happy.
I too wish all the best for Coraid.
-disclosure NetApp Employee-
Coraid must be doing a few things right, and as the number of interesting storage vendors declines thanks to consolidation in the market I personally take my hat of to them for daring to be different. Well before my time NetApp also used to sell just the controllers (or maybe it was just the ONTAP sofutware) and allowed resellers to pull together the hardware themselves. I imagine when we stopped doing that, people complained about that too. Changing your support and go to market model as you mature will always annoy someone, even if its done for the best long term interests of all the stakeholders.
From an egineering perspective I find the tech quite interesting, but the simplictiy in my mind seems to come at a price. For example, I imagine the ATA primitives would only allow for very vanilla storage operations. Without the things like extended SCSI copy, write same etc all of which are part of the SCSI command set, there is no way to take advantage of advanced storage managment technology like VAAI/Hole Punching for thin provisioning etc. Because FCoE does support this, you can take advantage of a lot more array based intelligence than you can if you limit yourself to the ATA command set. The are other uber-geeky stuff that may or may not make a difference such as better error reporting in SCSI, better command queueing for performance, skip-mask writes all of which are part of the reason why SAS drives outperform their SATA cousins.
The two main questions in my mind are
1. Why bother going to Layer-2 for this when iSCSI at Layer3 is mature, inexpensive and has broad multivendor support
2. If you are married to layer-2 why would you bother with ATA over when you can get a better command set from SAS or FCoE
There is an argument for convergence in that everything including internal short hop disk interconnects will run over Ethernet eventually which may or may not happen, but this would only become really interesting when/if the drive vendors like Seagate start producing native AtaOE drives, which I dont see happening any time soon. If they did I’d guess they’d implement a full SCSI command set rather than ATA, resulting in something like SAS over Ethernet, but that sounds so much like FCoE that I’d bet on native fully commoditised FCoE for the eventual winner (even if emotionally I’d much prefer FCoTR).
The main value proposition for Coraid seems to be that they sell their gear at about 40c / GB for 150TB+, if that’s the single most important buying criteria I’m pretty sure there are a bunch of other vendors out there who are prepared to sell storage at that price without asking people to take the risk of being so very different.
I’m kind of conflicted, with my NetApp hat on, Coraid are competition and my Conan the barbarian tendencies of wanting “To crush your competition– See them driven before you, and to hear the lamentation of their investors!” conflict with the technician in me that likes to see new an interesting ways of attacking storage problems. As a result I wish coraid luck … but not too much.
So, their new ZX series is interesting – using ZFS to do RAIN over AoE. Though, for me, that results in an awful lot of disk hanging off of a single non-redundant NAS head unit.
Also – the tech specs indicate they are using a RAM drive for the ZIL. I must not understand ZFS correctly because if I do, that’s a bad, bad idea? Unless they are using a card with battery backup of some sort.
Coraid has not released their OS to public, it is not open source? only the specification…only the driver (initiator) software is released by coraid …..there is mystery around the openness of Coraid? When VMware can release ESX source under GPL why not Coraid?
I appreciate this post. I knew the early coraid goes from LinuxWorld in SF back in 2006 and really thought they were great guys. Now that we are heavily engaged in storage as a leading line of business, I wanted to add the coraid offering because of its scale-out capabilities, commodity hardware and excellent cost per GB.
But after reading the reseller agreement I was shocked. I’ve never read so many “you will indemnify us” sorts of statements and “you will provide this” (or we may terminate) vs “we may provide this, unless we are unable.”
I’ve signed a lot of contracts over the years, and done a lot of business with a lot of large corporations both representing their gear and selling to others. You can tell a great deal about a company by the legal agreements they put forward, and in my experience, if a company is trying to nail you to the wall with a contract, its because they are the ones likely to behave in questionable ways.