The reason why FCoE is more compelling than iSCSI is because IP is a POOR (very) protocol for storage. Sure, you can do “jumbo frames” as a temporary bandaid for Gbit, but what about 10gE? And even then, you’re sacrificing more and more cpu overhead to run the IP stack… it’s just the wrong answer ultimately for storage. Any FCoE implementation will make better use of the available bandwidth than any config of iSCSI on the same.

Now.. with that said, if the idea is that storage MUST include ubiquitous routability across the Internet, then certainly iSCSI has a place.

So maybe FCoE doesn’t displace iSCSI, it just becomes the better choice for replacing FC than iSCSI. Does that make sense?

Personally, I think iSCSI is very wrong. But on the low end, I know it’s popular… it’s just also fraught with reliability issues… in fact, so much so, that I’d wager that the cost of reliable iSCSI is comparable to today’s FC (speaking 10gE iSCSI vs. 8Gb FC).

Cause if I remember right, one of the drawbacks to the AoE (ATA Over Ethernet) protocol is it is not routable, ie, can’t go to other networks.. (such as your disaster recovery site!)

This is not a problem though, as you can use a pair of systems (or more than a pair) to handle the disaster site communications. Scale out the AoE as far as you want (40k devices last I read), use machines (we are biased, our JackRabbit servers support the latest model AoE drivers out of the box, not to mention iSCSI target/initiators, and when FCoE stabilizes … ) to set up a mirror. Run whatever mirroring software you wish (BakBone, etc). If you want to do block level mirroring, set up DRBD, GNBD, …

AoE is nice in that it is quite simple. You can handle routing at a different layer, which allows you to keep AoE simple.

The issues which “plague” AoE (and iSCSI, and FCoE, …) is that the data is not encrypted in flight. So your storage networks need to be private, and not connected to your backbone. Backbone traffic should be done via encrypted links (which in the model noted above, is actually fairly easy to do).

@Anders: maybe in small shops (which is where we see plenty of uptake and where we try to push it hard), but where FC is entrenched, not happening. Storage admins who have a nice stable FC environment aren’t willing to throw their entire environment over to the network guys while hoping for the best. And I don’t blame them one bit. To be quite honest, when it comes to 10GbE, the only real demand I’ve seen thus far is in VMware environments, and generally in those cases, NFS has proven to be a better fit than iSCSI.

@Marc: The only person alive? Last I heard, QLogic had CANCELLED their 10Gb iSCSI offering in favor of FCoE. So I guess I’ll go with “no”. You might be the only person alive who think FCoE and iSCSI have ANY reason to co-exist. Either FCoE will *converge* the ethernet and FC worlds in a way iSCSI has failed to do, or it will fail entirely.

To TimC questions. The next steps for Ethernet are 40Gb/s and then 100Gb/s. Both Intel and Broadcom have talked about 40Gb/s ~ 2011/12 in the eetimes. 100Gb/s follows by a few years.

You are correct – 8GB FC is approximately equivalent to 10GB Ethernet because of the encoding schemes employed, but I your real-world projection that is will be faster are probably not correct. Latencies in the end nodes dwarf those in the network. FWIW, the converse argument made by 10 GB Ethernet fans is equally misdirected.

Then you wrote this: “I think FCoE might kill iSCSI, but I don’t think it’s killing FC.” ??? You might be the only person alive who thinks FCoE might knock off iSCSI.

First, you’re a bit inaccurate with the statement that 8Gb FC is slower than 10GbE. While on paper they are, it’s much like the hard drive manufacturers rounding fun. At the end of the day, with all the overhead, 10GbE ends up being slower than 8Gb FC. Maybe that will somehow change with FCoE, but I guess I’ll believe it when I see it.

This isn’t a whole lot different than the “OMG iSCSI is going to kill FC!!!!”.

I think FCoE might kill iSCSI, but I don’t think it’s killing FC. What’s ethernet’s next step? When do they plan on getting there? I know QLogic already has a 20Gb fibre channel interconnect protocol up and running on their new 8Gb FC switches.

Intel is promoting software FCoE (or iSCSI, they probably don’t care) with cheap dumb Ethernet NICs, but I doubt the FC vendors have the stomach for such creative destruction.

Then there is the sanity of the investment protection argument. When customers realize that they have to install an Ethernet FCoE to FC gateway (running at what speed?) to connect to existing FC equipment – how ugly is that going to be? Not so ugly, you say, because the switch vendors will be more than happy to provide blades with premium pricing to get the job done? Any way you look at it, the cost compared to vanilla iSCSI is unattractive.

All that said, I know there are lots of FC customers that want to believe in FCoE. Many of them will at least try it. They have to. They are stupid not to. Some will stay the course while others move to iSCSI. That migration will be easier then they think for most of their systems and data.

I’ve only bought iSCSI gear recently, although FCoE sounds interesting. TCP wasn’t designed for storage workloads, and may in fact have some problems as applied to storage data-flows (as mentioned in a paper linked here recently). I would of course really like to see an IP-over-Ethernet, multi-cast aware storage networking layer, which would be ideal for a clustered sotrage environment with remote DR.