I am the person who had the split brain problem. Datacore have recommendations for stopping this kind of thing i.e redundant links between datacore servers. I myself can highly recommend datacore and sanmelody. It is a very versatile solution, datacores support is second to none and the cachng is excellent. I have had 4 power outages (2 full campus wide and 2 building specific including the buildings with the sanmelody servers in). I can report I have never lost any data and my virtual machines have always restarted perfectly.

I never completely understood what was going to happen when I realised I was in a split brain situation in terms of getting the replication link back up again. I was not prepared to risk it so I used the vmware convertor to move the test vms from one esx server to the other (thus avoiding the replication part of the network). This is a secure method to resolving the split brain. I also took full backups using express before making the replication link live again.

I am not a San technician so you will have to excuse my lamens terms. But it is my thinking that If I had made the replication link come online again it should have synced everything over ok. The reason behind this is that only one virtual machine can be running on one san at anyone time. This means that the vms (although they were writing independantly) were only writing to their paritions on the disks and therfore changes could have been replicated over without any problems. Because essentially it could have been disasterous I took a different route to ensure everything was ok.

As I stated in my post on the vmware forum I have made my environment only ever point to one san for live vm’s. This is not a problem for me as it gives me control over the iscsi paths for my vm’s. I can turn off one san during the day if need be and my vms iscsi paths will failover perfectly. I can vmotion and turn off my esx servers. I have HA ability as well so all in all I can’t ask for more for the price. It was certainly an intersting project and I tried to follow best practises as best I could but as far as I was aware there was no documentation anywhere on the project i undertook. These sorts of implementations are usually done by consultants and I am a lowly sys admin with a love of technology. personally I think you have to be alittle mad to try and do it yourself but it is worth it.

The only sad thing is no-one actually understands the task I undertook…….

“ALWAYS MAKE SURE YOUR ISCSI PATHS FROM ESX TO SANMELODY POINT TO THE SAME SAN BECAUSE IF THEY DON’T AND YOUR REPLICATION LINKS BREAK YOU ARE GOING TO GET A SPLIT BRAIN SCENARIO.”

When using VMware HA or SRM you’re not just concerned about failing over your storage controllers e.g. DataCore servers on the same SAN, but failing over your servers on to completely different physical storage from one building to another, which is essentially what this customer was modeling. If you’re replicating data instead of sharing it, disk reserve operations are of no help at all - since reserving a disk on one disk volume has no effect whatsoever on the other volume.

2-node solutions are always (this is mathematically proven…) susceptible to at least one of two problems:

* A Safety problem (aka loss of data integrity, conflicting writes, etc.). This may happen because of a simple case of what people normally call a Split Brain, although there are more complex variants. Essentially, the problem boils down to in some cases allowing 2 nodes to write different data, some of which is lost or corrupted.

* A Liveness problem. The system is not HA, in some cases where one node fails the other node does not take over.

In the real world systems are more often susceptible to Safety problems than Liveness problems. I believe this is because customers are much more likely to notice a system being taken offline when only one node failed, than a system that allows some data to be lost and/or corrupted (which usually doesn’t happen and when is does is often not noticed…).

Based on DataCore’s description, it would sound like the system is not really HA, because only the node that holds the reservation will continue in the presence of a failure of one of the nodes. In a system with load-balanced volumes, this would suggest that 1/2 the volumes go offline when one of two SanMelody servers go down in a mirrored configuration. It also appears that if you snip the cables between two nodes that both of them continue operating… causing a Safety problem. Again, this is because customers would notice lack of Liveness, but normally not notice loss of Safety.

Aside from this, I just can’t imagine a customer ever purchasing a SAN that isn’t supported by Microsoft, but hey, live and learn.

John

Bte, I’ve downloaded a copy of sanmelody and am in the process of running results compared to iSCSI boxes and the Equallogic, using IOmeter runs so far I am seeing about 4 to 1 better results with Datacore sanmelody. I’ll follow-up as I get more data and do some runs.

Thanks for your comments Lukas, I totally agree. This poor customer is going to be fed up with all of us by the time we’re done here. Not trying to sound desperate, just full of passion for our products and never ending pursuit of the truth.

DataCore: The fact that a DataCore customer actually experienced a split brain is proof enough in my mind. How does either box know the difference between a box failing or the link going down? What if the failed box doesn’t release its reservations? More on this later.

Steven, I guess you didn’t notice. I was actually trying to be nice to EqualLogic for once.

First of all, a key benefit of clustering and virtualization is providing the capability to span volumes across multiple boxes (as EQL’s literature suggests.) Like LeftHand, as volumes are configured from spanning the disks on one EQL box to two boxes, the performance of those volumes increase because you have more disks working for you.

Regarding your comment: “It may lose certain volumes that are striped across multiple arrays, however, this is an optional deployment configuration, and volumes can be isolated to a single array.”

I interpret this to mean: If you’re worried about losing access to your data when a box fails, don’t let your volumes span multiple boxes. Let me think… performance is not optimal… and all volumes on any box that fails are still offline. I don’t get the argument.

As far as the capacity hit goes, yes there is a capacity tradeoff for data redundancy, and at least we give the customer a choice on a per volume basis, unlike EqualLogic. While were on the subject of “HUGE capacity impact.” Let’s take a closer look at the EQL data protection and capacity equation:

- EQL thin provisioning still requires that you reserve 10% of your volumes up front, and also requires reserves for snapshots and remote replicas (LeftHand’s thin provisioning is “reservationless across the board.) Add this to the hit for RAID 50, hot spares, and EQL software overhead and you end up with 46.9% storage utilization, without redundancy across boxes!

-Yes, and let’s not forget RAID 6. Why is Dell/EQL the only major storage vendor in the industry that doesn’t support RAID 6, or some means of protecting data against double disk faults in a RAID set? When you go to re-build a failed drive in their 7 drive RAID set onto a hot spare with 1TB SATA drives, you have a 51.5% probability of a BER event that could cause you to lose all your data (assumes 10^14 BER.) Even with Enterprise Class SATA drives (no data to say they use these) with a 10^15 BER, you still have more than a 5% probability of a BER event during a RAID rebuild.

If you don’t believe the story just ask EMC, HDS, HP, NetApp, and just about everyone else you can think of, why they have RAID 6 double disk fault protection.

We are happy to provide references as well from over 3,000 LeftHand customers.

BTW, no hard feelings. I love you guys. The drinks are on me if we ever meet face-to-face.

Secondly, while Datacore and Lefthand have nice features like Mirroring, they have it because they require it. An off-the-shelf server has many single points of failure. Lefthand overcomes this by introducing Netowork RAID on top of Hardware RAID. While this has a HUGE capacity impact in a 2 node configuration (which is a new configuration support for LeftHand) and has less capacity impact as you scale out storage nodes.

Equallogic handles HA in a more formal Enterprise fashion. The hardware is purpose built specifically for iSCSI traffic, which has helped make the PS-Series arrays the market leader in iSCSI performance. The hardware is also fully redundant, battery backed mirrored cache within the array, so Writes are Protected.

The issues around Datacore and FalconStor are similar, traditionally you put someone else’s storage behind those solutions, and have to manage each distinctively.

Go to http://www.dell.com/equallogic and sign up for a Webinar. You’ll see the ease of management, the ease of set-up and deployment. If that doesn’t convince you, Evaluate an Equallogic PS-series array, I can help you get a unit without a problem.

One last statement, the Lefthand VSA, publically states that there is about a 40% performance overhead, so not sure why you would look at that option. Lefthand does however make a good iSCSI cluster solution. I have enjoyed competing against them in many accounts.

I welcome you to call Equallogic references, we have MANY.

Also, about the DELL | Equallogic product line. We have several array options that all intermix as you scale. Everything from SATA, to 10K SAS, to 15K SAS. The line is now called in general the PS5000 series.

The plan is for an H/A setup in a class 1 datacenter with asynchronous replication over an existing DS3(..but dark fiber is in the works) to a remote site.

All things considered, the question could be framed, “Whom/What should be demanded for trial?”

I guess if you need true synchronous replication across a WAN link, I don’t know how you’re going to fit that into a 60/70k budget. The WAN link alone to facilitate the system being fast enough would blow your budget out of the water…

As far as what vendors are promising you… I’m sure you realize you should be taking all of that with a grain of salt. I also hope you’ve demanded an evaluation of units before even considering spending a dime on the product. If it does what they claim, they should have absolutely no issues giving you a trial run.