The 2nd installment of an occasional feature . . .
A reader writes:
I think your input would be valuable in helping me make a decision on storage for my company. I’ve done loads of research and I’m fairly certain I have good players narrowed down, but have reservations about both. . . .
Players:
-Datacore SANMelody H/A solution on HP hardware.
-Equallogic PS3800XVThe app
It’s is an up-to-the-minute commercial application supporting virtual machines. The VM’s run proprietary messaging/transactional servers that spend 99% of their disk I/O time appending very small messages – ~300 bytes – to transaction logs.
Update: After the initial comments, the prefers-to-remain-anonymous reader (BTW, I did check him out and his company is for real) added this clarification:
- Yes, there are DR and HA requirements.
- Each VM has its own transaction logs that can grow to GBs in size. These transaction logs are not for archival purposes, rather to recover state in the event of an application restart
- Traffic: Traffic will come in bursts and maximize at about 1500 iops between 10 separate hosts.
- Reservations: Is Equallogic a “true†H/A solution considering it does not support synchronous replication between completely separate hardware? Are the competitors claims of Datacore’s “unprotected cache†well-founded? (Datacore insists in H/A mode that all cache is synchronously written and requires a commit from its H/A partner before committing to client.)
- Storage size requirements are small, so I’ll pay for SAS performance over SATA terabytes.
End update.
Update II: The anonymous reader comes back with more crucial detail:
Let’s pretend the budget is around $60k-$70k. I know the two finalists can provide an acceptable degree of HA, DR, and iSCSI performance at that price. What products should one be looking at from HDS/EMC/NetApp? They were not considered initially for the perception of being unaffordable.
End update II.
Update III:
The plan is for an H/A setup in a class 1 datacenter with asynchronous replication over an existing DS3 (..but dark fiber is in the works) to a remote site.
All things considered, the question could be framed, “Whom/What should be demanded for trial?â€
End update III.
The StorageMojo take
It is interesting that this customer is NOT looking at the traditional OLTP storage vendors. This is a business-critical application – the company is handling Other People’s Money.
What are the questions the reader should be asking of vendors? How should the problem be framed? I surmise that price is an issue. Where else might the reader go?
I welcome comment from vendors, but please do us the courtesy of identifying yourself as such.
Comments welcome, of course.
I believe Dell has discontinued the 3800 and replaced it with the PS5000,which is their default platform now. One assumes he’s thinking 15k SAS drives??
Zilla: The beast who demolished EMC.
Were I buying the question I’d ask is how both systems deal with drive data integrity issues, Silent Bit Errors and all that jazz. I know how the Symm does it and Steve Todd has told us how the Clariion does it but I have *no* idea how these two do it.
I’ll admit I hadn’t thought about it until I read this but I’d want to know if I were the one flying the box.
Anyone?
WE looked at EqualLogic, but chose LeftHand Networks instead, based on price and features. Theirs is really a software-only solution running on commodit boxes (including HPs if you want them). THere is even a VMware version of the LeftHand software.
That said, be VERY skeptical of vendor performance claims. Get evaluation units, and make sure you test with a realistic workload (multiple clients, correct RAID levels, snapshots, etc.) EqualLogic’s boxes are very fast, faster than LeftHand gear in the same class, but the scalability model is very different. And at $60K per box, the “jumps” for incremental scaling are larger.
Marc Farley from Dell/EqualLogic here. Don’t assume the 15k drives are needed. You might be able to use 10K drives or even 7200 SATA drives. The disk performance on our systems tends to surprise most people. Do you have a way to setup a somewhat realistic test environment? If so, one of our VARs can probably get you a demo system to test your applications.
I work for a multi-vendor VAR in Czech Republic.
It would help if you told us on what hardware is the app running currently. How many disks of which type. How many IO operations per second does it get. The traffic pattern you described suggests usage of SAS drives.
With LeftHand running on HP320 or Dell2950 nodes with SAS drives you can be sure you get a competitive performance with true high-availability. Also, when the performance requirements increases, you can simply add another node to satisfy it. This works, really. When any node fails, the cluster survives transparently with no need to run any OS-level scripting. Simplicity is another feature to vote for LeftHand.
We also looked at equallogic and lefthand but chose datacore. For us the reason was performance (especially in suporting VMs) and the fact that it was a portable software solution. In terms of perfomance what we found with dataCore is that it does a great job caching, if you get yourself a decent Dell or HP server with some horsepower, allocate some memory for caching and you will see the performance and it far outpaced the equallogic we tested.
Some questions to ask?
What features are included, what options are needed (watch out for being ‘nickle and dimed’ along the way)?
What is the upgrade path, when you need to add more storage,etc – whats the policy, impact and cost?
Anticipate the unexpected? How do you handle growth?(We had a merger, now I have a mix of FC and iSCSI, since Datacore runs over both it was no issue).
I have to correct myself – if the app is just appending the log files sequentially and not seeking anywhere, then you might consider SATA drives. It has no impact on what I’ve wrote about LeftHand, just replace SAS with SATA.
Sorry, I didn’t read the definition carefully the first time.
As always there is nothing like having loaned kit from a vendor to see for yourself.
From your description “up-to-the minute” I was not sure if this implied high updates or updates every 60 seconds . If high updates then you will want to know the latency of updates from each vendors solution.
Are you updating the same transaction log from each VM or does each VM have a seperate transaction log.
How big does this transaction log grow ? Some applications will read the entire log to append to a file.
300 bytes is very small update but multiple this by number of updates per second and then by the number of VM’s will give you an idea of overall system I/O.
What is the critically of this system and its availability requirement ? Is there a DR requirement if so what is the SLA. How do you intend to backup an active transaction log and what is the restore requirement.
What were your reservations on your selections ?
Call your VAR and get the vendors kit on site – this is always the best test as every vendors solution has it issues and has to be tailored to your environment.
DataCore must cache writes on the appliance using unprotected DRAM to achieve acceptable performance. This is very dangerous. If the appliance fails, all data on the underlying storage system can be lost. Performance without this caching mechanism running on the appliance is poor. O/S reboot or motherboard failure can cause complete data loss. The data is also exposed to the kernel and all drivers. If there are any kernel or driver errors which muck with that memory on system power on, they have just introduced undetectable data corruption. Keep in mind that DataCore runs on Windows, need I say more?
Some other factors:
LeftHand’s Virtual SAN Appliance is certified as a VMware Virtual Appliance and is also SAN HCL certified. DataCore does not have either of these certifications.
LeftHand is the only SAN vendor that has VMware SAN HCL certification for a Virtual SAN Appliance.
DataCore’s proprietary drivers are not signed by Microsoft.
DataCore Alternate Pathing for Windows is not qualified for iSCSI.
DataCore is only Microsoft certified as a server software package, not an iSCSI storage solution.
No Microsoft DataCore+Hardware or MPIO certifications!
If you want great performance with advanced data protection (survive double disk faults in a RAID set, array/node faults and site faults without losing access to your data) then LeftHand is a great choice, being bias of course. Just so I’m not completely labeled for vendor bashing, I’ll have to admit EqualLogic is also a suitable choice with great performance as well, although they do not have a software only (virtual appliance) option.
Frank from Fujitsu. What is the main purchasing criteria going to be? Assuming it’s not price, then it’s performance….? Scalability? Support? Ease of use for software? Flexibility? If you have limited resources already from a head count point of view….buying the worlds’ greatest technology is not going to do you any good if you don’t have a solid IT Vendor/VAR behind you.
Performance is a real hard thing to judge even with SPC, because of varying technology, components, and architecture in each vendor..
I’d recommend a “bake-off” to bring in units and have a set of measurable benchmarks the units have to hit before committing yourself, which gives you some flexibility….if a vendors’ performance claims are not up to snuff for your application.
I posed this question. For clarification:
-Yes, there are DR and HA requirements.
-Each VM has its own transaction logs that can grow to GBs in size. These transaction logs are not for archival purposes, rather to recover state in the event of an application restart.
-Traffic: Traffic will come in bursts and maximize at about 1500 iops between 10 separate hosts.
-Reservations: Is Equallogic a “true” H/A solution considering it does not support synchronous replication between completely separate hardware? Are the competitors claims of Datacore’s “unprotected cache” well-founded? (Datacore insists in H/A mode that all cache is synchronously written and requires a commit from its H/A partner before committing to client.)
-Storage size requirements are small, so I’ll pay for SAS performance over SATA terabytes.
Thank you for all the great input!!
It’s hard to match LeftHand’s DR capabilities. For example, we can synchronously replicate to 3 sites and have 2 sites go down without loosing access to data, and bring any site backup up and incrementally re-sync it – all with fully automated VM failover and failback. Or even something as simple as async to a remote site over a slow link, also with failover, failback and incremental re-sync. Nobody in the industry can touch this. Sorry, probably overselling at this point, but check it out for yourself.
DataCore Synchronous Mirroring does not have quorum protection, which can result in a “split brain†(data is written to both copies while the link between DataCore Servers is down.) When the link is back up there is no way to determine which copy of the data is valid. This can cause total data corruption. And mirroring corrupted cache doesn’t work either.
Customer example: http://communities.vmware.com/blogs/pauliew1978/2007/11/13/split-brain-in-an-esx-and-sanmelody-environment
No, EqualLogic does not support synchronous replication between completely separate hardware. When an EQL box is down your SAN is down. Their boxes are rock solid, although no RAID 6.
1500 random disk IOPS can easily be achieved with a single 12 disk SAS node. Two nodes gives you 4000 random disk IOPS and node level H/A.
So… the obvious question anonymous:
Why is it HDS/EMC/NetApp haven’t made your list?
It seems of all the choices you have, your two finalists are the WORST fit for your requirements…
TimC,
Let’s pretend the budget is around $60k-$70k. I know the two finalists can provide an acceptable degree of HA, DR, and iSCSI performance at that price. What products should one be looking at from HDS/EMC/NetApp? They were not considered initially for the perception of being unaffordable.
What’s your RTO/RPO? What is the distance requirement between the two nodes?
I guess if you need true synchronous replication across a WAN link, I don’t know how you’re going to fit that into a 60/70k budget. The WAN link alone to facilitate the system being fast enough would blow your budget out of the water…
As far as what vendors are promising you… I’m sure you realize you should be taking all of that with a grain of salt. I also hope you’ve demanded an evaluation of units before even considering spending a dime on the product. If it does what they claim, they should have absolutely no issues giving you a trial run.
TimC,
The plan is for an H/A setup in a class 1 datacenter with asynchronous replication over an existing DS3(..but dark fiber is in the works) to a remote site.
All things considered, the question could be framed, “Whom/What should be demanded for trial?”
Give us a call at 1.866.4.IPSANS. We can offer you a synchronously replicated, 2 site, fault tolerant, high performance SAN starting in that price range, assuming you have the pipes in place. You could actually start out with async. and convert over to sync. using the same HW config, on-the-fly with no down time. Let us know – we have demo gear in the field that we can bring in to demonstrate this.
To be fair, I thought I’d post excerpts from DataCore’s response to the above criticisms:
(DataCore answer – Split Brain)
We do Synchronous Mirroring with Cache Coherence. We NEVER return an acknowledgement for a write UNLESS we have the write in TWO caches, one on each SANmelody server. We implement the full SCSI-3 reservation system – both soft and hard reservations. If the two SANmelody servers mirror channel or communication link fails, whoever holds the reservation on a virtual volume will keep the reservation, the SANmelody server will enter into “Write Through†mode for any of the mirrored volumes it shares with the partner, it will also flush out any outstanding cached writes, and then it will start a bit-map journal to keep track of the blocks that have changed so that it can rapidly recover the mirrored volumes once its partner becomes available again. Architected correctly, ie. two SanMelody nodes with HA pointing to arrays on the same SAN fabric, you don’t have an issue.
(DataCore answer – Unprotected Cache)
DataCore recommends a dual-SANmelody configuration where you have.. complete redundancy… The two SANmelody partners are active / active controllers, implement the full SCSI-3 recommendation, implement write cache mirroring between the two nodes, implement Auto-Failover AND Auto-Failback between the nodes.
(DataCore answer – Unsigned drivers)
Many drivers are not “signed†by Microsoft… if there were a problem with our drivers, we would know about it after over 10 years doing this… None of our thousands of customers have raised any concerns.
(DataCore answer – Alternate Pathing not qualified for iSCSI)
MPIO is the direction the industry is headed. We’re selling our Microsoft-foundation MPIO driver for Windows. The reason we do this is our MPIO driver supports both Auto-Failover AND Auto-Failback between the nodes.
Good thread, except for the lefthanded sales pitch (a bit desperate sounding) and vendor trashing (peaked my interest in Datacore so I’ll download a copy).
Can we get back to the original question, I am also evaluating Equallogic to support VMware.
You should start all over again and look at FalconStor. Does everything you want and tons more, plus it’s VMware certified and has VMware agents to get transactionally consistent data images rather than crash-consistent. It will also save you a ton on replication bandwidth because they do it at a sector level.
So, Steven here…currently working at Dell, but came from Equallogic. So there are a few things being discussed here. Firstly, I have to call out John S. from Lefthand on a FUD penalty. Equallogic does ont lose the SAN if a single ARRAY fails. It may lose certain volumes that are stripped across multiple arrays, however, this is an optional deployment configuration, and volumes can be isolated to a single array.
Secondly, while Datacore and Lefthand have nice features like Mirroring, they have it because they require it. An off-the-shelf server has many single points of failure. Lefthand overcomes this by introducing Netowork RAID on top of Hardware RAID. While this has a HUGE capacity impact in a 2 node configuration (which is a new configuration support for LeftHand) and has less capacity impact as you scale out storage nodes.
Equallogic handles HA in a more formal Enterprise fashion. The hardware is purpose built specifically for iSCSI traffic, which has helped make the PS-Series arrays the market leader in iSCSI performance. The hardware is also fully redundant, battery backed mirrored cache within the array, so Writes are Protected.
The issues around Datacore and FalconStor are similar, traditionally you put someone else’s storage behind those solutions, and have to manage each distinctively.
Go to http://www.dell.com/equallogic and sign up for a Webinar. You’ll see the ease of management, the ease of set-up and deployment. If that doesn’t convince you, Evaluate an Equallogic PS-series array, I can help you get a unit without a problem.
One last statement, the Lefthand VSA, publically states that there is about a 40% performance overhead, so not sure why you would look at that option. Lefthand does however make a good iSCSI cluster solution. I have enjoyed competing against them in many accounts.
I welcome you to call Equallogic references, we have MANY.
Also, about the DELL | Equallogic product line. We have several array options that all intermix as you scale. Everything from SATA, to 10K SAS, to 15K SAS. The line is now called in general the PS5000 series.
In all fairness your setup is basic and any of the products being pitched would work fine for you. Many other customers have gone down this path and defined their own requirements and that is where you should focus; defining your requirements on paper. We have a template requirements document that is generic enough to include all of the mentioned vendors and would be a good starting point for putting down on paper what is important to you. It is simply a bulleted list of requirements that have been requested over the past couple of years from dozens of customers facing the same challenge you are. I do not want to post a link to it but if you contact me I will be happy to send you a copy to do with what you want. I also formally recuse myself from participating so you can objectively work up your own requirements. Reach me at daleu (at) nobulkemail.com.
Steven, I don’t like marketing style sentences written to screw facts of other vendors to hide own product weakness. The important clear message to a man going to choose a new SAN for H/A is Equallogic has NO high availability feature to survive array’s (or site’s) failure transparently.
Synchronous mirroring is not something to be called “nice feature”, in my opinion it is a requirement to start talking about the SAN system as of “Enterprise”. People don’t buy VMware HA since it is a “nice feature”, they buy it to survive a site failure. Similar it is with storage. Synchronous mirroring means nobody has to do anything when a box or site fails. Asynchronous replications means possible data loss and some work to recover from a failure. Still better than no redundancy, of course.
The other fact – you claimed the PS-series to be a market leader in performance. Can you explain it? Can serve 900-1000 MBps sequential iSCSI traffic out of a single SATA box like iStor arrays do? I don’t believe so, perhaps you meant other sort of performance. If so, then make it clear, please.
Well, let’s keep this blog rolling.
Thanks for your comments Lukas, I totally agree. This poor customer is going to be fed up with all of us by the time we’re done here. Not trying to sound desperate, just full of passion for our products and never ending pursuit of the truth.
DataCore: The fact that a DataCore customer actually experienced a split brain is proof enough in my mind. How does either box know the difference between a box failing or the link going down? What if the failed box doesn’t release its reservations? More on this later.
Steven, I guess you didn’t notice. I was actually trying to be nice to EqualLogic for once.
First of all, a key benefit of clustering and virtualization is providing the capability to span volumes across multiple boxes (as EQL’s literature suggests.) Like LeftHand, as volumes are configured from spanning the disks on one EQL box to two boxes, the performance of those volumes increase because you have more disks working for you.
Regarding your comment: “It may lose certain volumes that are striped across multiple arrays, however, this is an optional deployment configuration, and volumes can be isolated to a single array.â€
I interpret this to mean: If you’re worried about losing access to your data when a box fails, don’t let your volumes span multiple boxes. Let me think… performance is not optimal… and all volumes on any box that fails are still offline. I don’t get the argument.
As far as the capacity hit goes, yes there is a capacity tradeoff for data redundancy, and at least we give the customer a choice on a per volume basis, unlike EqualLogic. While were on the subject of “HUGE capacity impact.†Let’s take a closer look at the EQL data protection and capacity equation:
– EQL thin provisioning still requires that you reserve 10% of your volumes up front, and also requires reserves for snapshots and remote replicas (LeftHand’s thin provisioning is “reservationless across the board.) Add this to the hit for RAID 50, hot spares, and EQL software overhead and you end up with 46.9% storage utilization, without redundancy across boxes!
-Yes, and let’s not forget RAID 6. Why is Dell/EQL the only major storage vendor in the industry that doesn’t support RAID 6, or some means of protecting data against double disk faults in a RAID set? When you go to re-build a failed drive in their 7 drive RAID set onto a hot spare with 1TB SATA drives, you have a 51.5% probability of a BER event that could cause you to lose all your data (assumes 10^14 BER.) Even with Enterprise Class SATA drives (no data to say they use these) with a 10^15 BER, you still have more than a 5% probability of a BER event during a RAID rebuild.
If you don’t believe the story just ask EMC, HDS, HP, NetApp, and just about everyone else you can think of, why they have RAID 6 double disk fault protection.
We are happy to provide references as well from over 3,000 LeftHand customers.
BTW, no hard feelings. I love you guys. The drinks are on me if we ever meet face-to-face.
John, did you even read the thread and follow the links you posted on split brain?? Gimme a break, your comments are real lefthanded (pun intended). Anyway after reading and checking the issue out it was a basic config set up and VMware issue not Datacore but the facts would probably just annoy you, I can tell you like throwing around fear, uncertainty and doubt as your sales tactic.
Bte, I’ve downloaded a copy of sanmelody and am in the process of running results compared to iSCSI boxes and the Equallogic, using IOmeter runs so far I am seeing about 4 to 1 better results with Datacore sanmelody. I’ll follow-up as I get more data and do some runs.
Ok, label me a FUD instigator. I’m honestly trying to do nothing more than educate customers on technology facts. If my facts are wrong, I’ll be the first to admit it. Yes, I did read it. The customer clearly states:
“ALWAYS MAKE SURE YOUR ISCSI PATHS FROM ESX TO SANMELODY POINT TO THE SAME SAN BECAUSE IF THEY DON’T AND YOUR REPLICATION LINKS BREAK YOU ARE GOING TO GET A SPLIT BRAIN SCENARIO.â€
When using VMware HA or SRM you’re not just concerned about failing over your storage controllers e.g. DataCore servers on the same SAN, but failing over your servers on to completely different physical storage from one building to another, which is essentially what this customer was modeling. If you’re replicating data instead of sharing it, disk reserve operations are of no help at all – since reserving a disk on one disk volume has no effect whatsoever on the other volume.
2-node solutions are always (this is mathematically proven…) susceptible to at least one of two problems:
* A Safety problem (aka loss of data integrity, conflicting writes, etc.). This may happen because of a simple case of what people normally call a Split Brain, although there are more complex variants. Essentially, the problem boils down to in some cases allowing 2 nodes to write different data, some of which is lost or corrupted.
* A Liveness problem. The system is not HA, in some cases where one node fails the other node does not take over.
In the real world systems are more often susceptible to Safety problems than Liveness problems. I believe this is because customers are much more likely to notice a system being taken offline when only one node failed, than a system that allows some data to be lost and/or corrupted (which usually doesn’t happen and when is does is often not noticed…).
Based on DataCore’s description, it would sound like the system is not really HA, because only the node that holds the reservation will continue in the presence of a failure of one of the nodes. In a system with load-balanced volumes, this would suggest that 1/2 the volumes go offline when one of two SanMelody servers go down in a mirrored configuration. It also appears that if you snip the cables between two nodes that both of them continue operating… causing a Safety problem. Again, this is because customers would notice lack of Liveness, but normally not notice loss of Safety.
Aside from this, I just can’t imagine a customer ever purchasing a SAN that isn’t supported by Microsoft, but hey, live and learn.
John
It seems to me that the user should be looking at host-based replication versus storage-based replication – since the apps are running in VMs I would go with InMage.net’s DRScout and just use an iSCSI storage array for data storage (Promise technology has a nice, inexpensive option in the vtm610i). With a high transaction volume, the host-based replication option can do real-time transactional replication and initiate host failover very rapidly. Another option would be CA’s XOSoft, but with InMage.net’s offload engine the VMs suffer much less of a performance hit.
Hi,
I am the person who had the split brain problem. Datacore have recommendations for stopping this kind of thing i.e redundant links between datacore servers. I myself can highly recommend datacore and sanmelody. It is a very versatile solution, datacores support is second to none and the cachng is excellent. I have had 4 power outages (2 full campus wide and 2 building specific including the buildings with the sanmelody servers in). I can report I have never lost any data and my virtual machines have always restarted perfectly.
I never completely understood what was going to happen when I realised I was in a split brain situation in terms of getting the replication link back up again. I was not prepared to risk it so I used the vmware convertor to move the test vms from one esx server to the other (thus avoiding the replication part of the network). This is a secure method to resolving the split brain. I also took full backups using express before making the replication link live again.
I am not a San technician so you will have to excuse my lamens terms. But it is my thinking that If I had made the replication link come online again it should have synced everything over ok. The reason behind this is that only one virtual machine can be running on one san at anyone time. This means that the vms (although they were writing independantly) were only writing to their paritions on the disks and therfore changes could have been replicated over without any problems. Because essentially it could have been disasterous I took a different route to ensure everything was ok.
As I stated in my post on the vmware forum I have made my environment only ever point to one san for live vm’s. This is not a problem for me as it gives me control over the iscsi paths for my vm’s. I can turn off one san during the day if need be and my vms iscsi paths will failover perfectly. I can vmotion and turn off my esx servers. I have HA ability as well so all in all I can’t ask for more for the price. It was certainly an intersting project and I tried to follow best practises as best I could but as far as I was aware there was no documentation anywhere on the project i undertook. These sorts of implementations are usually done by consultants and I am a lowly sys admin with a love of technology. personally I think you have to be alittle mad to try and do it yourself but it is worth it.
The only sad thing is no-one actually understands the task I undertook…….
DataCore VS Lefthand, EMC, NetApp
Ok. I am bias of Datacore for very good reasons; as we are their largest channel partner in North America.
There are a few good reasons to stay away from other storage solutions
All vendors require the purchase of their appliance and/or the software locks you in to a specific mac address so there is no portability of the product whatsoever.
DataCore has none of that; you can install the software on “any” storage that Windows can see period. You can install DataCore behind every single hardware manufacturer on the planet and DataCore will make it all look like one big array.
Oh yeah FYI EMC runs on Windows guys; and enough people like their products.
DataCore is now HCL certified!!! Official announcement coming in September.
DataCore outperforms all other virtual storage solutions software or hardware based. (see link under VMTN performance in download section of http://www.fairwayconsulting.com under DataCore)
DataCore allows for 100% investment protection when upgrading or adding features. The other vendor make you buy it all over again.
The other vendors will require continued purchases at “end of life” which is extrememly expensive and rigid on your planning for the future. DataCore allows you to purchase any hardware you choose and control the number of TB you need; not force you into buying a rack with a set number of TB that you don’t need.
I could go on but it’s unfair as there is no product that you can compare oranges to oranges on.
Sorry guys this is a no brainer…
One bit of advice though; please know you facts before spouting off about DataCore’s product line.
Oh for you guys that can’t get enough ahead of the technology curve; check out DataCore’s “Traveler” piece it’s like Tivo for your data.