With the IPO completed, interest in Isilon among StorageMojo readers has been growing. So I thought I’d take a gander at their pricing and see how it stacks up. I’m fast tracking this project – doing the writing and analysis concurrently – so when you get to the end you’ll learn what I didn’t know at the start.
Using the handy StorageMojo Isilon Price List I put together an Isilon system using their top-of-the-line 6000 series nodes. Like most storage vendors, Isilon doesn’t actually provide the information required to configure their systems. I can see why IBM doesn’t, but a new vendor like Isilon should, since most of their customers are fairly knowledgeable and control freaks as well.
Just like Legos, only not as colorful
Building an entry-level Isilon cluster is pretty easy, given that I don’t know any better. You need three:
IQ6000iIQ 6000i InfiniBand platform nodes @ $21,411 = $64,233
and then you need one:
IQSwitch – Flextronics 24-port Infiniband switch @ $7,609
and some cables, let’s say 12 (I’m a little hazy on Infinband cables, (well, right NOW I’m a little hazy on everything, due to the post-prandial libations) but as I recall they are 2.5Gb each, so if you want, say, 10GB, you need four per node) [Update – a couple of alert readers assure me that the IB cables are 4x, so I’ve corrected the following calculations.] but who knows, maybe these are 4x cables and Isilon is just being coy:
5 Meter InfinBand cable @ $239 = $717
and, of course, what would hardware be without the noble leavening of software? A moldering hunk of inert metal, you say? So let’s add the “OneFS File System”, that, in words that would do Hopkinton proud:
OneFS® is Isilon’s patent-pending operating system software that provides the intelligence behind all Isilon® clustered storage systems. It combines the three layers of traditional storage architectures – file system, volume manager and RAID – into one unified software layer, creating a single intelligent file system that spans all nodes within a cluster. OneFS combines mission-critical reliability and high availability with state-of-the-art data protection to help storage administrators worry less and do more.
Call me crazy but doesn’t that sound a bit like ZFS? Naturally, despite my scepticism about architecture-based evaluation, I’d like to know how OneFS actually handles large numbers of small files, since it was built to handle large media files – the founders are from RealNetworks.
Of course the patent abstract (#7,146,524) is a little less breathless:
The intelligent distributed file system enables the storing of file data among a plurality of smart storage units which are accessed as a single file system. The intelligent distributed file system utilizes a metadata data structure to track and manage detailed information about each file, including, for example, the device and block locations of the file’s data blocks, to permit different levels of replication and/or redundancy within a single file system, to facilitate the change of redundancy parameters, to provide high-level protection for metadata, to replicate and move data in real-time, and to permit the creation of virtual hot spares among the smart storage units without the need to idle any single smart storage unit in the intelligent distributed file system.
So for three nodes we’d need three copies of OneFS:
OneFS 6000 platform software license for Isilon IQ 6000/6000i product (non-transferable) @ $16,376 = $49,128
So for a mere $49,128 + $717 +$7,609 +$64,233 = $121,687 you’ll have 18 TB of cluster storage. Just $6800 a TB!
The StorageMojo take
Isilon folks, feel free to comment to make any corrections. Yet somehow, this doesn’t feel like the answer to 1 PB, or even 100 TB storage. Why? Well, let’s compare to Sun’s X4500 (Thumper) that is about a quarter of that price. Granted, not clustered, nor as easily managed, yet, it just seems like for really massive data stores, the price should be closer to disk costs.
Comments welcome, as always. Moderation turned on to feed my megalomania. Or to keep spammers at bay.
Isolon solution is not that good price wise; basically I abandoned the idea of it because I made a sun solution running ZFS with 3TB of storage for $2700. Sure its not clustered but I can have 3 of these for less than the licence cost of one FS and cluster that!
Bargin!
Isilon is pretty pricy, but their value is that nobody else does it like they do. So let’s compare with NetApp. If I buy a NetApp 6070 for an application that eats a lot of spindles for IO but not for storage, let’s say it only needs 10T of space, but needs like 200 spindles to keep up with the workload. Do we price that on $/Gig? I think this works for Isilon. It’s an iop driven storage appliance not a typical $/gig storage appliance.
I think it helps to think about it that way.
I can’t really say I’d consider a Thumper to be a valid comparison to a real clustered system like Isilon appears to be. Without a way to ensure high availability the thumper is just another computer with a bunch of disks in it, which while being eminently useful for many things doesn’t really allow it to fill the same shoes as a system with no SPOF.
I would compare the Isilon system not against Thumper but the new Engenio (aka SGI aka IBM) SAS RAID system, which has redundant controllers and multiple host ports so you can attach redundant NAS heads. Then my questions would be which can scale more, and is there a price crossover point or performance crossover point between the two systems. (Due to the IB switch, there is an obvious Isilon sweet spot at 24 nodes.)
BTW, one node needs one IB cable; all IB cables are 4x these days.
Small correction:
The general Infiniband cable is now 4x so you need only 1 cable for 10GB.
All,
Thanks for the IB correction, I’ve updated the post and the pricing.
Isilon does offer an IQAccelerator-i that has no storage and enables the cluster to handle more IOPS. As near as I can tell Isilon offers no metrics for what this does, so while I’d *like* to look at this as an IOPS solution – good point, Blake – they don’t make it easy. Anyone have some numbers?
I suspect that Isilon is not mentioning performance because it isn’t a strong point, even with the accelerator product. At $12k (hw+sw), you can buy a lot of SATA drives and get added performance, more redundancy and capacity.
I’m still trying to get my head around the Isilon value prop, so bear with me as I do some more research. They may be the guys who come out of left field and kneecap the array vendors and/or NetApp. Or not. Anyone with experience with Isilon, please write. If you just want to give me feedback with having it published , just say so in your comment or send me email from my contact page.
Robin
Isilon is built specifically for rich media files. From what I have heard, OneFS uses a 96KB block size to store such data (lots of metadata, I guess). Compare that to a general purpose storage product (NetApp = 4KB block size; EMC = 8KB block size) and you can see that an Isilon is really good at streaming media but it pretty much sucks at storing lots of small files.
With regards to clustering, Solaris (nee Sun) Cluster 3.2 is available as a free download:
http://www.sun.com/software/solaris/cluster/
http://www.sun.com/download/products.xml?id=4581ab9e
And possible to use with multiple storage solutions:
http://www.sun.com/software/cluster/osp/
What’s even more interesting is that Sun is planning to incorporate the StorageTech Availability Suite into OpenSolaris some type this month:
http://www.opensolaris.org/jive/thread.jspa?messageID=78537#78686
So even if you don’t have a SAN, you can add AVS replication and do HA fail over.
Seems that a lot of what used to be only available through hardware is slowly becoming commodity via software (though you’ll have to pay for support of course).
We (Zycko) are Isilon’s disty in EMEA, so yes I am baised, but I thought I’d offer a few points…
The Isilon IQ solution has a distributed file system and parallel Gig-E connectivity, so for every node you have host connectivity, IOPs processing power and 4GBs of pre-emtive cache.
This is complimented by load balancing and out of band intra-cluster ops being shared amongst the nodes, so this is a truly linear solution with no master node or NAS head as in the more traditional NetApp world.
The key USPs for Isilon are,
Scalability, up to a PB in a single global namespace
Ease of deployment, takes longer to rack than it does to configure
It is as easy to manage a PB as it is 6TBs
There is no RAID management / configuration
Just select your protection level and go
Lets be clear, this technology is specifically targeted at digital media or large files of over 1MB. It offers performance of up to 10GBytes (thats bytes not bits) from a cluster, and with the IQ accelerator node or EX (expansion node) Isilon are able to exactly match the customers throughput requirements in addition to the useable storage capacity.
So these are the market verticals.
Media and Broadcast (BSKYB, HIT Entertainment, Channel 4 are all Isilon users)
Oil and Gas
Online Content Delivery (My Space is powered by Isilon, so is the Kodak easyshare gallery)
Bio Medical
High Performance Computing (grid)
+ a few others
There will be a growth in the market opportunity as more companies offer rich media for download on the internet, then there is the emergence of VOD and IPTV…
Isilon is a round peg, and it needs a round hole for a good fit.
Isilon has competition, but you haven’t named any of them so far.
I have capacity and performance calculators, I cannot share this with you but if you would like an example please mail me.
I will leave you with this.
From an industry analyst source, 52% of new data is digital media.
David, thanks for the update on Solaris. I agree that the secular trend is to move functionality out of hardware into software once processors become cheap enough. I think the same thing should be happening with storage arrays.
Esteban and Gavin,
Thanks for shedding some light on Isilon’s target markets. I went through their website looking for some guidance on that very point and found almost nothing. They seem to be positioning themselves as general purpose storage. All in all, I’m surprised at how little information they provide about what they do and where the biggest benefits are. It just seems weird.
Also. that 52% number feels bogus. What is digital media? Powerpoint? PDF? Flash? Do we assume that these files are mostly over 1MB? Most JPEGs aren’t. Is this home data or business data?
Anyway, I’m doing some more research on Isilon – expect to see more later today.
Robin
When we spoke to Ipsilon they only did file access. No block level access via FCP or iSCSI. That seems to be a drawback in my opinion.
File access is where companies need faster improvements than what many NAS boxes offer. Block level access is often used for Exchange or DB stores but what about your other data? Many of the storage vendors are doing fusion products with NAS/SAN features to get the complete picture here.
Isilon just announced their IQ 200 on Monday. Check that out for lower pricing. I do not think it can be mix and matched with EX nodes but it runs OneFS and should function like a peer in most cases.
I was told by an Isilon worker that they aren’t trying to replace SANs but rather augment them with faster storage for files that would normally be on a NAS I presume.
A few quick questions
– Does node price include drives or thats additional cost?
– Do I use single Infiniband cable from each node to connect to the Infi Switch? or 2 ports/cables per node for redundancy.
– also wouldn’t you need 2 switches for redundancy again?
– Also will have to add the Ethernet switch cost for client connection
Gavin can you help me calculate performance on this system? Basically if I have a few edit seats(FCP) in addition to just doing data storage what should be the sustained bandwidth I would expect from the system. To give you a little more idea the requirement is:
35 FCP seats doing DV25 (4audio) editing
40TB usable capacity
Thanks for your help.
You’re correct in that 3 nodes is the minimum cluster configuration. However, 3 X 6TB/system = 18TB of _raw_ storage. Once OneFS does its thing by striping data and parity across the cluster, you really only have about 12TB usable.
Performance is crazy, though – you can basically scale linearly according to the number of nodes in the cluster. They can all read/write in parallel.
Some of the other questions…
– Node price does include 12 X 500GB SATA drives
– Typically you can use one or two Infiniband connections. Only the first one will be used, though – the second is only there for failover. They cannot be used in a load-balancing or double-the-throughput way.
– In a full-up redundant configuration, both the front end (GigE) and back-end (Infiniband) would be dual-homed. The two GigEs in front, while going to separate switches, would probably still be on the same broadcast network.
Node “configuration” is ridiculously easy. It literally takes more time to rack the damn nodes than to add them to the cluster. Adding nodes is a matter of pressing the front panel button once – basically an acknowledgment of which cluster to add to – and takes 1-2 minutes, after which the other nodes auto-balance the data across all nodes.
Does anybody have an idea what kind of configuration (which type and how many nodes/accellerators) I would need to get 1500 MByte/sec read and 500 MByte/sec write throughput? Is this kind of throughput even possible using TCP/IP?
much obliged,
Wim
I am really interested in knowing what Isilon competitors there are out there. I see software solutions Like Ibris and others that claim to provide the same clustered services as Isilon, but do it on other vendor hardware. Are there any comparable vendors out there to compete with Isilon?
That being said, or asked… I think avoiding a completely proprietary solution is important to budget minded companies. Sure Kodak can spend $6800 a TB without even thinking about it, but what about budget conscience companies that need to stretch every dollar.
Having a software solution across Dell or HP servers or other NAS appliances allows a company to change direction. If the selected solutions is not working then the total investment is not lost, because you still have the servers available that can be used however you like. It is very hard to throw out a $120,000 investment in 3 devices that provide 12Tb of storage, for a budget constrained company. And once you are in, you are in. There is no changing the software or hardware when a single company is providing both, and the hardware is function specific.
The fact is that clustered storage technologies are going to replace servers eventually as other companies develop appliance products to compete using clustered technology. It seems to me that Isilon has chosen to tap into the “get the most cash I can as quick as I can from the biggest companies†type of marketing mentality, as apposed to a “provide an affordable solution to as many companies as I can, while I am the leader†marketing philosophy. Is there anyone out there who provides the high availability, extremely fault tolerant solutions for the non-Kodaks?
These industrious and innovative companies should make money on their storage solutions, but I would prefer to buy three $10k servers (inflated) that have 6Tb of storage of cheap SAS, and then add the needed functionality. If I did, then I would have $90,000 left over to pay for the software and management overhead. I just can’t see that type of margin being acceptable for the average company that has storage and performance issues.
As a budget company needing to high throughput for millions of very small files, is there any recommendations that leverage fault-tolerance, performance, and ease of maintenance. I think if it takes a little bit of work to manage the system, we ca deal with that for $90k.
I’m not a Isilon vendor, but considering the price of the actual hardware their using, the price you have listed for the IQ6000 would make since if it was for hardware AND software. Can we confirm that you have to buy both the hardware and the software for each head? Or are the OneFS pricing only for purchasing on hardware that has not been under a maintainance agreement?
Just an idea, because you can get 500GB SATA-II for under $200 now. Since they’re not using RAID, the system could easily have hardware for $5000 which would be the addition of OneFS for that price.
Anyone have any more details?
-SMFX
Exanet is a software-based clustered NAS solution that competes heads to head with Isilon.
http://www.exanet.com.
thx all, this is all great info to read.
We’re an aerial photo company (we make stuff like on Google Earth) and are in the closing stages of investing in an internal storage + processing centre.
The 3 parties that will hand a quotation in are HP, Dell and ONStor and they’l do so this week. So coming across a name like Isilon now is great an unfortunate at the same time..
The vendors all need to come up with a solution that
1) now has to handle max 320 MB/s and max 75K I/O’s per sec, but can (easily) scale to more than that
2) now has to have 15TB of nett fast storage (eg SAS 15k rpm drives) and around 45TB of nett slow storage (eg 7.2k rpm SATA-II 750GB drives), but also can scale to much higher levels
3) has to be presented to the clients (people and processing nodes) as one simple and single cluster (or namespace or some other fancy word for that).
Anyway, HP’s proposal will probably look like this:
– 2 DL380 G5 clustered servers serving as gateways
– an EVA6000 system with the SAS storage and an MSA1500 for the SATA-II storage
– PolyServe software (now inhouse HP called EFS) to enable the clustering now and for later
HP will be able to scale performance and capacity with EFS
The offer will probably be between 200-250K EUR
Dell’s offer will be something like this:
– 2 NX1950 servers in a cluster
– a Dell-EMC CX3-40 with both SAS and SATA capacity
– for the clustering and namespace stuff they need to bring in outside software co IBrix from the US as they cannot scale above the config above without breaking the single namespace/cluster req.
My guess is they know what HP can do and have to win on pricing as ther config is less appealing because of multi vendor. I think they’ll come in between 150 and 220K EUR
Then ONStor:
– 2 bobcat 2260 gateways
– in their pantera clustered system with SAS and SATA
– the bobcats come with their smart software to enable clustering now and later
– they do not provide FC switches, UPS-es or racks
My guess is they’ll come in at 250-350K
Now the big question is whether I should seriously think about Isilon here. My gut feeling is yes, although they apparent;y are a really new player in the market.
But they offer a very appealing solution and I have an idea of the price. The only thing I cannot find on their site is the throughput and IOPS so I can figure out what;s needed.
Given the pricing on this site (thank you mr s-mojo!) this is what I think the Isilon config would look like:
– 3 x IQ6000 + 3 x software > 64K + 50K = 114K USD
– 2 x switch > 19K USD
– 10 x EX6000 > 130K USD
and then some support, cables, installation/training and other stuf for 20K USD
Adding accelerator nodes is uncertain as I don’t know the capablities within above config.
This would total about 270K USD of just over 200K EUR at today’s rate.
Now two questions for you pros our there:
– is this quick&dirty calc correct or not?
– should we seriously consider this as opposd to the other 3?
I would be much obliged if anyone can shed some more light on this new development for me!
Cheers,
Ezra