Amazon and Google have demonstrated commercially – as have a number of research projects such as Microsoft’s Boxwood – it is possible to build highly available storage out of commodity servers. The odd thing is that, AFAIK, there is no commercially supported open source software for people to try.
Hadoop hews too close to the Google model for most commercial applications. Lustre has a lot of great technology and the management complexity to back it up.
Where are the VC’s?
There are a number of venture-backed OSS companies. XENsource in virtualization. Vyatta in routing. But nothing in storage. I can’t figure out why.
Is there no demand?
I’m trying to think through the issues. I see a few possibilities:
- No VC believes OSS storage clusters are usable in commercial operations.
- No VC believes glass house IT will use them.
- VCs believe the big guys, like Sun, will own the market.
- No VC is smart enough to see the market. Storage arrays are only about $30 billion a year. Who could build a business on that?
None of those explanations are very satisfying. What do you think?
The StorageMojo take
It’s a mystery. OSS + commodity hardware = new value prop. A company devoted to developing and supporting that software would seem to be a natural. And yet – no such company.
Comments welcome, especially on this topic. What would it take for your company to use OSS storage? Where would you use it? Would you buy support for it?
Well there is a few companies in this space but they are just starting to appear and go out of stealth mode.
While not VC backed, I met the guys who do GlusterFS a few weeks ago. I played with the product and while it’s not nearly “ready” it seems like it is very much on the right track — esp the idea of using FUSE to build on existing filesystems. Has anyone else had experience with GlusterFS?
I think part of the problem is that OSS, in general, does not innovate, but recreates. So far, the big successes in OSS have been Unix-like operating systems, webservers, etc. But all of these kinds of things existed. The only really new thing brought forth by OSS that comes to mind is Beowulf.
I’d love to be wrong here, but it seems like ideas start off as closed-source, and then get recreated as open source (sometimes better, as in the case of OpenSSH).
I know a couple of these guys “socially” (if that’s what you call talking on the internet and meeting one for breakfast one time) but that’s the extent of my involvement with them.
http://allmydata.org/trac/tahoe
The Allmydata “Tahoe” project is a secure, distributed, fault-tolerant storage grid, available under a free-software licence.
You can store files in the grid and retrieve them with a cryptographically generated URI that securely identifies and decrypts that file. You can also organize files with filenames and directories, in the familiar style. The basic idea is that the data in this storage grid is spread over all participating nodes, using an algorithm that can recover the data even if a majority of the nodes are no longer available.
There is a company doing just this commercially. It’s called XIV and they are Israeli.
People and corporations already use OSS storage copiously – in small, turnkey (e.g., SNAP-Server-like) installations. A major problem with using it in large installations is lack of turnkey-style support – especially when they move into the realm where significant networking complexity (e.g., to ensure solid multi-pathing) is involved: they then are faced with using hardware which they often must quality themselves, software which is supported only on a ‘community best-effort’ basis (unless they want to develop their own in-house expertise or can find someone willing to take on that role commercially), and then configuring and qualifying the distributed system as a whole.
Or they can just buy a canned, proprietary solution from someone happy to do all those things for them. For most non-HPC institutions, that’s a no-brainer – especially given that even using an OSS platform they’d still usually need to pay someone to support it (and quite likely to configure it as well).
As I’ve noted elsewhere I believe that it’s possible to design distributed storage environments that need no more (non-network-related) configuration (actually, *less* configuration in some areas) than ZFS does. This could help reduce the need for storage admins (though they’d have an obvious personal conflict of interest in supporting such a platform), but would still leave the problems of base hardware qualification and network configuration to deal with – plus the continuing question of whether one wished to trust their corporate crown jewels to OSS. A company like Dell could handle the first two of these issues, but it would take either the calibre of the IBM service organization or really mature, stable software to handle the last.
– bill
What about Cleversafe, http://www.cleversafe.org ? They just recently rewrote their whole dispersed storage grid, dsNet, from the ground up. They’re an open-source project sponsored by a commercial company. I haven’t had a chance yet to try out the new code yet, but it looks promising.
The one open-source-based storage company I can think of off the top of my head is agami – it’s got linux inside.
We provide OpenFiler-based storage systems in India built up on commodity components. Please see the details at http://www.mettle.in/products/stor.html
These systems are deployed in configurations such as mail storage systems with service providers, file storage systems (both NFS and block export over iSCSI) and even as database storage (over iSCSI).
With every installation we busted the myth that only “enterprise” storage systems can meet their demands. And it’s going great.
OSS dss is in a “wait and see” period. OSS in general has had a special charm for SMBs simply because of its cost savings value propositions. Now IMHO, the key to OSS for storage growth and wide adoption is to be appealing to those SMBs as a simple, easily manageable solution (did not say cheap, because that is implicit assumption). So turn-key installations might help in that regards, in addition to a cheap support model.
Another thing to consider is the emerging SaaS paradigm. Most of the SaaS companies in business today (besides sales force) are finding difficult to make profit out the door or even years afterwards. The reason be it the cost of the supporting infrastructure which storage is a key part of. I.e. you can not have a online backup business built on top of expensive proprietary disk arrays. it just does not cut it at the bottom line. so the solution for these ventures is either to develop their own proprietary software for the storage grid, and it has to be on top of commodity hardware, or buy an already established solution, (EMC buys MOZY).
I think the real determining factor is the market dynamics and how it plays in favor of OSS. You have SAAS businesses that mostly targets SMBs and OSS that mostly appeal to SMBs.
I think SAAS will win the deal as bigger and larger companies move into this business model, and I don’t see any of them being generous with sharing the secret storage recipes with the rest of us. Thus the OSS on commodity storage will most probably end up, and should focus on the HPC market that is mostly established in labs around universities and research centers. I think VCs are getting this and that is what’s behind the hesitation in jumping in.
PJ,
I don’t believe that Agami is OSS… can you substentiate this?
Hi Robin:
A few come to mind on the software side of the equation. The OpenFiler project from Xinit systems in the UK. This is a very good project in a number of regards. We use this as a default appliance OS on our JackRabbit units. Then there is FreeNAS which is a BSD based system supporting similar things.
Both of these will provide NAS and iSCSI. With iSCSI, you can use things like GFS and other clustered file systems to provide a single namespace across multiple block storage pools, as well as redundant arrays of inexpensive nodes (RAIN) configurations. With the forthcoming pNFS and *overRDMA technologies (some already here in a number of projects), you can build very fast, very scalable, very cost effective solutions based upon open source technologies.
Those are the technological projects. The business issues are another thing entirely.
What is interesting is that in discussions with potential customers, we hear the same “what if they company gets hit by a bus” arguments used against small companies. Given what open source is, our response is quite simply, look at the hardware/software you have, and tag it with the words “expensive brick” if the company that markets it goes away, and you can’t get any support anywhere else for any price. Now try that with the open source versions.
Any completely proprietary solution is a brick in the face of uncontrollable external business events (competition, product churn, bankruptcy/acquisition). These uncontrollable business events represent risk, and in some cases a clear and present risk, upon which you rest your companies data.
I think the question should be, if you are not using OSS based storage, why are you risking all your data on a completely proprietary system that could become a very expensive set of bricks, through events that you have no control over?
Look at the NTAP vs SUNW kerfluffle. What if SUNW gets its permanent injunction? Any NTAPers out there worried about this? Does this represent business risk? (FWIW: I think NTAP is a good company, I am just pointing out the obvious in case of a runaway process that goes badly against them). Yes it does increase your risk.
Then again, we are biased, because in full disclosure, we design/build/sell high performance storage systems which use OSS as much as possible. Our belief is that OSS lowers the purchasers risks.
Joe
At GrokThis.net we use AoE to run our SAN, which has commercial backing of Coraid — they designed AoE as an open source solution. For networking, we use inexpensive gigabit switches. We then put Linux LVM on top of that, although we’re considering ZFS for future deployments. Another option for OpenSource storage is to use OpenSolaris with direct-attached storage with ZFS and iSCSI.
This is all open source. In fact, open source has a whole bunch of storage solutions, perhaps with more variety and certainly a lower cost — hardware-wise, not just software-wise.
There are a number of OSS storage companies that carter to the SMB market; mostly competing with Windows. But I would gander that the only thing “OSS” about many of these is the kernel and SAMBA. If I wanted something like that, I’d just build my own; no need to purchase a “solution” from some vendor.
There’s an obvious value proposition to OSS storage at the “high end”; that is if anyone could get it right. However, to truly build an “enterprise” class general purpose storage system requires mucking around with things like drive firm ware; and that is more work than we make it out to be. Most folks would be unwilling to provide this work pro gratis to their competitors.
Truly scalable “Enterprise” class DFS protocols and Cluster File Systems have existed for some time (Andrew File System) but most of these solutions requires using non native drivers or interfaces in the Operating System, making long term maintainability an issue. Generally, these environments are specialized to the point that it requires in-house support.
Max –
I’m going to have to ask for some specific examples of where custom drive firmware is necessary (or even just very desirable) for creating enterprise-class storage. XOR-at-the-drive doesn’t qualify any more: current commodity processors and RAM have bandwidth to spare for doing that. NetApp handles ‘ATA spasms’ (where a drive may spend over a minute making heroic attempts to read a marginal sector) in software, by just timing out quickly and returning another copy of the data if one exists – though that may be history now: ISTR that current (S)ATA drives provide the option (for exactly that kind of redundant configuration) to disable heroic recovery attempts. Contemporary drives are even starting to include ooptionalinternal scrubbing facilities for situations where higher-level software doesn’t need to exercise finer control over that.
One can always avoid the need for custom client software simply by using bog-standard protocols (such as CIFS or NFS for file-level access or SCSI/FC for block-level access) at the client connection leading to any desirable level of transparent revectoring at the other end of that connection. The primary benefit of using non-standard client software is avoiding that extra ‘hop’ (bottlenecking at the far end of the hop can usually be avoided by distributing aggregate client load such that different clients usually enter the system via different server nodes, all of which can revector requests appropriately) – but that’s just as true for proprietary solutions as it is for open-source solutions (unless your point was that the former might be more likely to be able to provide required client driver updates at some arbitrary point in the future – but the ability for *anyone* to step in and do so should it be necessary is one of the alleged advantages of open source).
As I suggested before, I believe the main problems involved creating an open-source/commodity-component storage solution center around managing the configuration – presumably a networked configuration, since commodity components don’t scale ‘up’ nearly as well as they scale ‘out’ (though with 16-core single-board products ‘up’ is already a good deal higher than it used to be…). However, enterprises already need in-house networking expertise for other reasons, so a modest amount of additional configuration there (especially if accompanied by clear guidelines) should not be an insurmountable obstacle as long as the storage-related network uses the technology that they’re already familiar with (I’m thinking Gigabit Ethernet for now with up to four links per box, because that’s dirt-cheap and not too bad a bandwidth match for individual boxes managing 4 – 40 drives per link depending on the typical disk utilization and amount of streaming bandwidth required by the installation’s typical workload). Keeping a clear map to facilitate repairs and making sure that the right boxes are connected to the right switches such that all data can be served up via multiple independent paths is perhaps the main challenge, unless you’re trying to cut corners to reduce bandwidth requirements on intermediate portions of the paths.
But then you do need the software required to make all this work without requiring a priesthood to administer it, and AFAIK it just doesn’t exist yet. Lustre is a start but appears to require extensive manual tuning and on-going upkeep. ZFS is a partial solution even in the single-box environment, and offers no scale-out facilities at all in this respect. Andrew was never much more than a (very interesting) novelty rather than anything resembling ‘enterprise-class’ storage: it was designed for an era of unreliable connections and tight network bandwidth where loose consistency might be considered preferable to no distributed access at all, but those days likely won’t be seen again until we begin trying to maintain consistency over interplanetary distances.
Amazon and perhaps Google have self-managing scale-out concepts to contribute, but currently seem to be interested only in using them internally rather than in taking them mainstream. Someone else certainly could, though, and then we might actually see credible open-source enterprise storage appear.
– bill
“Storage arrays are only about $30 billion a year. Who could build a business on that?”
If the OSS startups are lucky, they can turn that into a $3B market. 🙂
Agami itself isn’t OSS, but it’s built with OSS inside. See their ‘agamiOS’ page at http://www.agami.com/site/operating-system which clearly says ‘Linux 64-bit ccNUMA kernel PMD – RAID’ in the middle of it.
I’m sure they leverage this for things like ethernet bonding, not to mention little things like CIFS, NFS, NDMP, and iSCSI.
I’m also sure that they have some highly proprietary apps sitting in the middle of all this doing things like HA and user interface and such.
I’m testing glusterfs, the project roadmap looks promising.
http://www.gluster.org/glusterfs.php
Speaking of new entrants, and perhaps answering some of the requests noted in previous comments, NexentaStor is in late Beta and seems to cover a lot of what people are looking for. Yes, ZFS-based storage appliance is now here. I wrote about it recently myself: http://jmlittle.blogspot.com/2007/11/coming-out-party-for-commodity-storage.html
LeftHand network’s SAN/iQ is a clustered iSCSI SAN build on standard x86 hardware running Linux. Of course, all the “special sauce” is in their user-space file system and management tools, which are of course closed-source.
Their real sucess is that you don’t even know what’s under the hood… the management interface is dead-simple, and the clustering and failover are automatic. Configuring and testing those things in a “roll your own” solution built on Linux or even OpenSolaris is a total nightmare. This is why turnkey, “closed” storage solutions are still prevalent. It is also why turnkey, “closed” RDBMS software is still used in most mission-critical applications. Nobody sane trusts their financial data to MySQL.