8 Open Source Storage Projects That Want You

by Robin Harris on Wednesday, 25 October, 2006

Open source software (OSS) has had a profound effect on several software markets, such operating systems (Linux, OpenSolaris and the various BSD Uni), webservers (Apache), databases (MySQL), blogging (WordPress and others) and a number of others. The benefits to users include lower prices for commercial products, higher quality products – both OSS and commercial, and greater choice. All in all, OSS is Very Good Thing for the IT industry.

Alas, the OSS storage arena hasn’t been as blessed as, say, webservers, where Apache holds a majority of the server market. There are a couple of nascent storage management initiatives, Aperi and Storage Revolution. The latter is meeting at SNW to write a spec. I wish them luck. The SMI-S spec is big and getting competitors to agree on what and how they will implement it will take some doing, so it will be a moving target for some time.

Yet there are real open source storage products out there that you can contribute to today, even if you aren’t a coder. Many of these projects could use help with things like documentation, support, improved functionality and interoperability testing. You can work on these projects to gain experience that could lead to a higher paying job, or to meet the geek girl of your dreams. Or just for the satisfaction of knowing you are helping improve vital digital infrastructure for fellow practitioners around the world.

BTW, this is a partial list. These just looked like the best bets in a variety of applications.

The projects
Here’s a selection of OSS storage projects that are shipping today.

AMANDA backup
Advanced Maryland Automatic Network Disk Archiver is a 15 year old OSS project. You’d think they’d be done by now, but no, plenty of new work to do. I think that backup is the sweet spot of storage management software, since most commercial backup products are cash-cows and not getting much new investment. Also, AMANDA writes the commands needed to unpack the tape as the first block, so unlike proprietary formats, you can be confident of getting your data back in 20 years – even without the application. That is security. For coders they have a number of projects you can run with. Itching to write documentation? Check out the documentation page. AMANDA is commercially supported by Zmanda.

OpenSMS and OpenTMS
A big knock against OSS is that it isn’t “commercial grade”, though some of the commercial stuff out there isn’t “commercial grade” either. These two projects were commercial StorageTek products that the company donated to OSS. OpenSMS is a former HSM (excuse me, *ILM*) product, while OpenTMS is a mainframe tape management system. These folks would like help supporting the software on more operating systems – as commercial products they supported lots of them – and on more file systems.

Cleversafe’s Open IDA
A secure method of storing enterprise data on public infrastructure. You don’t need encryption because it takes several sites, that don’t know about each other, to put the data back together again. I also think it could be used in a campus environment to create highly available storage using low-cost infrastructure.

Openfiler
Openfiler is a NAS implementation with an impressive feature list.

File-based networking protocols supported by Openfiler include: NFS, SMB/CIFS, HTTP/WebDAV and FTP. Network directories supported by Openfiler include NIS, LDAP (with support for SMB/CIFS encrypted passwords), Active Directory (in native and mixed modes) and Hesiod. Authentication protocols include Kerberos 5.

Openfiler includes support for volume-based partitioning, iSCSI (target and initiator), scheduled snapshots, resource quota, and a single unified interface for share management which makes allocating shares for various network file-system protocols a breeze.

Check out Openfiler projects here. Openfiler is also commercially supported by Xinit Systems.

ZFS
ZFS is the very cool filesystem/volume manager/software RAID that Sun open sourced and that I believe Apple is porting to the next version of Mac OS X. There are opportunities to port ZFS to other operating systems and more. Get started here.

OpenAFS
OpenAFS is a version of the cool distributed file system Andrew, from Carnegie Mellon. AFS is also a commercial product offered by IBM through their acquisition of TransArc. OpenAFS has a number of open projects including ports, network enhancements, native Windows support and documentation.

ATA over Ethernet
AoE is an open spec protocol that several companies have already written initiators for. Its big advantage is that since it isn’t routable it doesn’t use many server cycles and its more secure. Using Coraid’s hardware you can build RAID arrays for less than a dollar per GB. Write a new driver or develop your own hardware, or ???. Check out AoE tools for more info.

As always comments welcome, pro or con. Moderation turned on to control comment spam but no registration required.

Update: I published the first half of this yesterday and completed it today. Rather than break it up into Parts I & II, I’ve combined them in a renamed version.

{ 7 comments… read them below or add one }

Miro Thursday, 26 October, 2006 at 4:57 pm

I would also add Bacula to the list (www.bacula.org) – great network backup tool using very scalable concepts. It can save the data on whatever medium you defined – disk, CD, DVD, tapes, multiple tapes. Works with barcode readers (I just tested it with Quantum Superloader 3 on FreeBSD, worked as a charm)

It is using MySQL as database to store all the files saved on the medium. If one need to restore a file, a virtual Filesystem is created and you can browse in it (coming from the database) until you find what you need to restore.
(nobody remembers the exact name of the file to restore, right…)

Overall it is pretty good system and can easy replace [put your major vendor tape arch. software here] without the licensing fees. Why not use the money for a better hardware instead of licenses. :)

Overall the OSS is here to stay and it will give the commercial software run for their money… Oh, wait the OSS is mostly free (as in Beer).

Also all the OSS projects are creating features based on what is requested/needed by the community, not based on marketing mambo-jumbo like – “Let’s leave this feature for the next version to make them upgrade” -> profit. Therefore the OSS software in most cases does a better job from an admin’s point of view.

Commercial support for most of the OSS is now largely available so there is nothing to fear about…

Javier Friday, 27 October, 2006 at 11:54 am

Robin,

I’m not sure if you have mentioned these other two before:

OpenNMS – http://www.opennms.org/index.php/Main_Page
Splunk – http://www.splunk.com/products/218

Ken Gibson Tuesday, 31 October, 2006 at 5:24 pm

Great weblog. Two more open source projects important to storage:

1) Remote DMA (RDMA) for NFS. This enables creation of ethernet SAN NICs that offload both NFS and iSCSI similar to the way HBAs offload SCSI and FC today.
Linux project is at: http://sourceforge.net/projects/nfs-rdma/
OpenSolaris at: http://www.opensolaris.org/os/project/nfsrdma/

2) pNFS Enhancement to NFS that eliminates the restriction that an NFS filesystem must reside on a single server. Files can migrate and be replicated across NAS servers and clients can always find their data through a lookup servce. Creates a framework on NFS clients for an ‘NFS Volume Manager’ that lets clients replicate, stripe, mirror data across multiple NFS servers.
OpenSolaris project here: http://www.opensolaris.org/os/project/nfsv41/
Linux project here: http://www.citi.umich.edu/projects/asci/pnfs/linux/
My summary of pNFS at: http://storagethoughts.blogspot.com/2006/08/pnfs.html

Matt Thursday, 15 May, 2008 at 8:55 am

Also, check out the StorageIM project on SourceForge (http://sourceforge.net/projects/storageim/). StorageIM is an SMI-S open source reference client that supports monitoring SMI-enabled Arrays, Switches, Fabrics, HBAs, Tape Libraries, VMWare, Linux host information and more. Active management functionality will be available shortly.

Whether storage end-users are aware of it or not, much of the equipment from the big vendors sold over the last few years. This translates into more storage systems that can be monitored and (moving forward) managed via StorageIM using SMI-S.

John Call Friday, 25 July, 2008 at 4:34 am

Aloha Robin,

I’m happy to have found an article about open-source storage on your blog. I’ve got to tell you that I recently purchased an IBM N3300 and spent a pretty penny on the software licensing for NFS, CIFS, etc… I’m glad I didn’t spend any money on the “Snap*” licenses though. I haven’t been impressed with the N-series. On the other hand, I spent an hour or two yesterday searching the internet for a free and open-source solution for NAS. I spent some time comparing FreeNAS and Openfiler and ultimately downloaded the later to have a peak. Wow, I am an hundred times more impressed with Openfiler vs. my N3300. There are a few quirks in Openfiler, but I wonder if it isn’t time for you to revisit this topic and see if we can generate some more conversation around who’s got the best-of-bread open-source NAS solution. FreeNAS vs Openfiler vs ???

Swaroop Friday, 11 September, 2009 at 12:03 pm

Just what I was looking for! I would want to pick one for my grad 6 month project.

Thanks for the post!

Matthew Friday, 31 August, 2012 at 12:47 am

What became of OpenSMS?

Leave a Comment

Previous post:

Next post: