The friendly folks at Scality have put up $100,000 to encourage open source development of useful cloud storage bits. It’s open to anyone, not just grad students.
Yup, it’s corporate self-interest at work – Scality sells object-based cloud storage software – but they’re taking an enlightened approach. The resulting code will be open source and is intended to work with a variety of object-based cloud storage services – such as S3 – not just theirs.
Real money
The defined projects have bonuses of $2,000 to $10k. The projects include a Gallery plugin for the
. . . full replacement of the underlying filesystem based storage of content/objects with object storage using the REST interface . . . .
There’s a WordPress plugin project to add an object storage backend to the popular CMD. The $10,000 prize goes for a KVM virtualization storage engine for the Linux Kernel Volume Manager that provides:
. . . block level storage volumes that can be attached to KVM virtual machines. The solution should not require any central node, for example, no central meta-data server and provide a completely stateless operation model.
Here’s the list of defined projects.
Even better
And almost â…” of the money remains uncommitted. If you have an idea for domesticating object storage in the cloud – propose it.
The StorageMojo take
Objects are the future of large-scale storage. If cutting edge stuff gets your heart pumping, this is a good place to start.
Or just collect some cash and move on. Your choice.
Feel free to ask questions in the comments. I’ll ping the Scality guys to get answers.
Courteous comments welcome, of course. I’ve done some work for Scality and like the team. I’m also wondering why Amazon hasn’t done something like this.
But who on earth would want to use a virtual storage device type that is in reality a OSD emulation on top of block storage with WAN type access performance? A cloud OSD would probably have access latency in 100s of ms – without IO queue size in the order of 10K cmds would probably perform extremely poor. Reliability considerations put aside who would want to put his data on such a thing?
PF – absolutely. Amazon’s EBS latency + performance is bad enough and that isn’t even going over a WAN (well not a very distant one!)
PF, Nate –
Amazon and Google, among many others, use object stores for their web services. Yes, the disks lay the data down as they always have – in blocks – but the layer above that is an object store, not a file system. No emulation involved.
As for latency, yes it is an issue, but not for every app. A Gartner survey recently found that 10% of corporate IT spend is for cloud computing, so they’re running something up there.
See my posts on the Google File System for more info on their use of object storage.
Robin
Giorgio from Scality here.
@PF, well to get the best performance out of an object storage system, there’s not doubt speaking its native protocol is the way to go instead of emulating another protocol on top.
This is where we are trying to help with our bounty program, instead of emulating a file system or a block device, plug your application directly with a fast, native object storage client implementation.
There’s no OSD emulation here, typical time to transfer for an object of around 100KB is below 60ms.
Not 100% of all applications can make good usage of this type of storage but I think that a large majority of applications can and this number keeps growing, fast.
For example, storing your photos or other type of personal data does not need SAN type latency.
If I’m reading it correctly, that Grand Prize is a doozy. It would allow a workload to run on Linux and connect natively to Cloud Storage without the use of a gateway or Portal. No API programming, no changes to existing code lines. The applications would be blissfully unaware anything changed.
The KVM merely presents a block storage volume point and Applications are off and running.
That is very interesting. Latency will be an issue, but I’m sure some smart person could build in a disk buffer.
This is going to become an everywhere technology before long. Apps will have it, hypervisors will have it, Volume Manager will have it, and even storage arrays will have it.
Makes you wonder.
The $10k could be awarded to the authors of Sheepdog.
Robin – I was referring to Amazon EBS (Elastic Block Store) which is a block storage system(format/mount like a normal file system), rather than S3 which is the object-based storage system.
http://aws.amazon.com/ebs/
EBS you’ll find wide ranging user experiences, most of them consistent in that the experience is not consistent. Amazon is trying to address this with a new product that I’m told is called something like “Committed IOPS”, or something like that. But it seems to be in perpetual beta. Stability problems abound..
EBS is mainly used by people for things like databases, or persistent storage for EC2 instances, you can even use them as root devices in the event an EC2 server fails you can map it to a new instance some how(don’t know how). While I’ve been using EC2 off and on the past couple of months by no means am I an expert in it, but I know enough, now more than ever that it’s something I can’t wait to stop using. Wrote a big ~2k blog on it this morning in fact 🙂 [btw thanks for the advice on my blog much appreciated]
@Greg yes, that grand prize is a very very groundbreaking idea, booting a VM from an external cloud storage without any change to existing code lines. Fully transparent.
Droplet includes a disk cache (either simple read cache or write back cache like a RAID card with battery would do) to fight latency issues.
@Bernie, yes Sherdog is promising but I don’t think it’s usable yet, actually I tried to contact the author to see if he would port it to droplet and win $10000 but so far no news.