OpenStack Swift software defined storage

by Robin Harris on Tuesday, 18 March, 2014

Back in 2006 – before Barack Obama was famous – StorageMojo evaluated the Google File System and concluded

Looking at the whole gestalt, even assuming GFS were for sale, it is a niche product and would not be very successful on the open market.

As a model for what can be done however, it is invaluable. The industry has strived for the last 20 years to add availability and scalability to an increasingly untenable storage model of blocks and volumes, through building ever-costlier “bulletproof” devices.

GFS breaks that model and shows us what can be done when the entire storage paradigm is rethought. Build the availability around the devices, not in them, treat the storage infrastructure as a single system, not a collection of parts, extend the file system paradigm to include much of what we now consider storage management, including virtualization, continuous data protection, load balancing and capacity management.

GFS is not the future. But it shows us what the future can be.

The future is now downloadable
Software Defined Storage with OpenStack Swift by Joe Arnold, CEO of SwiftStack, describes a GFS-like object storage system.

  • Commodity hardware. That doesn’t have to be at the same rev level or generation, servers and disks.
  • 3x replication and smart data placement. Like the original GFS.
  • Object storage. The only way to fly for high scale.

But Swift is more than GFS. It supports user accounts and managing those accounts. S3 API support. Global clusters. Authentication services. Block storage. And more that is needed outside Google’s data centers.

The StorageMojo take
Ten years ago, when I started blogging, there was just one way to do enterprise IT: buy – or maybe lease – storage and networking boxes from vendors with gross margins in the 60% range. That expense sank many dot bomb startups even faster than their iffy business models.

Virtual servers were the hot new thing thanks to their intoxicating flexibility. The intoxication continues.

If VMware was the gateway drug to data center virtualization, Amazon Web Services was the 2×4 head thwack that CIOs needed to stop playing golf with vendors and start playing a new game: adapting scale-out strategies to enterprise-scale computing. While not everyone agrees, these strategies are key to bringing big data apps in-house – the first step to applying these to our cooling storage needs.

OpenStack Swift is a good example of the trend. SwiftStack’s book is an excellent intro to implementation at the enterprise level. The technology is there, but that doesn’t answer the CFO’s questions about how this saves money and positions the firm for the future.

That’s for another post – or two.

Courteous comments welcome, of course. Joe’s book is available from SwiftStack in hardcopy or as an eBook.

{ 3 comments… read them below or add one }

Andi Wednesday, 19 March, 2014 at 12:15 am

While looking forward to read that book, i like to add that with Ceph there is another open solution available with the same feature set.
The problem with both we see at the moment is that with the recommended 3x replication they both loose the commodity hardware price advantage compared to other solutions who do “parity” based availability.
But yes, it’s great to see a “swift” in the storage game, we need true scale-out, we need livecycle management which does not just work on powerpoint presentations, we need data protection for petabytes and we need seamless migrations

Arik Wednesday, 19 March, 2014 at 3:32 pm

Hi Robin,
My two cents:
Swift and Object Storage are definitely good ways for high scale, but there are still few points for improvements:
1. why to keep 3 (or more) copies instead of using a more economical solution that provides even better availability like Erasure Coding. There are already few vendors and few solutions that deploy it.
2. What about performance ? If I want to scale not only on my archiving or repositories I need a solution that can meet picks and high workloads.
Add these 2 to Object Storage and life is sweet(er).

Wim Provoost Monday, 2 June, 2014 at 7:52 am

Hi Arik,

just a note on your first point. The next release of Swift will have erasure coding baked in (expected May/June I believe). See the SwiftStack announcement for that

Leave a Comment

Previous post:

Next post: