I had a con call with Chris Gladwin and Russ Kennedy of Cleversafe a couple of weeks ago. They’ve come to market with a product line that seeks to deliver:
- Massive scalability to meet growing digital content requirements
- Unprecedented Security and Privacy for critical digital assets
- Survivability against disasters, dishonesty and time
- Extremely cost-effective infrastructure compared to traditional methods
That’s a quote from their pitch.
Cleversafe’s product line
Cleversafe, IIRC, started as a software company, but their announced products come in nice rack-mountable boxes. There are 3 of them:
- CS Slicestor – Dispersed Storage server – $11.3k
- CS Accesser – Dispersed Storage router – $12.3k
- CS Manager – Dispersed Storage network manager – $12.3k
The Slicestor is a 1U storage server containing 4 disks. The Accessor slices up the data and distributes it – think slice router. The Manager works out of band to monitor and manage the storage network components.
I assume the pricing includes some room for volume discounts. There is an open-source version (c. 2006) of the software. The company intends to offer a software-only version as well.
Why hardware?
The Conventional Wisdom in VC circles is that tin-wrapped software ramps revenues faster – hey, you’re selling tin + bits – at the cost of lower margins and loss of focus.
Qualifying hardware is non-trivial; so you tend to stay on one platform longer than you should. At liquidity event time, software companies fetch higher multiples, so it may be a net loss. VCs live by the Golden Rule: he who has the gold makes the rules.
What it does
Cleversafe has an iSCSI or block storage interface. It takes the data, slices it into small pieces using Information Dispersal Algorithms and then ships the slices off to storage either locally or around the world.
In the latest version you can specify how many slices the system makes and how many slices are required to rebuild the data. If you have 11 data centers around the world, you can specify that, say, 6 are required to recreate the data.
You could lose access to 5 data centers and still recover. If the local controlling authority busts into 3 or 4 data centers, they get nothing. Pretty cool if you worry about corrupt government officials getting hold of your company secrets.
The company is planning on adding FTP, CIFS and NFS in the fullness of time.
How well it works
Cleversafe claims that given sufficient low-latency bandwidth the dispersed storage is as fast as a local disk. That’s a tall order, but for now I’ll take their word for it.
Who should buy it?
The company is aiming the Dispersed Storage Network at ISPs to offer as a service and multinationals with round the clock operations and critical data.
How it works
Cleversafe uses Cauchy Reed Solomon erasure codes to slice and dice the data. These codes have several advantages:
- More capacity efficient and failure tolerant than parity codes
- Doesn’t require a license
- Code and decode are faster than other stack operations
If you’d like to play with Cauchy Reed Solomon, check out Dr. Jim Plank’s software page which includes
. . . Reed-Solomon coding, Cauchy Reed-Solomon coding, general bit-matrix coding, Reed-Solomon coding optimized for RAID-6, and Liberation coding. The documentation provides some tutorial material on matrix and bit-matrix based erasure coding.
I met the good doctor at FAST, where he was delighted to find that Clevesafe – also a FAST presenter – was using techniques he’d worked on a decade ago.
The StorageMojo take
I’m impressed with what Cleversafe has done. They will look even smarter after EMC’s Hulk/Maui announcement this spring. I suspect they’ll be bought by year’s end.
Kudos to the Cleversafe team.
Comments welcome, of course.
Another potential plus to the hardware route – better customer satisfaction. It can be non-trivial to get software to work reliably with all hardware — for example, this is one plus Apple has over Vista (Apple’s supported hardware list is very small versus MS). I also wonder how much impact drive/controller compatibility issues can have on reliability – especially consider the research you’ve highlighted recently. If Cleversafe is pushing reliability, that could be another reason for starting with a hardware box.
Yet another case of interesting technology in search of a problem, with the hope that enough people won’t think too hard about more conventional solutions with equivalent (and sometimes superior) features.
Michael Rabin at Harvard has been describing IDAs at least since 1989 – FAIK he invented them (and/or the term). But finding uses for them has proven elusive: about the only sensible one I’ve heard of involves circumventing deliberate governmental censorship – but my impression is that even Freenet uses simple (usually encrypted) copies rather than IDAs, though OceanStore uses IDAs IIRC.
The problem with distributing data around the world as Cleversafe describes is that you have to visit (p-m) locations around the world (or locations on disk, even if all the disks are local) to get *anything* you can use – which is expensive in terms of both disk utilization and (if the pieces are far-flung) latency and network bandwidth. Contrast this with a parity-style approach where you need just *one* access (unless you’re in the process of recovering from a disk failure) to read useful data, and at most four (rather than the full ‘p’) accesses to write useful data – and can often arrange for that single read access (or two of the four accesses required for a write) to be local (even if your parity arrangement is distributed) when data exhibits significant geographic locality of reference.
Worried about security? That’s what strong encryption’s for – you don’t need any slicing and dicing and distribution to the four winds, and it works just fine in combination with a distributed parity-based redundancy scheme (if indeed that makes sense for you, though for most users a single remote mirror location will probably be more reasonable, with additional, possibly parity-based, redundancy at one or both sites as appropriate: tolerating a single full-site disaster is usually considered pretty adequate, and the savings in both site cost and performance overheads compared with distribution around, say, a dozen sites is likely to dwarf the cost of mirroring the storage – and RAID-6 will allow up to two disks to fail at each site if RAID-5 isn’t considered sufficient).
So while Cleversafe is indeed both clever and safe, it’s not clear that it’s particularly useful compared with the available alternatives.
– bill
I think ‘bill’ is confused about how IDAs are used. They’re not primary storage, but rather a really cool way of having secondary storage that is resilient to N out of M failures.
Consider a ‘Time Machine’ volume that runs over iSCSI (were it that Apple hadn’t pulled the iSCSI initiator from Leopard). With a bit of front-end software, Cleversafe would be a really interesting offering, much safer (and more convenient) than a USB disk, or even Apple’s new ‘Time Capsule’ with its single (non-“server class”!) drive.
Strong encryption won’t help you if the law of the land says that you must divulge your encryption key. The RIPA law in Great Britain is a good example.
http://www.theregister.co.uk/2007/11/14/ripa_encryption_key_notice/
Excerpt:
“Section Three of the Regulation of Investigatory Powers Act (RIPA) came into force at the start in October 2007, seven years after the original legislation passed through parliament. Intended primarily to deal with terror suspects, it allows police to demand encryption keys or provide a clear text transcript of encrypted text.
Failure to comply can result in up to two years imprisonment for cases not involving national security, or five years for terrorism offences and the like. Orders can be made to turn over data months or even years old.”
Now if your data does not reside in a single country you might be able to make the case that you don’t have to turn over those parts that are not in the country with the restrictive laws…
It will be interesting to see how this develops.
I’m afraid that Jim is a confused about the difference between how a technology *may* be used and whether it is in fact actually useful in any practical sense.
For any practical purpose, resiliency through any 3 our of ‘M’ failures using RAID-5 at two separate sites (or any 5 out of ‘M’ failures using RAID-6) is more than sufficient: the IDA approach may be ‘cool’, but that’s about the limit of its interest (especially given its performance and cost drawbacks) – though the ability to use a storage ‘utility’ to store copies of your data off site certainly has value in many situations (its value just isn’t increased by using IDAs to do so).
And if you can place servers in locations where the law can’t compel you to divulge their contents (Ausmith1’s assumption above), then there’s no need to use IDAs in that situation either.
– bill
First of all, it is a well known joke in the coding community that Rabin invented the Reed-Solomon code: IDA is indeed the more famous Reed-Solomon code in its exact form invented in JPL in 1960’s, and now used everywhere from DVDs to cell phones. This shows how ignorant the storage and networking people are!
Second, the M out of N concept has been around for at least 3 decades. The devil is in details. Computationally, Reed-Solomon code is the only code that can support any M and N, but at a cost of high computation overhead, even with the Cauchy implementation ( which CS is using ). That is why from DVD player to cell phone, there are special circuits doing those decoding operations; and that is why Jim Plank dedicated so much his time to find good encoding and decoding algorithms for the Reed-Solomon code, and still with little significant results to change the landscape.
Third, from published CS white paper, they only implemented the Reed-Solomon codes up to N = 11, i.e., at most 11 data centers. Hopefully they have gained better knowledge of coding, last time I checked a year and a half ago, they hard-coded all the equations for encoding and decoding, without any systematic way of expanding or shrinking the system.
I like to see CS succeed, but unless they already have new competent technical blood, they had a very shaky technical base to start with, and as an ISP I wouldn’t buy their products now even if they are (relatively) cheap; and as EMC, I certainly wouldn’t buy them now, since there is nothing there in-house R&D people don’t know or cannot build. Admittedly, CS did a much better job in marketing than product developing, such as the NYT article a year ago …
I agree with Bill. The greatest concerns I have is the network costs and latency. I don’t see how this is going to work for anyone with any significant amount of data. This is a solution for a smaller company or one which is looking to hide data by scattering it across multiple countries.