Second, the M out of N concept has been around for at least 3 decades. The devil is in details. Computationally, Reed-Solomon code is the only code that can support any M and N, but at a cost of high computation overhead, even with the Cauchy implementation ( which CS is using ). That is why from DVD player to cell phone, there are special circuits doing those decoding operations; and that is why Jim Plank dedicated so much his time to find good encoding and decoding algorithms for the Reed-Solomon code, and still with little significant results to change the landscape.

Third, from published CS white paper, they only implemented the Reed-Solomon codes up to N = 11, i.e., at most 11 data centers. Hopefully they have gained better knowledge of coding, last time I checked a year and a half ago, they hard-coded all the equations for encoding and decoding, without any systematic way of expanding or shrinking the system.

I like to see CS succeed, but unless they already have new competent technical blood, they had a very shaky technical base to start with, and as an ISP I wouldn’t buy their products now even if they are (relatively) cheap; and as EMC, I certainly wouldn’t buy them now, since there is nothing there in-house R&D people don’t know or cannot build. Admittedly, CS did a much better job in marketing than product developing, such as the NYT article a year ago …

For any practical purpose, resiliency through any 3 our of ‘M’ failures using RAID-5 at two separate sites (or any 5 out of ‘M’ failures using RAID-6) is more than sufficient: the IDA approach may be ‘cool’, but that’s about the limit of its interest (especially given its performance and cost drawbacks) – though the ability to use a storage ‘utility’ to store copies of your data off site certainly has value in many situations (its value just isn’t increased by using IDAs to do so).

And if you can place servers in locations where the law can’t compel you to divulge their contents (Ausmith1′s assumption above), then there’s no need to use IDAs in that situation either.

- bill

Consider a ‘Time Machine’ volume that runs over iSCSI (were it that Apple hadn’t pulled the iSCSI initiator from Leopard). With a bit of front-end software, Cleversafe would be a really interesting offering, much safer (and more convenient) than a USB disk, or even Apple’s new ‘Time Capsule’ with its single (non-”server class”!) drive.

Michael Rabin at Harvard has been describing IDAs at least since 1989 – FAIK he invented them (and/or the term). But finding uses for them has proven elusive: about the only sensible one I’ve heard of involves circumventing deliberate governmental censorship – but my impression is that even Freenet uses simple (usually encrypted) copies rather than IDAs, though OceanStore uses IDAs IIRC.

The problem with distributing data around the world as Cleversafe describes is that you have to visit (p-m) locations around the world (or locations on disk, even if all the disks are local) to get *anything* you can use – which is expensive in terms of both disk utilization and (if the pieces are far-flung) latency and network bandwidth. Contrast this with a parity-style approach where you need just *one* access (unless you’re in the process of recovering from a disk failure) to read useful data, and at most four (rather than the full ‘p’) accesses to write useful data – and can often arrange for that single read access (or two of the four accesses required for a write) to be local (even if your parity arrangement is distributed) when data exhibits significant geographic locality of reference.

Worried about security? That’s what strong encryption’s for – you don’t need any slicing and dicing and distribution to the four winds, and it works just fine in combination with a distributed parity-based redundancy scheme (if indeed that makes sense for you, though for most users a single remote mirror location will probably be more reasonable, with additional, possibly parity-based, redundancy at one or both sites as appropriate: tolerating a single full-site disaster is usually considered pretty adequate, and the savings in both site cost and performance overheads compared with distribution around, say, a dozen sites is likely to dwarf the cost of mirroring the storage – and RAID-6 will allow up to two disks to fail at each site if RAID-5 isn’t considered sufficient).

So while Cleversafe is indeed both clever and safe, it’s not clear that it’s particularly useful compared with the available alternatives.

- bill