Here is my response:

Curtis,

Good stuff! I realize that the de-dupe ship has sailed and no one is going to call de-dupe compression. My interest is the marketing of new technology: how do you communicate to maximize uptake? My point is that by inventing the term de-dup, the companies hurt themselves.

Other markets aren’t such purists. MPEG-4 is my favorite example, since it is popularly known as compression, and it is a toolbox of compression techniques, not a single algorithm, which share a lot of similarities with de-dupe technology. De-dupe has more in common with image compression than text compression.

Nor is de-dupe implemented the same way by the vendors, so it isn’t a single algorithm either. Data Domain has a patent on a technique for figuring out how to split the data into the chunks they use. Diligent does it differently, and if it figures a block is similar enough they’ll delta the two and store the differences. In either case, both techniques look like out-of-order MPEG-4 compression.

The technology aside, I believe the de-dupe folks set themselves back 12-18 months by inventing a new term for buyers to learn. De-dupe has some wrinkles that you’ve ably pointed out, yet from the perspective of accelerating the product uptake, hardly worth the confusion the industry created for itself.

Great technology, lousy marketing. I’ll link to you from my post on StorageMojo.

Robin

Even for situations in which that’s not the case, there’s a very easy solution: replicate the single copy a bit more than usual. You don’t need 5,000 copies of a datum to make it secure beyond any reasonable doubt, you don’t even need 5: 3 or at the very most 4 will do nicely – and it you want, you can compress the overhead of a large segment down to that of little more than a single copy by using double- or triple-parity RAID to store it.

As for complexity, there’s really not noticeably more than has existed in Unix (and VMS, and RSX) file systems for time immemorial: multiple pointers to a single copy of data is precisely what hard links are all about (and Unix even handles it right by using link counts, though it took VMS a lot longer to do so, since it wasn’t intended to be a generally-used feature there).

Deduping only on the backup stream by definition relegates the facility to backup-only use. I strongly suspect that conventional backup mechanisms may go the way of the dodo within not all that many years, but that deduping may even increase in importance as larger and larger objects get routinely stored (and potentially duplicated). In any event, deduping your on-line storage (as well as any backups) has significant benefit.

- bill

The technology is very impressive.

To Joshua’s point about the clarity of the term de-duplication: it is a matter of timing, isn’t it? With first use, the term needs to be explained, so it is less clear. Reading through the technically astute commenters here, one sees that the clarity is arrived at through time and discussion, since de-duplication has been used before.

Then there is the question of intent: given that you’ve now spent precious time and energy imparting the meaning of the word de-duplication, how does this affect the customer’s buying behavior? Does it change the business value of your product? Does it create valuable differentiation? Will they buy sooner? Buy more?

Now I think the folks at Data Domain might say that since they’ve explained de-duplication they can now explain why they are better at it. Because as their marketing implicitly recognizes, de-dup is riskier, since you are now relying on one copy, plus pointers, plus an index, plus a bunch of software, instead of a bunch of copies – slow and expensive to create and difficult to track and read copies – but there is safety in numbers.

This is turning into another post, so I’ll stop here.

Robin

Now, one might argue that if you look at all your backing storage as a single very large object the two mechanisms would *then* be identical, but this is not the case, because in-object compression works within relatively small scopes (say, a few KB for a file system using compression, because it has to be able to reconstruct data in a reasonable amount of time, which means it can’t afford to read in large numbers of disk sectors before it can extract the information requested; compression in sequentially-accessed files can have somewhat larger scope, because it can, within reason, be assured that all earlier parts of the file will be available for decompressing the next part).

Deduplication, by contrast, works *only* on relatively large, identical byte sequences: a) the sequences must be sufficiently large – and therefore sufficiently few in number – that indexing them across the entire system (with fast access if you want to dedupe synchronously rather than as a background operation) is feasible, and b) the sequences must be sufficiently large that going elsewhere to access one (e.g., if you’re deduping at the block level and part of one file is duplicated in another) won’t seriously slow down access (though when *entire* small files are so deduped that’s not a problem, since the entire access gets revectored).

- bill

If you use de-duplication, the system will keep one copy and leave pointers for the other nine copies. Hence, you end up with 1GB of data vs. 5GB of data. Add compression on the back end and you end up with 500MB….10% of what you would have ended up with using only compression.

Obviously de-duplication implementations aren’t this simple, but this should illustrate the difference sufficiently for most. In -=essense=-, de-duplication does the exact same function as compression…but in -=practice=- that function is implemented quite differently. The difference between the two is more dramatic than the difference between, say, bzip and gzip.

So I don’t really have a problem with the new term – it more clearly defines the function. Clarity is a good thing.

Explain how mpeg4 isn’t compression?

Would I be correct in assuming that real computers actually remove duplicate files during de-duplication and not, say, doing compares and storing the deltas?

MPEG4 compression does exactly what the moderns call “de-duplication”, with one tiny difference: de-duplication does out-of-order MPEG4 compression.

This kind of hair-splitting sales prevention is what happens when you let techies, like Prof. Li at Data Domain and Neville Yates at Diligent, do end-user marketing.

I remember engineers grousing about calling 100mbit ethernet ethernet, since it wasn’t CSMA/CD like “real” ethernet. I have yet to meet a single actual customer who a) cared, or b) knew the difference.

I estimate they set themselves back at least 12 months by insisting on a new word. But I’m willing to look on the bright side: maybe their stuff wasn’t working and they needed the extra time. I’ve seen that happen too.

Robin