It’s time for primary data compression

by Robin Harris on Tuesday, 5 July, 2011

Deduplication has been accepted as an enterprise-class compression technology. Is it time for data compression to be a standard feature of primary storage?

I’ve been doing some work for Nimble Storage a cool Valley startup. Talking to co-founder Varun Mehta, he mentioned that Nimble’s storage/backup/archive appliance does data compression on all data, all the time.

That’s right, primary storage on Nimble’s box is always compressed. Not only that, all their performance numbers are quoted with compressed data.

They aren’t kidding.

Venerable compression
Data compression is one of the oldest computer storage technologies around. Bell Labs mathematician Claude Shannon published A Mathematical Theory of Communication in 1948 which, among other things, laid out the math behind compression.

The ratio of the entropy of a source to the maximum value it could have while still restricted to the same symbols will be called its relative entropy. This is the maximum compression possible when we encode into the same alphabet. One minus the relative entropy is the redundancy. The redundancy of ordinary English, not considering statistical structure over greater distances than about eight letters, is roughly 50%.

In line compression has been part of every enterprise tape drive for decades. The algorithms – Lempel-Ziv was big 20 years ago – have been tuned to a fare-thee-well.

Compression is as thoroughly wrung out as any technology in the data center.

So why don’t we use it everywhere, like Nimble?

Not about capacity
The doubling of capacity from compression is not the big win. The larger benefit is that it more than doubles the internal bandwidth of the array – because bandwidth is more expensive than capacity.

And bandwidth is more important than capacity. As John von Neumann noted in his First Draft of a Report on the EDVAC (pdf):

This result deserves to be noted. It shows in a most striking way where the real difficulty, the main bottleneck, of an automatic very high speed computing device lies: At the memory.

Varun reports that Nimble’s comdec operates at wire speed on a multicore CPU, no ASIC or FPGA required. It must increase latency, but given Nimble’s focus on full stripe writes the increase in bandwidth must more than make up for it.

The StorageMojo take
Since it is possible to perform wire-speed compression/decompression with a commodity CPU, why not everywhere?

Will RAID controllers stumble reconstructing compressed data? Is compressed data more prone to corruption? Is bandwidth so cheap that we don’t need more?

I don’t think so, but I’m open to dissenting opinions. With disk capacity growth slowing comdec everywhere is a good way to increase performance, reduce $/GB and have something new to show customers.

Courteous comments welcome, of course. StorageMojo dove into this 5 years ago in 25x data compression made simple.

{ 29 comments… read them below or add one }

Jean July 5, 2011 at 6:00 pm

Not a first. Back in mid 90’s StorageTek 9200 followed by a series of other product models like V2X or current Oracle VSM use compression by default. I think they were the first one doing this at that scale.

The compression chip is at the channel interface level. So no performance hit. They are using technique and same chip found into tape drives.

The biggest advantage is they were capable to move compressed data to backup tape directly or remote copy to another machine (High-Speed Data Mover). They saved tones of remote copies and backup hours in a year.

They were also the first with thin provisioning and snapshot.

Software of this machine can be understand here:
The machine itself here

Fazal Majid July 5, 2011 at 7:26 pm

We use OpenSolaris ZFS’s LZJB compression on our production PostgreSQL databases, running both OLTP and analytic workloads. Enabling compression produces a major boost on performance, even though it is done in software, not ASICs, because the servers have CPU to burn, but disk I/O is usually the bottleneck for databases.

J. Ph. Papillon July 5, 2011 at 11:49 pm

The bigest win for compression is network bandwidth. It makes me crazy when I lead an audit and find non-compressed Windows shares. And I am not a Microsoft addict.

nate July 6, 2011 at 7:12 am

Wouldn’t compressed data incur a pretty big performance hit for random I/O workloads (which is most workloads?) Space savings with compression go up with more segments of data being compressed in one group, so if you want to retrieve a bunch of different 8-16k blocks you may end up having to retrieve several MB worth of data to get to those blocks, in some cases you may get lucky and happen to land several of the blocks you want in one of the larger bulked compressed blocks.

I suppose if your using SSDs, compressing data on those could give a good benefit since SSDs likely have so much available I/O capacity they can take the hit, and the $/GB on SSD is still of course much higher than other platforms. And compression would be good for cold data on regular disks.

I can certainly see for highly sequential workloads inline compression being good though.

I’ve looked at Nimble storage’s site on a couple of different occasions but never noticed any performance claims.

Erkki July 6, 2011 at 8:05 am

Mixed about this. I would not chase for performance over data security. For data security I consider my first experiences with tape drives at mid 90s. I entered a day job, where a really important database backup was done to tape daily. I can’t remember, how I noticed that the tape drive did compression at drive’s hardware, which woke my worries, if the compression made the backups tape drive specific. As my first independent decision I configured the tape drive so that it did not compress, despite disabling the compression brought up new problems like tape switching. I also remember spending hours trying to find, from various sources, if the tapes with compressed data were usable with a new tape drive from the same manufacturer or even from a different one. I was left without an answer. Eventually the tape drive died and I found that tape drive availability from the old’s manufacturer had become practically zero. I bought a brand new DAT drive, as the previous was. The new drive had tape compression capability, as normative with tape drives. I tried to read some old tapes with the compressed data and could not read a bit from them. That was my lesson that I haven’t relied on tape hardware compression ever since and bothered to check how vendor neutral a displayed compression feature might be.

Nicko July 6, 2011 at 1:58 pm

Its a logical approach, spend CPU time and introduce a slight delay to lower the data going to a slow resource. Its the exact same argument when dealing with LFN (Long Fat Networks), big, fast pipes with long delays . Multiple GigE fiber lines going 3000+ Miles, have inherent latency that can’t be avoided. Its not a matter of bandwidth its the 30-40ms delay from one end to the other. Just like a disk, it has an inherent delay in the operation.

So the general solution is you introduce delay to reduce the overall dataset being transmitted. If you add 10ms compressing and de-duping to the 40ms round trip time, but in the process cut the data size down, the overall time to transfer the data can be lower. Which is exactly what Nimble is saying, spend CPU time and add overhead to push widen the bottleneck.

As to compatibility with tapes as an example, that generally doesn’t come into play with disks in most business environments. In all the places I’ve worked you buy a SAN, you expand it by adding shelves, and then eventually it gets replaced or left as a lower tier storage. The hardware remains generally the same as it was the day it was purchased till its retired. I never really end up yanking disks out of my old CX700 and need to toss them into my AMS2500, or and still read the data, or take my NetApp shelves and run them on a Isilon. So unless one day Nimble just abandons compression mid-stream most companies would never see issues like you have with 10 years old archived tapes.

Umesh Maheshwari July 6, 2011 at 3:18 pm

(I am the CTO at Nimble Storage.)

> “So why don’t we use [compression] everywhere, like Nimble?”

The biggest challenge with compression is storing variable-size compressed blocks on disk. It’s not about CPU anymore, it’s about the disk layout. The Nimble system is able to concatenate variable size blocks of logically random addresses into a single RAID stripe.

Most other systems have a slotted-view of disk space, with slots of say 4KB each, making it difficult to store variable-size compressed blocks into these fixed-size slots. It seems WAFL does it by compressing a batch of N logically-consecutive blocks and storing them in a smaller number of slots. But this creates a huge read-modify-write overhead for random-write workloads such as OLTP. (Updaing a single block requires reading, uncompressing, re-compressing, and re-writing the whole batch.) Perhaps this problem is what Nate is referring to in a previous comment.

Jean July 6, 2011 at 4:01 pm


Depending on the way you’ve setup your backup and what backup software you have used, the compression can happen at different place. Most software compression setup turn on server compression and not tape drive compression. So your server CPU got involved and not the tape drive.

Tape compression happen by default on all tape drives (LTO and all Enterprise). This is true since the mid 90’s. You can turn on/off compression on the tape drive by using the right driver setup. On Unix the default is /dev/rmt/cbn where “c” mean compression, “b” for Berkley and “n” for non-rewind driver. Using /dev/rmt/bn will turn off compression on most tape drives.

They use loss less compression like ZIP. So no data can be lost like some people think. It is not like video…

Tape compression has been very effective over the years saving millions of dollar on media and time. Very safe to use.

BTW the new tape compression chip also does encryption at native tape drive speed (>270MB/sec). No latency cause by them.

Steve Shockley July 6, 2011 at 5:05 pm

Years ago (2004 or 2005) I did some testing of built-in Windows 2003 disk compression on a moderately busy file server (1000 users, DL380 G4). In almost every test, compressed data sets equaled or outperformed uncompressed data. In practice performance was good, but it required frequent defrags because Windows compression fragments files.

David July 6, 2011 at 5:50 pm

Isn’t data security one of the big obstacles?
If you encrypt the data first (at the host side), then it’s uncompressible by definition. (Because encrypted data is completely random, and therefore uncompressible)
If you compress it first, then encrypt it – you have to provide the encryption keys to the storage device or subsystem.
This raises some very contentious security arguments – not technically complex to overcome, but to the security-paranoid the issue of enterprise encryption key security is a massive problem.

Nicko July 7, 2011 at 8:49 am

Umesh, do you do any intelligence on the compression side? Usually you encounter a mix of data on NAS systems some compressed formats, some not. Do you do any peering into the blocks to try to do a quick cost determination on if compression will buy you anything? Or do you attempt to just re-compress all incoming data?

nate July 8, 2011 at 2:59 pm

Nicko said –
“If you add 10ms compressing and de-duping to the 40ms round trip time,[…]”

40ms round trip? Holy crap what kind of storage system are you talking about? that seems really horrid. If your talking about a basic workgroup file server, no big deal, most anything else you really don’t want to go above 30ms, some newer storage systems average under 10ms round trip times (with spinning rust).

Greg July 9, 2011 at 6:54 pm

The primary win for compression is actually about memory cache. As much as the internal bandwidth of the array is bottlenecked, there is a lot more efficiency to be gained by keeping more data in cache.

Dave July 10, 2011 at 9:58 am

Don’t think this would help large file media-centric storage as much as with small file highly compressible files.

Dean Clarkson July 10, 2011 at 6:15 pm

Yes, totally agree that primary storage data compression is very useful. That said, this is nothing new. Other file systems have had variants of this for at least 6 + years now. Furthermore, variable data size after compression is not an issue with software RAID mechanisms that support variable block sizes, thereby converting every write to a full-stripe write. Just wanted to say that this “new fangled cool stuff” has been in use for only at least 5+ years on other file systems.

By the way, I do want to point out that not all compression algorithms are created alike. For e.g. gzip (especially, gzip-9) requires significant computing power. Hence, with “weaker” processors, one should not expect line speed throughput on Gigabit Ethernet, let alone on FC. On the other hand, other compression mechanisms (Lempel-Ziv based) are not as heavy on compute resources.

Another point I would like to make is that compression is not a replacement for deduplication. Each serves its purpose. Even in non-backup applications, such as Virtualization, deduplication when used properly results in significant gains in both capacity as well as IOPS/latency/throughput. So, trick is to do both, and do them well together. 🙂

Jean July 10, 2011 at 7:03 pm

Encryption normally occurs after compression. All Compression/Encryption chip do it that way for tape drives.

Here is one of the vendor with this technology

For the others.
About 80% of NAS market can benefit of file compression. YES some files cannot be compress. Media server NAS (TV, Music, Images) type of file services do NOT benefit of compression. Like I said above, it is for less than 20% of NAS in the enterprise.

To see if your data can be compress, simply use backup image to LTO tape. If you data compress well on tape then it should be OK with compression on disk.

Pradip Kulkarni July 10, 2011 at 11:30 pm

IMO, I see 2 issues with block level (I mean below file systems) compression, one is that the disk throughput/GB is already falling with the rate at which raw disk capacities are increasing, and compression adds to that problem and this issue is significant esp for primary storage more than anything used for long term archival.
The second issue, is that for OLTP and many random r/w intensive apps, there is more data than necessary that flows over the interconnects and bus’s to the CPU for compression and decompression lowering the aggregate throughput and overall system scalability. some of the OLTP environments like databases have their own compression and that probably is the better place to do compression (or even file systems) as these layers know exactly how much to pull off of a disk making it more efficient.

SIYA July 11, 2011 at 1:18 am

Nimble does data compression on all data, all the time….???

Ive had a difficult getting a good compression ratio on a seismic data.

jacob marley July 11, 2011 at 4:14 pm


I believe Nimble’s array’s can enable/disable compression on a per volume basis.

Might be a setting that has to be set at volume creation time; not sure about this point.

udubplate July 14, 2011 at 9:49 am
James July 14, 2011 at 7:20 pm

@Umesh, Nimble Storage

Jean’s earlier post makes valid points in regards to the pre-existence of primary storage variable compression, deduplication and RAID mechanisms that account for them.

Nimble seems to be taking credit for uniqueness in approach when the wheel was invented elsewhere many years back.

Darren McBride July 15, 2011 at 3:56 pm

J. Ph. Papillon- What on earth are you talking about? If you mean Windows NTFS compression, default behavior is to decompress when files as they are accessed or copied and then recompress them (assuming destination is compressed). Read last sentence of 1st paragraph Or are you compressing Windows Shares with something ELSE? What am I missing?

Tuesday, 5 July, 2011 at 11:49 pm

The bigest win for compression is network bandwidth. It makes me crazy when I lead an audit and find non-compressed Windows shares. And I am not a Microsoft addict.

Robin Harris July 15, 2011 at 4:49 pm

James, a couple of points. 1st, Exar specializes in software and PCIe hardware for compression and encryption, not as a vendor of general purpose end-user storage systems. Nimble is.

2nd, Nimble is not saying that no one has ever done compression, dedup, SSDs or RAID before. What they’ve done is put the pieces together in a novel way, to give users an appliance that collapses 3 processes into 1 box.

IMHO, that will prove an attractive and durable value proposition.

Bob Farkaly July 18, 2011 at 3:21 pm

Robin, responding to your comment about Exar, we are working with OEM customers to integrate our primary storage in-line, line speed deduplication, compression and encryption into products. I am the product and marketing manager for the BitWackr product which is a turnkey offering for OEMs that places primary storage deduplication on the marquee. Some of our OEM customers are shipping products with BitWackr technology. I offer one of our customers as an example: Others are coming. Compression by itself in fine, but when combined with deduplication, the resulting data reduction can be stunning.

Chris Fricke July 18, 2011 at 5:36 pm

@Jacob Marley: Incorrect. Unless there’s some secret sauce behind the scenes I don’t know about there are no options for customizing or turning off compression on a per volume basis. Or at all for that matter.

@everyone else: One of the nice things about the Nimble array is simplicity. Yes it has compression. Yes it has SSD cache and other stuff to make the data move around. But you don’t have to worry about any of it. Some admins will like that and some won’t but it’s nice that someone is giving us the option.

With regard to speed all I can tell you is that a single Nimble array is noticeably faster (as in even the end users noticed) than a cluster of EQL running SAS on R10. Just sayin.

John Matze July 18, 2011 at 8:18 pm

BridgeSTOR began shipping a deduplication/compression/thin provisioned array in November of 2010. BridgeSTOR ships 4 products. 2 for primary (VMware and NAS) and 2 for backup (Backup Exec and DPM). Our systems have been optimized to use internal memory or SSD for our hash table lookup. You will find a huge difference in performance when using internal memory over SSD technology. We also use compression. It’s only logical that when you find a duplicate block of data that you compress it. The compression becomes a multiplier of the data. I see no reason of using an SSD if you’re compressing only. As others we will be introducing a full SSD array with 3:1 deduplication. It makes a full SSD array very affordable. Using the Exar chip also allows us to use more affordable processors as the offload chip can do the hashing/compression/encryption while the main core processor can do storage functions. Our current products ship with a single quad cores and we still get great performance.

Peter Baer Galvin July 21, 2011 at 12:08 pm


Does Nimble offer deduplication in addition to compression? I have
checked out the web site and they talk a lot about compression, but
not deduplication.

Jacob Marley July 21, 2011 at 8:02 pm

@Peter Baer Galvin

Nimble does not offer deduplication.

NOTE: work uses a Nimble storage array; I am not the storage admin just a storage geek 🙂

MSP Online Backup July 27, 2011 at 1:26 pm

Our application (online backup platform for MSPs) is very sensitive to the economics of storage density as a cost/GB metric, so I’d welcome this type of innvoation as it would make storage much simpler, *if* it made it cheaper.

Just making it more dense via compression doesn’t mean I get a better cost/GB stored, and that is what most people are after.

Our analysis backing up billions (and billions) of files has shown that in some cases the overhead of compression isn’t worth it, and in some cases it really is. We’ve found that some pre-compression tests are actually economical from a computational perspective, ensuring that the cycles to compress are only executed when warranted, on a file-by-file basis.

Leave a Comment

Previous post:

Next post: