Yippie-ki-yi-yay
As we say here in ranch country.
Sun sent out a press release on the NetApp fracas today. I didn’t have time to parse it, so here’s the raw intelligence:
Sun was legally obligated to respond in Texas to the initial suit brought on September 5, 2007 by Network Appliance to forestall competition from the free ZFS technology. Today we filed additional counterclaims in California, and specifically under the Lanham Act and California Business and Professions Code, based on Network Appliance’s false statements to the public about the alleged use of Network Appliance patents in ZFS. In parallel, we will be bringing a motion before the court in California asking that the case filed in Texas be consolidated with the case filed today for trial in the Bay Area, headquarters to both Sun and Network Appliance. Today’s filing includes counterclaims against the entirety of Network Appliance’s product line, including the entire NetApp Enterprise Fabric Attached Storage (FAS) products, V-series products using Data ONTAP software, and NearStore products, seeking both injunction and monetary damages.
Since Sun was forced to litigate, we feel California is a more appropriate venue to do so for several reasons. First, Sun and Network Appliance are both headquartered in Northern California, within 10 miles of each other. Second, most discovery will take place in California, as many of the key inventors on the patents and primary counsel for both parties are based in California. From both a judicial and economic standpoint, it makes much more sense for the case to be in California.
For more information about Sun’s counterclaims, visit our General Counsel’s latest blog posting: http://blogs.sun.com/dillon/entry/the_netapp_litigation_continued. You can view today’s filing on Sun’s website at www.sun.com/news, and you can check out our Open Source Community Support page at www.sun.com/lawsuit/zfs.
The StorageMojo take
Sun is pulling every string they can to win this in the court of public opinion. At least the public that buys storage.
What some commentators miss is that Sun only has to persuade a small percentage of NetApp buyers to reject or stall purchases to have a massive impact on NetApp’s share price. Here’s why.
Most tech companies – and I’ll assume this is true of NetApp – have back loaded quarters. A high percentage of the sales don’t come in until the last week of the quarter. By then, of course, all the expenses are fixed: components ordered; inventory built; 3 martini lunches expensed.
This means that the last few percent of sales make the quarter profitable – or not. If 90% of NetApp customers love them and continue to buy, but 10% decide they hate them and don’t buy, NetApp has a bad quarter.
Even a NetApp customer who loves them and hates Sun won’t increase their purchases to offset the lost sales. Why would they? The same kit will be cheaper next quarter, especially if the market stays soft.
Will Sun’s gambit work? Stay tuned.
Comments welcome, of course. Marketing is a contact sport.
Update: Since this suit is about ZFS, some of you may be interested in this article A look at MySQL on ZFS that compares the performance and management of MySQL on UFS and ZFS.
Robin,
I think that you fall into the analyst’s trap of thinking that current events really affect (or is it effect) decisions that are long term in nature. First, the reason for buying NetApp or not is the feature and its inherent flexibility – not really a short term issue. Second, the timing of purchases is really something that is tied to internal processes of most purchasers not something that the purchasers move around. Third, lots of purchasers are not sophisticated enough to understand the need for a good analysis of the value proposition of the different storage vendors and how they would substitute one vendor’s product for another depending on the relative price. Last, if I remember right, NetApp is on yearly comp plans not quarterly. This helps stop the back loading quarters.
Keep up the good work.
John, what you say is often true, however I would not underestimate the influence of the operations people in a company who might well not like this move. They often don’t set budgets or make the final decision, but they are very influential among those that do.
Netapps is firing a shot across the bow of the open source community- this will only vilify them and give Sun more mindshare and good will in the industry.
Y’know, Robin, I find it kind of disgusting that (having at least temporarily given up your irrelevant cheerleading for Sun as the alleged open-source good guy) you now choose to characterize this more like a competition on American Gladiators rather than on the actual merits of the situation. It’s sort of like doing the same thing with the Iraq war, though of course only very qualitatively rather than in remote sense quantitatively.
While I understand the magnetism that ‘the court of public opinion’ may hold for a confirmed marketeer, rather than babble on enthusiastically about this ‘contact sport’ like so much of the rest of the world is doing don’t you think it might be more useful to try to offer some insight into the underlying issues (or lack thereof)?
As for the comparative analysis that you cite, the major conclusion one can draw is that for MySQL ZFS’s main advantage is in eliminating the need to use a more contemporary file system like XFS or JFS (rather than a mid-’80s design like UFS) and an underlying RAID to distribute load (though had MySQL used ZFS’s RAID-Z with a parallel workload they would very likely have found that the conventional RAID had a significant performance advantage over ZFS). That said, ZFS is indeed somewhat easier to manage than such a combination (though whether it’s *significantly* easier to manage will vary with the installation) and offers the “small performance gain [that] can be obtained” by disabling InnoDB’s doublewrite buffer (though pays a significant penalty for this when a table must be scanned sequentially, since ZFS’s write-anywhere policies severely fragment the table file – funny how that didn’t get mentioned, though it’s possible that InnoDB, unlike more serious databases, just doesn’t bother to try to maintain its tables in scan-optimized order).
It’s particularly significant that as InnoDB’s own buffer pool size increased, any ZFS advantage disappeared – because what ZFS was being compared against was UFS direct I/O (i.e., no UFS buffering at all). Therefore, ZFS was getting the use of all the RAM that InnoDB did *not* use for its own buffers, while that RAM was just sitting idle during the UFS tests: no wonder ZFS looked better than UFS at low InnoDB buffer sizes.
So the uncritical acceptance of ZFS hype continues, aided and abetted by people who really should know better but just aren’t bothering to look closely.
– bill
Bill,
With all due respect, you seem to have no idea how the court of public opinion works. First, it isn’t rational, so get over it. Second, the perception of relative differences is more important than some absolute difference. Thus Sun may be a suspect OSS citizen, criticized by Linus and others, but when compared to NetApp, whose OSS contributions are much slimmer, they look pretty good. That is what counts.
As for me being a cheerleader for Sun: do a search on StorageMojo and read every post. I don’t cover everything Sun does, but on the storage front I have pulled no punches. Paradoxically, though, Sun is the only major player with nothing to lose by upsetting the current, very cozy, storage market. Since I think the storage market is badly broken I applaud their efforts.
Your comments about ZFS vs UFS performance are well taken. However, why did the reviewer turn off UFS caching? As I read his comments, it is because they actually slow the system down for modern configurations. Perhaps more modern file systems do this better than UFS. Can you point to any benchmarks that substantiate this?
As I read the tea leaves of the marketplace I see an increasing focus on ease of management rather than performance. If ZFS performance is “good enough” and the management is significantly easier – which the combined functions of file system and volume management in ZFS appear to achieve, based on all the reviews I’ve seen – then ZFS has, IMHO, a significant and sustainable competitive advantage. And superior data integrity is another plus.
Robin
1. Of course I understand how the court of public opinion works, Robin: I just don’t believe that those workings are often worthy of much respect, but rather that knowledgeable and responsible individuals should make some effort to educate the public about actual substance rather than just watch the entertainment from the sidelines.
2. If it wasn’t clear that I was referring to your cheerleading for Sun in the particular area of ZFS, then I apologize. And indeed you don’t pull any punches in doing that, but unfortunately continue swinging wildly based on your early preconceptions rather than being willing to educate yourself in the matter. Perhaps such behavior is so deeply ingrained in the genes of marketeers that it’s beyond your capacity to rise above it, but I’ll continue to try to make sure that it doesn’t rub off on others without appropriate rebuttal.
3. As for your sophomoric observation that contributions to open source are ‘what counts’ in this matter (unless you’re simply referring to what the Great Unwashed might mistake for relevance, in which case you might want to clarify that), ask yourself whether you believe that it would be equally reasonable to favor Microsoft in a lawsuit over, say, Sun simply because Microsoft’s charitable contributions (I use the term loosely, of course) outweighed Sun’s, rather than observe that this is only what one would expect of a company with so much greater revenue and profit than Sun has. NetApp really doesn’t have anything save for its special file-system sauce to subsist upon, whereas Sun is not only far larger but far more diversified, hence can find it infinitely easier to throw bones to the open-source community without jeopardizing its existence – or even decide to embrace open source honestly (though it’s not clear that has really occurred) as part of its overall strategy for profitability. NetApp has been very active in supporting and developing open standards, however – as has been noted elsewhere.
4. Sure, Sun has nothing to lose by rocking the storage boat, and all other things being equal I’m pleased as punch that they’re trying to. That does not, however, in any way justify their possible use of protected IP that they don’t own in the process (it’s that old problem of ends not justifying means, if you want it in a more general form that might get past your myopic focus on how ‘cool’ you think ZFS is and how happy you are to see it being open-sourced, at least sort-of). Once again, this confrontation has nothing to do with open source: it’s all about the legality (or lack thereof) of Sun’s attempt to productize (and monetize, via using it to make their systems more attractive) technology whose ownership is under dispute – a mundane issue for the court to decide if Sun and NetApp can’t work it out on their own.
5. The reviewer explained clearly why one *normally* disables UFS caching: “Storing two copies of the data effectively cuts the machine’s RAM in half.” In other words, UFS’s cache *normally* takes up RAM that could much more effectively be dedicated to InnoDB’s cache (having one large cache is far more effective than using an equal amount of RAM divided between two caches, one in front of the other). However, he then disabled the UFS cache *without* giving the RAM thus freed up to InnoDB’s cache, while allowing ZFS full use of the RAM that InnoDB’s cache wasn’t using, thus (whether deliberately or due to simple brain-fade) creating an apples-to-grenades comparison that UFS could not possibly do well in save when most of the RAM was given to InnoDB (at which point – mirabile dictu! – UFS pulled ahead of ZFS).
6. Ease-of-management is indeed ZFS’s strongest potential attraction, though many installations won’t find their current configurations sufficiently difficult to manage to be interested: managing a conventional RAID is pretty well automated by now, and given ZFS’s brain-damaged RAID-Z implementation the residual management overhead of a conventional RAID-5 array may well be significantly outweighed by the latter’s performance advantages in environments dominated by small, parallel random accesses. In fact, the biggest management challenges occur when an installation outgrows a single-box solution, and ZFS doesn’t do squat to help that problem.
7. Superior data integrity is only a marginal plus – we’ve been over that before. It does represent a step forward for most non-NetApp environments, but, again, not such a significant step that people who understand error rates would find that alone sufficient to change platforms.
It still mostly (note that I did not say ‘entirely’) boils down to hype, Robin. You’ve obviously swallowed it pretty much whole, and (perhaps quite correctly) believe that the industry can be convinced to as well.
But that doesn’t make it true, and as an engineer (rather than a marketeer) I just don’t like it.
– bill
Bill – you seemed to be obsessed with proving ZFS is bad, etc. You may not like it but there are more and more quite large (and small) installations using ZFS when not only it’s easier to manage but performance is really good, often better then HW RAID. I’ve been working with quite large ZFS installation when we were able to switch from medium class HW arrays (IBM, EMC, SUN) to simple jbods on ZFS. Not only management is far easier, it opened us new possibilities, also the performance is actually better with ZFS on jbods using RAID-Z2 comparing to RAID-5 on these arrays.
Every environment is different – sure. ZFS (and any other fs) won’t be a best solution for everyone – but as I’ve been using it for years now and also I’ve been using many other solutions I can clearly see the benefits of using ZFS – the experience is just great.
btw: my friend showed me new MacOS on his laptop and yes, there’s ZFS built-in. Not as a default file system (yet) – smart move by Apple.
I’m glad that you’re happy with ZFS’s performance, but lacking any specifics about the character of your workload and what other file system you’re comparing it to that means next to nothing. By how much does it out-perform your IBM, EMC, and Sun ‘HW RAIDs’ (specific model information would be nice, and if the workload is at all bandwidth-intensive knowing how you connect to them is critical), under what workload, using what file system? I’m not questioning your allegation, just noting that to be meaningful to anyone besides yourself it needs a lot of supporting information. By how much if any does it out-perform *software* RAID implementations (again, under what workload, using what file system)?
I’ve always clarified my suggestion that ZFS’s RAID-Z was ‘brain damaged’ by explaining that I was referring to its dramatic inferiority to RAID-5 when exposed to concurrent workloads of small-to-medium-sized accesses. I first raised this issue on Sun’s ZFS discussion forum (or it could have been in one of the developer’s blogs) well over a year ago, and I now see that people are actually *experiencing* it and talking about it there and even starting to explore the kind of alternate design that I suggested as a remedy. So if you’re happier with it than with your previous solution, then either your previous environment was incompetently configured or you don’t have the kind of workload that exposes RAID-Z’s weaknesses.
And since I’ve always given ZFS credit for its ease-of-management features, there’s no obvious disagreement on that score.
If you’d like to present something more *quantitative* I’d be interested in looking at it, just as I was interested in looking at what turned out to be an apparently incompetent quantitative analysis performed by Mr. Duncan (I’ve not yet seen any response to my criticism of it that I linked to on the ZFS discussion forum, which one might suspect suggests that I wasn’t *all* that far off base).
– bill
[Robin – I obviously didn’t proofread my previous comment adequately, so just substitute this one if it’s easy to do. Thanks.]
You appear to confuse legitimate criticism of those blatantly over-hyping ZFS as some kind of attempt to prove that ZFS is ‘bad’. Attempting to dismiss legitimate criticism as bias is one hallmark of those who are biased themselves; attaching such an attempted dismissal to a topic where ZFS’s technical limitations are of relatively tangential importance is another.
I’m glad that you’re happy with ZFS’s performance, but lacking any specifics about the character of your workload and what other file system you’re comparing it to that means next to nothing. By how much does it out-perform your IBM, EMC, and Sun ‘HW RAIDs’ (specific model information would be nice, and if the workload is at all bandwidth-intensive knowing how you connect to them is critical), under what workload, using what file system? I’m not questioning your allegation, just noting that to be meaningful to anyone besides yourself it needs a lot of supporting information. By how much if any does it out-perform *software* RAID implementations (again, under what workload, using what file system)?
I’ve always clarified my suggestion that ZFS’s RAID-Z was ‘brain damaged’ by explaining that I was referring to its dramatic inferiority to RAID-5 when exposed to concurrent workloads of small-to-medium-sized accesses. I first raised this issue on Sun’s ZFS discussion forum (or it could have been in one of the developer’s blogs) well over a year ago, and I now see that people are actually *experiencing* it and talking about it there and even starting to explore the kind of alternate design that I suggested as a remedy. So if you’re happier with it than with your previous solution, then either your previous environment was incompetently configured or you don’t have the kind of workload that exposes RAID-Z’s weaknesses.
And since I’ve always given ZFS credit for its ease-of-management features, there’s no obvious disagreement on that score.
If you’d like to present something more *quantitative* I’d be interested in looking at it, just as I was interested in looking at what turned out to be an apparently incompetent quantitative analysis performed by Mr. Duncan (I’ve not yet seen any response to my criticism of it that I linked to on the ZFS discussion forum, which one might suspect suggests that I wasn’t *all* that far off base).
– bill