Nexenta is the next open source storage company – and the first to use ZFS. They are aiming at the enterprise storage market for 2nd tier storage.
I talked to Evan Powell, the CEO last week, after a StorageMojo reader and Nexenta user, Joe Little, tipped me off to the product. Joe is a systems architect in Stanford’s EE department and is using Nexentastor in production. That is a link to his blog where he talks about it in more detail.
Linux + Solaris + ZFS + glue
Evan described Nexenta’s value-add as upgrading the standard software by
- removing unneeded packages
- hardening internal interfaces
- wrapping it up in a nice GUI with clean underlying APIs
They’ve made a point of creating a clean API and using the same abstraction layer for their GUI and the CLI. The intent is to make it easy for enterprise admins to put Nexentastor into production. The behavior under the GUI and the CLI should be the same.
Did I mention the low-end version is free?
The StorageMojo take
With OpenSolaris adding CIFS support OpenSolaris is the open storage OS of the future. I applaud Nexenta for getting out there now and wish them the best of luck.
Comments welcome, of course. Anyone besides Joe have experience with Nexenta to share?
Note: In a haze last week I started this post and accidentally published it – where it went out immediately over the RSS feed. My apologies if it looks like I’m repeating myself. This is a new post with the old title.
It’s not just CIFS, it’s the COMSTAR project and SCSI target support which will really help; the initial drop from last week (http://www.opensolaris.org/os/project/comstar/InstallingComstar/) works nicely too. This is what x4500s are for.
BUT.. what about the lawsuit from NetApp against ZFS ( SUN ) … if NetApp is correct, then doesn’t Nexenta go belly up?
BUT.. what about the lawsuit from NetApp against ZFS ( SUN ) … if NetApp is correct, then doesn’t Nexenta go belly up?
Probably not. They would just need to find a “non” infringing file system.
If their value add only consists of:
* removing unneeded packages
* hardening internal interfaces
* wrapping it up in a nice GUI with clean underlying APIs
They’re going to need a lot of luck.
The NetApp suit is a non-issue for users. Sun is committed to indemnify customers against NetApp’s claims, a non-uncommon strategy while these issues are litigated.
My guess is that NetApp is realizing that they really are in a no-win situation here. The CIO of JP Morgan talked about their need for open-source storage a couple of weeks ago – and Morgan is a big and formerly satisfied NetApp customer.
The softness in the high-end market – which could get a lot softer if the subprime mortgage mess isn’t reasonably well-sorted in another 6 months – just adds to the pressure on NetApp.
Warmerhoven doesn’t get it. NetApp’s board would be smart to put Dave Hitz in charge right now, because this is a secular techno trend crunch. It all translates to business results, but the driver is the impact of open source on a formerly cozy 60%+ margin business. Maybe Peter Wang, Ed DeCastro or Ken Olsen could explain it.
Robin
RE: “Maybe Peter Wang, Ed DeCastro or Ken Olsen could explain it.”
I would love to hear you wax eloquent on “Great CEOs Who Missed the Turn!”.
Bill Todd might have a comment or two.
Frankly, IT computing and Storage is “Dullsville” right now anyway. Why?
No imagination… No creativity… The list goes on…
Hmmm – wonder who you actually are…
I’ve never blamed KO for DEC’s demise, because I don’t attribute that to any change in his vision. He always believed in “Do the right thing” and letting engineering and customers drive the company, but after the company just got too damn big this approach didn’t work any more (due primarily to management depth which exceeded the ability of this philosophy to pervade it and the resulting creation of fiefdoms that competed destructively rather than – as had previously largely been the case – constructively).
I suppose if one considers ‘missing the turn’ in an administrative sense (not coping with rampaging corporate growth) rather than in the sense that you seem to be interpreting Robin’s comment (not adjusting to market changes) then one could consider that a deficiency – but given that he rode DEC from zero to second only to IBM over the course of over three decades I find it difficult to think of *anyone* with a record that would qualify them to disparage his performance, and suspect that he might yet have pulled DEC back had he been given the opportunity to (and in any event he could scarcely have done worse than Palmer did).
NetApp doesn’t have anything like the same kind of problem: it started life engineering-driven, but became a more conventional large corporation long ago. It has a solid customer base similar in some ways to EMC’s, especially among those who wouldn’t even *begin* to think about taking a serious flyer on new technology from fledgling companies questionably qualified to support it. Sure, some of them will dabble their feet in the water, and eventually will quite possibly cautiously wade in farther, but they’ve already been doing so with previous open-source storage products and with proprietary start-ups and ZFS brings nothing of significance to the table (unless possibly Sun’s own storage offerings, though even they will have a somewhat up-hill battle here) that significantly changes the situation.
One of the worst things that can happen to a development group is to start believing their own marketing group’s hype rather than continuing to strive for real excellence, and at least some parts of ZFS (and a *large* portion of its small but vocal user base) seem to have succumbed to that. Some issues are technical (such as the obvious performance deficiencies with RAID-Z and with table fragmentation in OLTP use, or the silliness – faithfully propagated by Nexenta, I see – of claiming to be ‘zettabyte’ in nature when there’s no way in hell to get beyond petabytes on the single-node configuration which is all that ZFS can support, leaving aside other scaling obstacles), others are perceptual (such as the real quantitative importance of ZFS’s added integrity – especially when weighed against the potential down-sides of moving to a new platform, though of course since WAFL offers comparable integrity guarantees that’s not a differentiator at all there).
So there was more than a hint of hubris involved 3+ years ago when Sun engineers began touting ZFS as “the last word in file systems”, and reluctant as its fanboys may be to come back down to earth the real world won’t have the same problem (because their feet never lost touch with the ground in the first place).
– bill
Bill,
You seem to be laboring under the misapprehension that ZFS has some sort of marketing group pushing it. If they do it is a well-kept secret. We’re talking about Sun, the company whose marketing jumped the rails some years ago.
My only contact with Sun employees regarding ZFS has been the development team themselves. No marketing folks in the room. They even made me pay for my lunch! Yup, they really are engineers. AFAIK, I’ve provided more marketing visibility for ZFS than anyone at Sun, with the exception of their CEO.
My enthusiasm for ZFS is based upon my own evaluation of the architecture, not the blandishments of non-existent Sun marketing. I follow the ZFS discussion forum so I have a pretty good sense of where the product is right now. I’m well aware that no matter how great the architecture it is the quality of the implementation that will make or break ZFS.
I’m excited by ZFS on Mac because I’m tired of running Disk Warrior every few months to fix file system data corruption. ZFS will also force Microsoft to spend a few bucks on file system engineering, which will be a good thing for the Windows faithful.
Bill, let’s just agree to disagree on ZFS. You focus on the problems and I’ll focus on the promise.
Robin
Actually, I was giving the Sun engineers the benefit of the doubt by assuming that someone else had pushed them into calling ZFS “The last word in file systems”: if *they* were the ones who decided to call it that, then the word ‘hubris’ may be inadequate to describe their degree of self-delusion. I’ve had recent interactions with some of them at the zfs-discuss forum and can’t say that I’ve come away impressed – and as far as your own experience went, had they been *real* engineers you’d likely have wound up paying for their lunches as well as for your own.
It’s too bad that your enthusiasm for ZFS’s architecture isn’t based upon an adequate understanding of it: your implied premise that it is ‘great’ leading to the implied conclusion that only poor implementation quality might impede its march toward world domination is a classic case of GIGO.
Not being a Mac user, I’m not familiar with the need to fix file system corruption every few months, so I can’t guess whether it derives from causes that ZFS would help with or not. But what *is* obvious is that most file systems do not corrupt their directory structures every few months (even in the presence of occasional power failures and system crashes) such that a major overhaul is required, nor does the underlying hardware, so you could replace your current Mac file system with just about *anything* and experience the kind of relief that you’re looking forward to (unless the corruption is coming from Mac OS internal bugs which damage data before it’s written to disk, in which case ZFS probably won’t help you either). In general, you appear to have a naive confidence that ZFS will solve most of the storage problems that people encounter, whereas in fact the majority of those problems occur outside the realm that ZFS can do anything about.
And the idea that Microsoft will see ZFS as something it must react to is laughable.
ZFS’s strengths (ease of storage management – at least until you start using redundant storage, marginal improvement in already-good reliability, good write performance, and the banishment of fsck – though it’s kind of late to that last party) are more than outweighed by its faults (brain-damaged fragmentation policies and RAID-Z design, lack of user – let alone user and group – quotas, serious scaling limitations by virtue of being confined to a single host node, requiring unreasonably deep indirect-block trees, and using centralized storage management, space demands of its full-path copy-on-write approach that prevent snapshots from being used as an effective substitute for ‘continuous data protection’, and failure to deliver on its alleged banishment of volume management when redundant storage is required, since disks must still be organized into mirrors or RAID-Z groups – leaving aside the very natural reluctance of serious users to adopt a not-yet-mature implementation). So most of the ‘promise’ that ZFS might once have offered got broken early in its design stage – if indeed it ever promised anything at all that wasn’t based on a somewhat flawed understanding of how storage really works and what hard problems still exist to be addressed.
ZFS reminds me strongly of the early years of Itanic in its reliance on hype rather than on substance – the main difference being that instead of being hard-sold by a couple of large, self-serving corporations it’s being touted by a volunteer cadre of enthusiastic and well-meaning but technically-challenged supporters. I didn’t appreciate the blatant over-hyping of Itanic (though my involvement was ignited by the way that Compaq lied and broke long-standing commitments in ditching Alpha), and I don’t appreciate it with ZFS (having been disgusted by Schwartz’s cynical attempt to turn a conventional inter-corporation patent squabble into an open-source crusade) – so as long as you continue blindly ignoring its faults ans over-selling its virtues, I’ll continue debunking that.
– bill
RE: “Hmmm – wonder who you actually are…”
They call me “nobody”…
E2EIoD (End-to-End Information on Demand) is what I am…
I enjoy your comments and the technical depth of them. I personally think your ZFS comments are a bit over-done but you have stated your reasons and they are valid, if personal.
Robin’s are also personal as mine are. We happen to like the feature/function set of ZFS at the moment. I do not speak for Robin. I infer that value judgment from his public comments.
I have no axe to grind with Ken Olsen. Never met the man but he was a giant.
Rod Canion of Compaq, another giant, used the same method of “management by committee” as DEC. When Sevin-Rosen removed Rod they cited an inability to respond quickly enough to the market. Sevin-Rosen was the venture capital firm that backed Compaq. If you are supplying the money no one can remove you. You can go broke and remove yourself.
I’m more interested in “Lessons Learned”. Some CxOs got the axe because they were “too quick” to respond to market changes. It is tough to get it right.
Take ZFS. The first new file system since ReiserFS that will make an impact in my SOHO. I loved the Global File System (GFS). Red Hat still offers it but it is not being updated. No one wants it. GFS was the most Storage aware, Storage centric file system I had seen up to that time. For sheer throughput XFS won all my benchmarks hands down. Its throughput was only limited by the hardware.
Do I want to commit my IS (Information Stack) to the risk of using a file system that there is no clear support for? People were hopeful that ReiserFS would mature and become a major player commercially. Not going to happen now.
Who can afford GPFS? Does it really work as well as people think? Is it targeted at the general market or HPC (High Performance Computing) only? What are my options? Is ZFS an option? People seem to like its Storage awareness and tools. We have to wait and see if they stand the test of time and very diverse use.
The whole world is crying for a file system that cares or at least seems to love us.
ZFS is a step in that direction…
What I am looking for is the ability to talk directly to OSD (Optical Storage Device) with REST like Amazon S3 offers or SOAP like Google offers. And some others. It will take either a very “smart” file system (as we know them) or a very “dumb” one to do this. Depends on the engineering trade-offs.
This give me direct object addressability and the potential to create Information address URLs that are “vector” in nature rather than scalar only. Information objects (vector) and information objects (scalar) would then have trajectories based on Business needs (BPM).
No file system is even talking about handling this type of object addressing and “vector” metadata, AFAIK.
I really don’t need to know what the file system is or care. ZFS lends itself to this nicely.
As does Flash disk. Rotating rust could slowly move safely (low IS risk) to the back of the bus while Flash is maturing.
I believe this is the “Wave of the Future”.
Ah, yes – Ben Rosen: he also orchestrated the removal of Eckhard Pfeiffer (who had rescued Compaq from its doldrums in the early ’90s) when Pfeiffer started actually trying to leverage the technology that he had acquired with DEC rather than concentrating solely on the low-margin PC biz – and replaced him with the inimitable ‘Curly’ Capellas, who began his reign with the start of a clandestine two-year plan to kill Alpha (all the while reassuring customers just how fully ‘committed’ Compaq was to it and encouraging them to commit their *own* futures to it) and proceeded to ride Compaq straight down toward the ground until he was rescued by Carly and nimbly escaped with his own well-prepared parachute.
As for ZFS, I’ve found that after my first *very* positive reactions the more I get to know it, the more it disappoints. By contrast, ReiserFS was a continual font of innovation, even if not all of it exactly hit the mark it was aimed at (and given the shoestring on which it was developed it has considerably more excuse for this lack of perfection): it certainly broke new ground in efficient management of small files and the use of generic clustering mechanisms for improved aggregate access performance, without sacrificing much in the way of conventional performance in the process.
GFS was kind of a mongrel, though. I first encountered it in 1998 when it was largely a vehicle aimed at leveraging a new disk feature that its developers had sold Seagate on as a product differentiator (disk-based distributed locking mechanisms). This idea was completely bananas (not only was it difficult to map locks to and poor in scaling, but it performed worse than a pure software distributed locking solution), and in fact GFS eventually abandoned that aspect of its nature and adopted IIRC a distributed locking approach somewhat similar to VMS’s, but for as long as I remained marginally acquainted with it it never seemed to understand how to approach things like distributed caching efficiently, so I’m not surprised that it never enjoyed great popularity (and curious about what you specifically think made it uniquely ‘Storage aware, Storage centric). My general impression was that its originators had been somewhat more interested in feathering their own nests (both academically and financially) than in pursuing technical excellence, but perhaps they were merely enthusiastic and inept (in fact, something like the ZFS architects seem to have been – so I’m curious why you consider ZFS to be especially Storage aware as well).
‘Optical storage device’, or ‘object storage device’? I haven’t recently looked at the definition of the latter, but a decade-plus ago when Garth was defining it as a single ‘smart disk’ it was about as silly as disk-resident locking – and while the advent of cheap on-disk flash helps alleviate one of the most obvious obstacles it’s still pretty silly today. Single disks just *aren’t the right level* at which to implement this technology, though Lustre got it reasonably right by doing so at the server-node level.
It’s easy enough to create a file system that supports individual objects, though – and even that clusters them effectively (with application guidance – again, see Reiser for one possible approach here, though my own inclination is that you don’t need a clustering mechanism that’s orthogonal to a conventional directory structure but can get by equally well just by using directories as the clustering mechanism – i.e., to govern clustering by object naming, while retaining the underlying object ID for random access, possibly through a ‘secondary index’). Supporting object hierarchies (let alone more general directed graphs) transparently is another matter entirely, though. I’d be curious to know exactly what you’d like to be able to do in this area: your brief sketch above suggests that you might be asking for organizational mechanisms in the storage system that I’d be more inclined to think belong in application logic.
– bill
— Error Correction —
“(Optical Storage Device)” should read “(Object Storage Device)”
Thanks for catching that, Bill.
RE: “your brief sketch above suggests that you might be asking for organizational mechanisms in the storage system that I’d be more inclined to think belong in application logic.”
That is exactly correct.
I also want backups to be done in the Storage device instead of on some computer stack. Like EMC Calypso did. A backup is one form of a replication.
Storage vendors cringe at the mention of this.
IMHO, the bandwidth would be better spent doing eDiscovery on the Storage “object” Content and other tool tasks like the currently highly popular deduplication (I cringe).
Apparently Storage box bandwidth is severely limited by the creative genius of the designers. Or maybe it is cost. Bandwidth scales cost directly. Maybe even with a multiplier.
I am afraid if the object interface is left to the applications area we will have a bigger mess than EAI (Enterprise Applications Integration) was.
Something will emerge. The CORBA, COM, DCOM, etc. standards have all been blown away by Web Services and Web Frameworks. You just define your own “bean” and off you go until you have to talk to someone else or have to merge the acquisition of a company or alien Information, etc.
We might get some standard object definition for the “Stored” object. Once in the application you can do anything with it you want. When it comes time to “Store” it again it has to conform to some Storage standard to be read successfully by anyone.
“Cosmic”, “Galactic”, “Interplanetary”, “Multi-planet”, “Single-planet” and Global name space definitions, processes and procedures have to be dealt with.
Once a Storage object is created its name lives forever. The ownership and Content may change as it is redeployed.
Like telephone numbers. You have to get, or query, some object wrapper Information to get Expiration, Persistence, and current owner Information. Some people have been doing this with XSLT for some time.
The technical solution is not the show-stopper. It is getting the agreement from the people involved, as with all standards. Many people feel that “Standards” cost them their “market edge”.
I view these as “Lower Metric Events” that should be managed by the Event Management System (EMS). The EMS is under the control and supervision of humans subject to the humans correct interpretation of the events reported.
When a replication runs I only want to know the results. Successful? How Successful? Not Successful? Mission Critical Human Intervention Required? Human Intervention Required? Why Not? What Processes were used? What Resources were used? How was the Well-Being Index affected? Did ROI/TCO for that LOB (Line Of Business) Change and How?
These are the important issues in determining how IT affects the bottom line.
Is IT a “Profit Enabler”, a “Profit Enhancer” or just TCO?
I don’t care what Operating System, File System or hardware enables this.
As long as it meets SLA specifications for Reliability and Validity.
This may not be the ideal location for this discussion, but as long as Robin doesn’t mind and since it started here …
“I also want backups to be done in the Storage device instead of on some computer stack.”
A ‘storage device’ can reasonably back up only what it understands. E.g., a block storage device can’t *reasonably* perform file-oriented backups: it’s just not worth the interface complexity that a general implementation of this would take, given that a file-oriented storage device would be a much better fit for this. Now, if the file-oriented device supports file and directory attributes that can be set individually to control how backups run (e.g., which files are backed up and how often), that device could reasonably use that information to perform bulk backups – but the implementation of such a ‘storage device’ might well just be a server system that understood how to handle the transfer to the backup device while continuing to serve its clients.
Running data – even bulk data – through a ‘computer stack’ is not that expensive compared to the overhead of moving it on and off a platter, so the main reason for avoiding passage through such a stack is to allow that stack’s bandwidth to be used for other things, and the client/server paradigm does this fairly well (at least in terms of keeping the load off the client’s stack).
Another function that can be better implemented in the storage rather than at the client is the COPY function.
“IMHO, the bandwidth would be better spent doing eDiscovery on the Storage “object—
I have no idea what you meant by the above.
“Content and other tool tasks like the currently highly popular deduplication”
If you mean that the storage device (or distributed system) should handle data deduplication, I agree (NetApp already does, for working storage as well as for backup storage) – at least up to a point (deduplicating files could piggyback on a file-structured background integrity scan, but might have unreasonable overhead if performed more aggressively).
“Apparently Storage box bandwidth is severely limited by the creative genius of the designers”
There’s often oodles of bandwidth available *inside* the box, and to some degree limited bandwidth into/out of the box can be compensated for just by using boxes in parallel.
I’m still not clear about what you’re specifically looking for in terms of support for ‘objects’:
Would a permanent unique object name in the form of a UUID/GUID be acceptable?
How important is it that the storage device understand relationships (e.g., hierarchical) among objects? Such understanding need not be embedded in the storage device itself to make it sharable (e.g., applications could use a common access library to achieve it – and embed the information about which such library to use in the database such that at least it’s to some degree self-describing), nor would embedding it in the storage necessarily improve performance all that much if the cost of simply sending a request is small compared with the cost of executing it (at large client/server separations this becomes less true, but that could if necessary be ameliorated simply by creating an intermediate intelligent server processor near the storage). Understanding inter-object relationships *in the storage itself* could be a real interface can of worms – as well as being difficult to extend as new needs became apparent.
Similar comments apply to understanding (e.g., validating) the internal structure of objects – though there may be exception cases where the system uses portions of the internal structure of objects to implement its desired features.
Some objects (e.g., conventional files) must be ‘heavy-weight’ objects that contain information controlling their access permissions and timestamps, perhaps arbitrarily-extensible user and/or application commentary, and even their identity (e.g., a UUID/GUID). Other objects (e.g., conventional records) cannot efficiently tolerate the burden of maintaining or even just including these features (though including a bit-per-feature menu of what’s present might not be unreasonable) – but may then have to inherit at least some of them (e.g., access permissions) from something like a ‘container’ object that has them (and propagate things like timestamps back to that container object). The system obviously has to have a hand in maintaining some of these attributes and at least providing access to others. What other similar needs may there be in this area?
With universally unique object IDs you don’t necessarily need separate namespaces within the storage system (aside from those that you can create from, e.g., a directory hierarchy).
But I still feel as if I may be missing entire dimensions of what you’re talking about.
– bill