StorageMojo

Update: It is official. See here. Then come back and read the rest of this post. Thanks to alert reader Oskar for the tip.

French website Mac4Ever reports - thanks to Babelfish translation:

. . . the few innovations of Leopard, one read these last months, several rumours on the integration of a new filing system, the ZFS, which presents a revolution in the field of the storage of data. This rumour seems to be confirmed, since one sees finally apparaitre in the utility of discs of Leopard, an option to create images and partitions in this format.

They include this section of a screenshot:

Which seems pretty convincing. Or at least someone had fun with Photoshop. For the record, I believe.

I forget, so why is this cool?
See ZFS On Leopard: How Cool Is That? for more details, but the main coolness is:

• No more disk utilities. The entire data store is self-validating.
• No RAID controllers needed: ZFS gives you fast RAID for free. Just add drives.
• No more volumes and, therefore, no more volume management. ZFS eliminates the whole volume concept. Add a disk to your system and it joins your storage pool. More capacity. Not more management.
• Easy, fast backup through snapshots which means that Time Machine could give you a view of your data every hour on the hour, all day long.

But That’s Not All!
For in-depth treatment of ZFS see here and here. Includes links to more technical info and benchmarks.

FWIW, StorageMojo called it first . . .
Competitive analysis is not for the faint of heart, and I confess I wavered on whether or not Apple was really doing ZFS, until the iTV announcement three months ago. Then it all became clear - to me - while the silence from the rest of the Mac community was deafening (see Means, Motive & Opportunity: Apple Kills the Media Center PC). I even tried to bet the inimitable John Siracusa of Ars Technica a drink - only to find he doesn’t bet or drink!

The StorageMojo take
Why is this even slightly important? For the same reason I said four months ago:

StorageMojo.com has devoted time to this issue because today’s computer business is largely driven by consumer computing, not enterprise computing. Putting a really modern integrated file and storage management system on a consumer OS would raise the bar for everyone else.

No doubt Apple will first recommend ZFS for Mac OS X Server, just as they did with HFS+. Yet it won’t be long before it becomes the default file system, and not a moment too soon.

Update: I updated this post with some additional explanatory material, for those few people who haven’t been following ZFS and Leopard as avidly as I have.
YAU: Looks like Leopard’s Disk Utility also supports iSCSI natively. Another plus.

Comments welcome, of course.

Remote PC backup made easy
Walt Mossberg of the Wall Street Journal reviews (subscription required, I believe) two remote PC backup services, Carbonite and Mozy, this morning.

Mozy: now they have something
I wrote briefly about Berkeley Data Systems, parent of Mozy, in September and concluded

They compete with Carbonite with a twist: like a friendly pusher, the first two GB of backup are free. Cool. Now they should just do away with the 30 GB limit on their $5/mo plan and they’d have something.

As of today they’ve dropped the 30 GB limit and now they have something. But they’ve also dropped the reference to “. . . potential telekinetic security breaches.” These guys are getting serious! Darn.

Mossberg’s money quote:

Of the two products, I prefer Mozy. Carbonite is a little quicker and simpler to set up, but it’s more limited. If you want to go beyond the default backup choice — your most common documents and settings — you have to troll through your hard disk to select additional folders and files for backup. Mozy also has a default setting, but makes it much easier to alter or customize it.

Mozy offers more-versatile restoring and scheduled backups, and unlike Carbonite, will back up an external hard disk. Mozy will also send you a DVD of all your files, for a fee. Carbonite won’t. Mozy also keeps multiple versions of any file for 30 days. Carbonite doesn’t.

Still, you won’t go wrong with either of these two services, and you’ll sleep better at night.

I agree. None of the other online backup services comes close to either Mozy or Carbonite on price, ease of use, or features. Every business laptop should be backed up to one of these services so IT can focus on more important things.

The StorageMojo take
I’m glad to see Mozy took my most excellent advice and dropped the 30GB limit. Most home users will never come close to that, as I’m sure they’ve figured out. Sadly, neither company supports Macs, although Carbonite is on track for support in April. I’ve pinged Mozy for their date and if they respond, I’ll update this post.

Update: Josh Coates, the founder, says that MacMozy is in alpha right now and they “. . . hope to release it as a beta to the public in the next couple of months.” So us MacHeads may see MacMozy in April as well. Not as good as Right Now, but better than Never. Thanks, Josh.

Comments welcome, of course. Moderation turned on to defeat the banality of the Empire of Spam.

The good people at Texas Memory Systems read StorageMojo.com and at my invitation offered this response to my post An SSD For The Rest Of Us. I think they did a pretty good job of laying out the, IMHO, historically under-appreciated RAM SSD. Naturally I have a couple of comments at the end.

*Dear Storage Mojo:*

*We are excited to see solid state disks get some coverage in your blog even if we might have different ideas about the DDR RAM solid state disk market.*

First, we like flash memory too. We like that it has high density and is non-volatile. It provides a great storage media for USB drives, cameras and even iPods. We would even want one in a notebook so that it will survive the fall a hard drive cannot, and the low power usage really adds value when you are running on batteries. The fact that all of these industries have adopted flash has led to a big drop in its price. The success of flash has cost someone some business… hard disk companies making microdrives. Interestingly, we have yet to feel any competitive pain from flash drive manufacturers.

It is worth pointing out that companies have been promoting flash memory systems for commercial business applications for years. Their prices have always been lower, their densities higher and they have always been non-volatile. Inspite of these factors, none of these companies have traction in the enterprise applications area that we sell into.

Why is that? Flash disks use flash memory and flash memory has some inherent limiting characteristics:

1. it’s write speed is poor (comparable to disks)
2. inspite of the best efforts of the industry it is still easy to wear out the write cycles, and finally
3. the read speed is still a lot slower than our read speed

As an addendum to this note, we have included a description of the hard work a flash drive has to do just to get a write done.

Compare the performance specifications of the Samsung flash drive to the RamSan system:

RamSan peak IOPS: 400,000 (read or write, random or sequential)*

Samsung peak IOPS: 2,200 (Webserver IOmeter pattern) (reads only). Note that for the Database pattern it is actually lower then most single hard disks (~85 IOPS).

A flash disk in a notebook is hardly likely to wear out the write cycles from operating system boots and the occasional save of a document, but that same disk in an enterprise storage system will wear out well before the ROI is realized. If we could write 400,000 IOPS to a flash drive (which we can’t because they are way too slow) it would take us seconds to wear out the drive).

Flash systems will inevitably gain traction with the lower end of the solid state disk market but as the last few years have shown we are not really stealing market share from each other as much as we are both growing our respective markets. The flash drive companies are growing and the intensity of that market is reflected in the pace of acquisitions. Independently, Texas Memory Systems has grown its DDR RAM solid state disk market over 50% this year. The future of our combined markets has never been stronger.

Regards,

Texas Memory Systems

The StorageMojo take
TMS makes a number of good points. The most powerful, IMHO, is that their RAM SSD is a big honking fast machine. Stick 8 FC ports on one and you might actually come close to those 400,000 IOPS. There really is no substitute for a big RAM SSD if that is what you need. TMS is focused on the high-end and flash is not a threat there - today.

On the other hand I detect a bit of whistling past the graveyard bravado in the TMS reply. When a product comes along on the steep downward pricing curve that flash has - much faster than disk or even RAM - that offers some of your significant advantages - non-volatility, good performance (for some apps) relative to disks - at a fraction of the cost, you have to think hard about the possibility of substitute architectures emerging.

Conjecturally, such an architecture might replace the implicit centralized schema of the TMS FC products with a distributed, shared-nothing plan. Local attach is both faster and cheaper, so if a web server farm of pizza boxes were equipped with flash SSDs, one might see a sizable fraction of TMS performance at a much lower cost without the FC management headaches. Other, less obvious, options no doubt will be invented.

Once people see a much cheaper alternative to RAM SSDs, the juices start flowing and inventions occur. If I were TMS, I’d ask a couple of my better engineers to work part time on creative flash-based SSD architectures. Somebody will, it might as well be you.

Fighting rear-guard marketing actions - protecting a declining product or technology to keep the high-margin revenue coming in - isn’t much fun, but it sure is profitable. Especially in an area as conservative as storage. So it is a big deal when someone quits trying.

Which is why I’m so [shocked/surprised/pleased] to see Gudmundur Einarsson, CEO of Tandberg Data face reality and admit the facts in a press release:

“Using RDX QuikStor, organisations that currently rely on tape to protect, archive and interchange their data, will benefit from enhanced performance, instant random access, high reliability and, most importantly for small businesses, a lower cost than competing low-end tape solutions. . . . ”

Low-end RDX product - but can the high-end be far behind?
The RDX product packages cheap, 4200 RPM, 2.5″ disks in what T’berg calls a rugged, shockproof cartridge with a 10 year life in 40, 80 and 120 GB sizes. The unit starts at about $300, so the margins look good for T’berg and the resellers should be able to move lots to the SOHO and SMB markets.

Calling high-end tape vendors
Tandberg stresses that the RDX is a low-end system. Yet people have been predicting for years that disk would overtake tape. How close are we? As the table shows, pretty darn close.

Alert readers will notice that DLT is much costlier than LTO, reflecting the premium tape cost that includes Quantum’s lucrative licensing fee and the inability of an aging architecture to keep up with the newer LTO. Which is why DLT is well on its way to that big archive in the sky.

I’ve also left some signifcant costs out. To put an HDD in a removable enclosure runs about $40, quantity 1. And I’ve left out the cost of the tape drive, which for these top of the line Ultrium 3 starts north of $2,000. Also, I don’t take into account the 2:1 compression manufacturers commonly assume, and that customers may or may not see.

The StorageMojo.com take
At the rate disk prices drop, this picture will look very different in 6 months, when the 1 TB drives are shipping and the industry shift to perpendicular recording is complete. And if the folks trading at Storage Markets are correct that we will see GB price parity next year, then 2.5″ will become very attractive as a backup medium when properly packaged and priced. The tape business is beginning to unwind, and while it won’t go away any time soon at the high-end, it brings almost nothing to the SOHO and SMB market space.

Expect to see the D2D data reduction companies do very well over the next three years.

Comments, as always, welcome. Moderation is turned on to keep out comment spam, and registration has been turned off to make commenting faster and easier.

Apple’s “sneak preview” of iTV raised questions for most folks: why announce now? Will people download video like they do audio? Will the other studios join up?

For me though, iTV answered a big question: is Apple porting ZFS to Mac OS X?

So I’m calling it: yes, Apple is porting the very cool ZFS to Mac with intent to kill the Media Center PC.

We do have some facts: Chris Emura, of Apple’s CoreOS group, expressed interest in porting ZFS to Mac OS X; we also know that the first developer’s release of the next version of Mac OS X, Leopard, is ZFS aware, which means someone, somewhere, is working with ZFS.

Still, Steve Jobs hasn’t stood in front of thousands of Mac faithful and said the magic words: “One last thing” to announce ZFS. Despite the solid evidence I’ve noted, there is no smoking gun. So this is a circumstantial case, and a very strong one, because billions of dollars are riding on this.

Regular viewers of Boston Legal or Perry Mason know that circumstantial cases are presented when conclusive evidence is missing. You demonstrate means, motive, and opportunity. Apple has all three, in spades.

Means: Can Apple Do It?
Easily. ZFS is an open source product from Sun Microsystems. Darwin, the Unix operating system at the core of OS X, is designed so different file systems may be plugged in. Unix, after all, began as a research tool and file systems are a continuing area of research.

More importantly, Apple has already ported one Sun open source product, the very cool DTrace and layered on some UI Mojo to make it sweeter. ZFS has lots of features that cry out for an intuitive user interface. Cheap snap shot copies: Time Machine. Creating RAID systems or restoring failed disks are others.

Opportunity: Does Apple Have Time To Do It?
You might think they have lots of time, because who cares when a replacement for HFS+ arrives? And you would be very wrong. Because once we understand the motive, we’ll see that Apple needs to get this done by early 2007: in time for iTV. A senior Unix developer estimated that porting ZFS from one OS to another would take at most a year. Full integration might take more time, but since when did all features ship Day One anyway?

Motive: Why Would Apple Care?
Motive has always been the real problem. Even other defenders of ZFS had to stretch for a reason. Journaled HFS+ isn’t perfect, but it is competitive with NTFS and the other common filesystems out there. My original thought was “here is this great free product so why wouldn’t you use it.”

Well, as others have noted, while plugging in a new file system isn’t that hard, there are a host of issues that also need investment, such as migration, case-sensitive file names and creating the front ends for all the cool things you can do with ZFS. Steve may not care much about plumbing but he is all over user experience. Migration in particular is difficult for home users who don’t have empty external hard drives.

Now We Know
The motive is now clear: HDTV content to feed iTV, the pre-announced Mac Mini look-alike due in early 2007. With HDMI to speak to HDTVs (and preserve DRM) this is an Airport Express for video. How does this impact storage?

Video Downloads: Big and Getting Bigger!
Here’s how. Imagine you’ve built the world’s largest and most successful online music store and sold billions of dollars of hardware to play that music. Each of those tracks cost $0.99 and is 3-5 MB each. People can easily back them and even if they have a few hundred, it is maybe a GB or two. Easy to back up on a few CDs or DVDs. And they are on your iPod anyway. So HFS+ burps on your music and other than yelling at an underpaid Apple tech support guy, what are you going to do? If it wasn’t backed up, whose fault is that?

Enter The Terabyte Media Collection
Now you want to build the world’s largest and most successful online video store, with DVD and HDTV quality content. You are a little ahead of the market, but that usually works out. You want people to buy movies as freely as they now do tracks. Yet there is the scale problem: movie files are 1000x the size of audio or photo files. Not only that, the studios don’t want you to back them up to DVD or anything else.

“Halfway Through T3 the Hard Drive Started Clicking?”
iTunes music is automatically backed up if you have an iPod. Movies aren’t. Movies are large - 1 to 2 GB today - and much larger with HDTV and DTS sound. If you want people to store and play movies digitally, both purchased and home video, they need safety and capacity. No disk tools. No RAID set-up. No volume management. Suddenly storage quality and ease of use becomes a critical success factor for a new billion dollar business.

ZFS Is The Answer
Steve Jobs has two questions. First, how can I sell more online content and equipment to play it? Second, how can I kick Microsoft’s butt? By solving the high-capacity storage problem for HDTV content way better than Microsoft can, he’s got a great answer to both questions. He’ll never utter “ZFS” to a starstruck MacWorld audience. But he will wheel out a half dozen features, like Time Machine, based on ZFS, that will instantly become must-haves for the home digital media center.

Ladies and Gentlemen of the Jury
Apple Computer had the means, ZFS; motive, a big market; and opportunity to murder the Media PC. The state asks that you find Apple guilty in the first degree.

Update Turns out that HFS already has support for case-sensitive file names. It is in an extension called HFSX. For (way more) info check out this Apple Tech Note.

Comments Welcome, As Always

Almost two years ago I wrote about the catfight between disk and flash (see The Limits of Flash). It’s an entertaining market because we get to see the cut and thrust of the disk vs. semiconductor brawl at its rawest. I marveled then at thumb drive prices that were only $70 per GB down from $1000 per GB a few years earlier versus a disk drive at $40 per GB.

Rescan The Market
So this morning, when I saw that Buy.com is offering a 4GB USB disk for $45 after rebate, I decided to check out the low-end of the thumb drive market. And sure enough, I found some end-of-life 4 GB flash drives online for $38, or $9.50 per GB. I marveled anew.

Flash’s Downside
Flash drives are much smaller, more rugged, lighter and, their downside for the cheapest drives, slower. High-performance flash drives use dual-channel controllers and two flash chips in a RAID-0 configuration to get higher performance and, since two chips cost more than one, are a little more expensive. What surprises me is what a lousy job the vendors do of educating people about these differences.

All Flash Drives Are NOT Created Equal
This is a price sensitive market, so many vendors source their chips on the spot market, so they really don’t know what combination of controller and memory chip they’ll have. Thus they don’t make many promises. The more expensive flash drives are lower volume and thus use the same components. You can tell difference because the vendor will usually make explicit performance claims.

Net Net: Disk Is Losing At The Low-End
The economic trends are clear and compelling. In less than two years, low-end flash prices have dropped by more than 85%. In the same period the competitive disk product price per GB dropped “only” a little more than 70%. IC economics vs mechanical device economics. Unless the disk guys figure out something great in the next few years, flash’s 15-20% annual price improvement over disk will keep it winning more business at higher capacities every year.

This is is going to be a long fight and us consumers will benefit enormously.

Comments welcome, as always.

Over at InfoWorld, Tom Yager has posted a fascinating article titled Linux will get buried.

It’s Not About Apple
He proposes that Apple’s Unix revenue will overtake commercial Linux factory-install revenue by mid-2008. Which seems reasonable: Apple sales growth is on a roll. Apple OS X revenue per user is twice that of Windows. And Apple ships several million units a year. But StorageMojo.com doesn’t have a dog in that fight, so whatever.

The Linux Backplane
What did catch my eye is Yager’s take on what Linux really means:

. . . Linux is . . . a kernel, not an application platform. Linux is a backplane for device drivers, file systems, protocol stacks and low-level programming interfaces. It is a substructure for application services. The Linux kernel is . . . commercial quality and familiar. It crosses architectural boundaries cleanly. It bulks up and strips down in the time it takes to recompile. . . . It’s a standard. . . . Push a button, you’ve got an enterprise database, configured, loaded with sample data and listening for connections. Want a J2EE server with that? Flip this switch, it’ll unpack itself, sniff out that database you installed and mate with it.

There are a number of enterprise software backplane companies, such as Tibco and webMethods. Backplane Linux isn’t in their league, yet the ability to create specialized and optimized appliances from free software sounds very promising. Especially for SMBs and the VARs who serve them.

I got a taste of that wonderfulness with the one-click WordPress and MySQL installation at DreamHost (note: I get a commission if you use the handy “stomojo” coupon under the blinky ad - go ahead, it’s a good deal) which is Linux-based. I have no idea what goes on under the hood at DreamHost, but easy software installation and configuration is very attractive to this “knows all this stuff is there and still doesn’t like messing with it” guy.

The Really Cool Part
Then Yager goes one step further: the freeze-dried appliance on a flash drive.

Imagine that your server room has a bank of USB ports, and that every enterprise application you want to run exists, pre-installed on a stripped, standardized Linux, and in a freeze-dried state, on a flash drive. Plug in a drive, and within a few milliseconds you have a self-contained instance of an enterprise application. If you need more database instances, put in a blank flash drive and tell the existing database instance to replicate itself.

Tom is thinking of a datacenter, yet I could see this working with SOHO/SMB networks as well. The VAR has a variety of freeze-dried apps, like CRM, VoIP phone systems, job-shop resource planning, etc. plugs it into a cheap server and voila, the app is up and the VAR knows exactly how that system is configured for ease of maintenance. User-land configuration, like GUI options, accounts and ACLs get stored on the flash drive. Need application security? Unmount the flash drives and lock them up in a safe.

Please, Pick Holes
in Tom’s or my scenarios - or better yet, extend them with other ideas about applications. It sounds cool, but maybe it is one of those things that sounds cooler than it is. Your comments welcome.

Update: In the comments Mark claims that the ZFS reference in Apple’s Leopard, the next version of Mac OS X, proves that Apple is, in fact, working on porting ZFS:

All filesystems on MacOS X are implemented using what’s called VFS plugins, these interact with the kernel. The vnode is the focus of all file activity, that list tells me that the vnode is now ZFS aware which means there is or was a corresponding VFS plugin in a lab somewhere.

Look at the file list, those are all the VFS types the MacOS X XNU kernel supports…+1. If they’ve added ZFS to the vnode.h list it means it was or has been in development. If it was in development it’ll vanish from that list when the next build is released, if it still is in development it’ll stay.

I haven’t seen this interpretation anywhere else, so I’m off searching for independent confirmation, as I don’t have the Unix chops to make heads or tails of it. Yet for those of us who want to see computing’s state of the art advance this is a heartening thought.

Thank You, John Siracusa
Ars Technica’s John Siracusa went out of his way to make me [where "me" = the class of folks who thought Apple's new Time Machine meant ZFS on Leopard] feel better in his recent post Time Machine and the future of the file system when he said

. . . it’s seemed like accepted wisdom among the denizens of Mac web forums and blogs that Apple was moving to ZFS. Time Machine seemed like an official confirmation of what everyone expected. Just google for “zfs leopard snapshots” to see how many people came to the same conclusion when Time Machine was announced. All the pieces fit. Too bad it’s not true.

The snapshot/ZFS revelation was debunked nearly as quickly as it sprang up. Although all of WWDC except for the keynote is covered by an non-disclosure agreement, the particulars of Time Machine’s implementation were some of the very first technical details to leak.

Leak Here First!
And some of that leaking first occurred here on StorageMojo.com, I’m pleased to say. See the comments to the post Is Apple’s Time Machine Built On Sun’s ZFS?. If you are wondering why ZFS is cool, see ZFS: Threat or Menace? Pt. I.

But Wait! There’s More!
Over at this Apple Insider thread someone had the bright idea to ask a Leopard beta user to search for references to ZFS. And guess what? He found one:

Originally Posted by shadow
Someone who has Leopard: go to Spotlight and search for ZFS. Got something?

Yes. There is no file system bundle for it, nor is there a mount utility or any other one (no fsck, now newfs, etc.). There is, however, a changed vnode.h:
code:
enum vtagtype {
VT_NON, VT_UFS, VT_NFS, VT_MFS, VT_MSDOSFS, VT_LFS, VT_LOFS, VT_FDESC,
VT_PORTAL, VT_NULL, VT_UMAP, VT_KERNFS, VT_PROCFS, VT_AFS, VT_ISOFS,
VT_UNION, VT_HFS, VT_ZFS, VT_DEVFS, VT_WEBDAV, VT_UDF, VT_AFP,
VT_CDDA, VT_CIFS,VT_OTHER};

I Have No Idea
What including ZFS in an incomprehensible list means, if anything. I don’t feel too bad, because none of the smart folks on that forum seemed to know either.

So, John, Why Do You Say That?
While not holding out any hope of ZFS on Leopard, John did allow as how

. . . what I did expect was a new file system from Apple. Not a port or a fork of an open source file system, but a brand-new, home-grown, kick-ass file system created by Apple’s own team of engineers.

John, why fan the flames of Apple’s NIH syndrome? With some exceptions, Apple is not a plumbing company, and file systems are plumbing. Sun supported the ZFS team for six years while they built ZFS. What is the chance of Apple mounting a similar effort? Somewhere between zip and near-zip IMHO.

The Takeaway
StorageMojo.com has devoted time to this issue because today’s computer business is largely driven by consumer computing, not enterprise computing. Putting a really modern integrated file and storage management system on a consumer OS would raise the bar for everyone else. Until something better happens along, ZFS appears to be the best option. Here’s hoping the Apple engineering gnomes are beavering away to port ZFS to an Apple OS near us. Soon.

One Terabyte Disk Drives Pre-Announced
According to Cnet a Terabyte drive to debut later this year. They quote Seagate and Hitachi Global Storage execs talking about their intention to announce one TB drives later this year. 2007 delivery is implied, in case you were thinking you could put one under the Christmas tree.

The usual questions arise: Who needs a one TB drive? Is one TB “too big”? Why does the storage industry persist in labeling disk drives using powers of ten (1000) instead of powers of two (1024)?

A Trip Down Memory Lane
The first one gigabyte 3.5″ disk drive came out in, if memory serves, 1993. Since there were still 5.25″ disk drives shipping with larger capacities, there wasn’t much question that people could use the capacity. Yet it was more expensive per byte, so the really cost sensitive folks, like EMC, stayed with the larger form factor. In fact, by the late 1990’s EMC was almost the sole customer for 5.25″ disks, taking something like 90% of the industry production.

For Databases, Disk Drives Are Always Too Big, Until They Aren’t
The database people felt that these drives were “too big” - somewhere another DBA is saying that about these drives - even as today they happily use 146 GB FC drives. When 18 GB drives came out, I had customers who insisted on buying 9 GB drives because 18 was “too big”. Perhaps it does take the database developers a while to figure out how to use the larger drives.

A Whole Gigabyte On A PC
In 1993 I couldn’t imagine what one would do with a gigabyte on a PC. I’m having that problem today with one TB on a PC, although I know I could rip my entire CD collection into a lossless format and easily use up even more capacity. Oddly enough, I won’t do that until I have some sort of backup capability, since I don’t want to invest the work in ripping everything only to lose it in a drive crash. So I’ll need something like 3 TB of home storage for that job.

Smaller Is Better, So Why Isn’t It Cheaper?
What I am wondering is when the industry will go to the 2.5″ form factor completely and drop the 3.5″ drive. Related to that I wonder why 2.5″ drives are so much more expensive than 3.5″ drives. Check it out yourself: a 40GB 3.5″ costs about 2/3 that of a 40GB 2.5″. The material costs should be lower, the motors smaller and cheaper, shipping and distribution costs lower. The read/write head cost would be the same, but not more.

Even the learning curve effects shouldn’t be all that huge given how many 2.5″ drives are built for laptops. Margins are higher, but that can’t explain the entire delta either. So what causes this huge differential? I sure hope the vendors aren’t getting together in some back room somewhere and divvying up the market. That would be wrong, illegal and very, very stupid.

I would appreciate it if a reader from the industry would explain this in a comment. Use an alias and bare all - I’ll never tell who sent me the email.

When Will We Switch To 2.5″ Drives?
I’ve been wondering about this one for a while. In The Innovator’s Dilemma, a must-read book for every storage aficionado, Christensen makes the case that form factor changes have been the prime mover behind the rise and demise of most disk drive companies. Now that we are down to so few vendors I doubt that will happen again. Most vendors make all sizes, unless, like Toshiba, the specialize in small form factors.

Time For The High-Performance Enterprise 2.5″ Disk
When I was with DEC’s StorageWorks group, the DEC disk drive folks designed a high-performance enterprise 2.5″ drive. They built some mock-ups of applications, such as an entire RAID array in a 5.25″ form factor. At the time the only interest was for laptop drives, where power consumption, not performance, was the critical success factor. So that effort went nowhere.

Yet it may be time for a change. Power issues are looming larger, and smaller drives are more efficient. SATA interconnects lend themselves to plugging drives on motherboards or into backplanes. The scarcity of accesses relative to capacity also points to a move to a smaller form factor. Ultimately though, it may be the continued market share gain of laptops and small desktop PCs that will provide the final push to EOL 3.5’s.

It’s The Capital Investment
Why? Because as production goes up, costs go down. Assuming the vendors aren’t engaged in price-fixing - never a totally safe assumption in the storage business - as the shrinking cost differential leads more customers to buy them and they overtake the current leader. The Mac Mini is an example of a “desktop” system using a 2.5″ drive. A Gartner Group report last year 2.5″ drive factory utilization is almost 88%, which suggests that the vendors are selling almost everything they can build. Nor does it seem they are rushing to build more capacity, unlike the LCD makers. Darn.

Question: Is Google Driven by GB/$ or GB/Watt?
Historically, it has been GB/$, not maximum capacity. With the focus on power and cooling (DC power for the data center, anyone?) dropping 3-4 watts off of 100,000 servers would save some real money - even with cheap Columbia River hydropower. Not to mention how much more tightly servers could be packed. I’m sure someone at Google has looked at this. If they moved that way it would be a big boost for the 2.5″ enterprise drive.

OK, Why ARE Disk Drives Rated In Powers Of Ten?
I’m not sure anyone really knows, but this gentlemen has one of the more entertaining theories. For the record, the terms kilo, mega, giga, tera, peta, exa, zetta and yotta officially refer to powers of 10, not powers of two. So the drive marketers are, for once, on the side of the angels.

I wrote this as a comment originally, and since I am sitting here at a bar that claims the World’s Largest Selection of Draft Beer I realized it would make a good post, where “good” is a measure of speed. I am in a race against time: my blood alcohol level; ability to write; battery life; and my desire to sample as many fine Belgian beers as possible. So pardon my recycling.

There Are Worse Things Than Statistics
To paraphrase, there are lies, damn lies and storage performance numbers - I hesitate to call them statistics for fear of giving statistics an even worse reputation than they’ve already got.

This came up when some devoted readers questioned the benchmarks used for the ZFS vs Hardware RAID numbers. I didn’t dig into the benchmarks (those Unix system results may mean something to you, but they give me a headache) Robert used for his test to see what the mix of I/O sizes are. The typical strategy for describing array performance is to run a test that will give the absolute best possible number for the attribute one is measuring.

Surprise: Vendors Use The Absolute Best Numbers They Can Somehow Justify
For bandwidth that means reading and writing really big files - which is fine if you are doing video production or 3D seismic analysis - and totally irrelevant for almost all common workloads. For IOPS numbers that means the smallest possible I/Os as fast as possible - which usually means everything is sitting in cache. While that is nice when it happens, that is also an unlikely event in the real world.

Welcome To The Unreal World
So other than storage marketing people being lying scum, what is the point of benchmarks that only reflect un-real-world performance? Consider all storage benchmarks as simply telling you what the absolute maximum you could ever see in that metric - the vendor’s guaranteed absolutely “will never exceed” number. If you have good reason to believe you’ll need more than that then be afraid - be very afraid.

So What’s Left?
In the real world, with a mix of I/O sizes and rates, you’d be shocked at what “performance” looks like. Running 2k I/Os on the biggest Sym or Tagma you can imagine - ’cause you certainly can’t afford it! - on dozens of servers across multi-dozen FC’s and I suspect you’d see, maybe, with luck, 100MB/sec of bandwidth. A 3ware controller would probably do single digits. No bad guys here, this is just the nature of the storage I/O problem.

The Industry’s Storage Performance Secret Decoder Ring
Like the all-too-obvious breast implants some women favor these days, storage performance numbers reflect what the industry thinks practitioners want, while practitioner’s admiring glances confirm how willing we are to be entertained by a polite fiction. Like a good action movie, we know it isn’t real, but it gets our heart pumping anyway. So what is really true about performance?

Roughly:

• Latency is usually lower with a smaller array, since you don’t have millions of lines of code and multiple switch operations to traverse
• IOPS scale for large arrays mostly as a function of parallelism - more I/O ports, more I/O processors, more cache, more interconnect bandwidth, more spindles - not because each individual I/O unit is blindingly fast
• There are only a few vendors of most of these components, so the big arrays are built out of commodity parts. Architecture and firmware are the major differentiators. So, for example, cache access times are fairly constant unless using expensive static RAM. FC chips come from what, two vendors? Microcontrollers from four? Disks from three? What do you expect?

Of course the price-point engineered stuff will be slow. But I bet there is little difference in per-port performance between an enterprise modular array and the big iron Sym’s and Tagmas.

I’ve never seen a direct comparison of single FC port performance across big iron and modular arrays, which also suggests that it isn’t all that different. If you have data that suggests otherwise I encourage you to post it. I’d love to be proven wrong.

OK, The A. V. Brother David’s Triple Has Kicked In, So What’s The Point?
To me, the point of the ZFS benchmarks is not the absolute numbers, which are respectable for either case, but that the money spent for the RAID controllers bought nothing. I’d argue that even if the software were 20% slower, you’d still want to lose the hardware RAID and its associated bugs, power consumption, cost and maintenance.

Dateline: The Yard House, Long Beach, California. And yes, I love Belgian beers and ales. As well as their chocolate. Don’t get me started on how badly Belgians market their great little country. Did I tell you about the bar in Bruges that has over 400 Belgian beers and ales on tap? Maybe another time. . . .

Update Over at TechRepublic, Scott Lowe offers another view of AoE here. If I were an SMB VAR, I’d be checking AoE out.

It Is About Time
Here’s a potential game-changer - especially for the SMB market. It is low-cost SAN functionality based on local Ethernet. From a company named Coraid. Available for Windows, Linux, Solaris, FreeBSD and Mac OS X.

Wait a minute? Isn’t that iSCSI? It is a block device after all. Nope. Different. IMHO, better. There are some Don’t Gets, and a lot of Don’t Needs.

Putting Local - And Storage - Back In LAN
Coraid’s innovation is the open ATA over Ethernet (AoE) protocol. The big Don’t Get is that the protocol isn’t routable - it is strictly local - no IP involved. So the Don’t Needs include no TCP/IP overhead, no TCP/IP offload engines, no CPU-cycle sucking and latency-inducing TCP/IP stacks. AoE sits right on the data link layer - level two - of the ISO network model, so with a switched LAN - is there any other kind these days? - you get very low latency and full network bandwidth across a low-cost, industry standard LAN.

The other big Don’t Get: expensive and finicky Fibre Channel HBAs, switches and storage, along with the extra bandwidth FC offers. Like FC, AoE appears to make very effective use of available bandwidth - maxing it out with storage traffic. You’ll want a dedicated storage network to run AoE across.

Practice Makes Perfect
Even though it is cleared for use with Oracle, it probably isn’t a solution, today, for habitually late adopters. You’ll need to think through your security and system management processes to ensure that data doesn’t get munged by an inattentive sysadmin. A dedicated AoE SAN is a start, and VLAN techniques can help partition off potential damage-doers. The key: it just looks like a disk, and anything goofy you can do to a disk you can do over AoE.

Write Once, Read Never?
So far it appears that Coraid is the only company building AoE hardware. It doesn’t appear they are trying to keep anyone else from doing it, only it just hasn’t happened yet. That might be a worry for some folks. So in a smart move, Coraid has a Linux tool called srcat a tool for recovering data from the raw disks on a Coraid JBOD or array. So if the company goes belly up, controller breaks, no replacements available, you can still pull the drives and use srcat to pull the data off. Neat.

StorageMojo.com Take
Congrats to Coraid for a creative way to bring the benefits of network economics to storage networks, just as some of us thought FC would 10 years ago. By creating an open platform and protocol, they’ve started the open-source equivalent of a SAN. If you require - or would like to be able to afford - a lot of storage capacity, you should certainly check these guys out.

Update: Over at Tech Republic, Scott Lowe offers some more info on AoE. The (literal) money quote:

AoE is cheap! An array capable of supporting up to 11.25 TB from Coraid starts at less than $4,000 without disks. Today’s price for a 750-GB disk at NewEgg.com is $400 and the unit supports 15 disks. So, for less than 10 grand, you can get 11.25 TB of shared block storage. If you do the math, that runs at about $888/TB or $0.87/GB. Not bad!

So I’m trying to make some lemonade to go with my crow steak after mistakenly tying Time Machine to Sun’s very cool ZFS. Given that Apple has incorporated Sun’s DTrace into the Darwin kernal, I still have hope they will do the same with ZFS. In my optimistic view, they just haven’t yet. As one of the clearly in-the-know commenters stated

ZFS is not in the Leopard discussed at WWDC in any capacity.

Which is a lot different than saying it will never be in Leopard.

Yet what some of the clearly better-informed-than-me readers say is valuable in and of itself.

What Is The Role Of Journaling?
One reader stated that Time Machine is enabled by journaling. Journaling has been available, in some way shape or form since Mac OS X Server 10.2.2. Yet the modest claims Apple makes for HFS+ journaling don’t square with the reader’s assertion:

Journaling accelerates the recovery time after an unexpected shutdown, significantly improving the availability of server and storage systems. When journaling is turned on on a storage volume, the server automatically tracks file system operations and maintains a continuous record of these transactions in a separate file, called a journal. The operating system can use the journal to return the file system to a known, consistent state after a failure.

In essence then, journaling speeds up the process of consistency checking. There is nothing in journaling that squashes the standard processes of bit rot. There is nothing in journaling per se that enables Time Machine. It isn’t clear that even joining the record of operations with a backup application is all that important. So I don’t get where journaling fits in.

It’s A Backup Application With A Pretty Front End
More likely is this reader’s comment:

It works just like Backup 3. It creates a sparse image and then copies the files onto it in stages. It’s not ZFS or using any special filesystem tricks; it’s just copying files incrementally on a schedule.

This seems to get to the heart of the matter. Time Machine is a pretty front end to a backup application. The presentation focused on finding lost files, and said almost nothing about full restores, but they are implied.

The Limits Of Time Machine
Until I get my hands on it I won’t know for sure, but based on what readers have said I would guess these assertions are likely to be true:

• You can’t boot off a Time Machine backup
• If you create and delete a file before the nightly backup Time Machine can’t recover it
• If your system disk fails you must replace it AND perform the restore - which would take hours
• Requiring a second drive to find lost files is overkill for laptop users

StorageMojo.com Verdict
I don’t think Time Machine will cause me to change my backup strategy. The fundamental problem of disks going away is a greater concern to me than any one file. That is why I use CarbonCopyCloner to create bootable backups, because if my laptop drive fails I can be back up in minutes, not hours or possibly days - not a lot of SATA notebook drives in small towns.

Requiring a second drive to use Time Machine will be easy for Mac Pro users with multiple drives, but not for my MacBook. Time Machine is a great front end with a weak engine. Let’s hope ZFS is in the works.

Update: Sadly, it appears I’m wrong. See the update at the bottom of this post and the comments. It was fun while it lasted. And my congrats to the Apple Time Machine UI team still stand. Brilliant. The original post follows.

Lest The Suspense Kill You: The Answer Is Yes
First of all, congrats to the Leopard Time Machine team for putting a beautiful and intuitive face on recovering data from snapshot copies. It is no accident that Scott Forstall, who introduced Time Machine at Apple’s World Wide Developers Conference, has the title “VP Platform Experience”. The Time Machine interface is brilliant.

ZFS On Leopard
I love competitive analysis, and this is a great example of the process. As I’ve noted before (see ZFS On Leopard: How Cool Is That?, Apple Mojo In High-End Storage and Bring Me The Head Of WinFS ) there is documented interest inside Apple in ZFS (for more info on ZFS see ZFS: Threat or Menace?), the coolest file system cum storage manager out there.

ZFS is open source, so it fits within Apple’s miniscule storage budget, and it has a lot of great features that would kick the Mac’s storage capabilities way beyond the aging NTFS used in Windows Vista.

Some of those great features are reflected in the Time Machine announcement.

• Time Machine “makes a complete backup of all the files on your system” including system files
• Infinite snapshots - the basis for Time Machine’s “time-based browser”
• Totally automatic - you select the drive to backup to, select the items you don’t want backed up - and it goes

Teasing Out The Truth
Now, I’ve been in Apple’s position of wanting to talk about a future product without blabbing about how the magic works. I’ve gone through the announcement video and all the Apple webpages that reference Time Machine. They’ve been carefully scrubbed of any reference to the underlying technology. That is the first clue: the lack of chest-beating about this great new technology says it is not Apple-developed. Even with the DTrace announcement, where they buried mention of DTrace, they did claim some invention.

Clue #2: TM Is Creating Disk Clones
Looking at the list of features above is telling as well. The first feature could be implemented using the Darwin command-line utility ditto. While nowhere does Apple mention creating a bootable copy, if you copy all the system files, including the invisible ones, preserve symbolic links and resource forks - and they do stress the “complete backup of all files” - we can safely assume the backup copy is bootable. So either they are using a Unix/Darwin utility or they are copying the entire filesystem.

Clue #3: TM Is More Than Cloning
The ability to preserve a file system’s state at a particular time is known as a snapshot or point-in-time copy. The Mojo is conceptually simple:

• A snapshot copy is created
• Thereafter, any disk blocks that are changed are first copied
• When returning to the snapshot copy, the copied blocks are substituted for the newly written blocks

This Copy-On-Write (COW) strategy minimizes data bloat - only the changed blocks are copied. Time Machine’s name and interface make snapshot copy management intuitive. TM’s snapshot copy capability is not easily added to Apple’s HFS+, which is why it hasn’t been done to date. Yet it is integral, and cheaper than overwriting existing data, in ZFS.

Clue #4: It Is “Set and Forget”
All backup software is fundamentally a kludge, since it is backing up files rather than blocks. So backups need to be scheduled, the application needs to run, and if the system is turned off or asleep, backup doesn’t happen. None of this applies to TM - so TM is not an application - but a system service. It has to be part of the file system, not some asynchronous agent that checks after the fact.

Clue #5: Remember When Apple Bought That Storage Software Company?
Me neither. Time Machine is non-trivial stuff, way beyond what Apple has done with file systems before. If they don’t claim they invented it, and they didn’t buy it, then they must have gotten it for free. ZFS is most advanced and solid open-source file system out there, so it gets my vote.

Any Caveats?
Well, there was the FireWire drive icon on the desktop of the demo system. ZFS would normally create a storage pool out of additional storage, rather than working with an individual virtual drive. However, if I were Apple, I’d want to reduce the changes users see, and having drives appear on the desktop is expected behavior.

Also, the option to not backup files surprised me - since ZFS works with file systems and manages data blocks. How could ZFS know to not back up a file? My theory is that TM accomplishes this trick using the individual file’s or folder’s Access Control List (ACL). I’m not a software engineer, so there may be a smarter method, but it is certainly plausible.

You Heard It Here First, Folks
So I conclude that Yes, Apple is incorporating ZFS into Leopard. Which is very cool, not only for us backup deprived users, but also for enterprise users of Xserve systems. ZFS offers many more capabilities than Time Machine uses. Over the next 10 months Apple will have a great opportunity to polish those up as well to give OS X users an unparalleled storage and data integrity experience.

w00t!

Comments always welcome!

Update: John C. Randolph says I’m wrong (see comments) and while he doesn’t identify himself further, a moment’s Googling reveals a John C. Randolph, Sr. Cocoa Software Engineer, in Apple Worldwide Developer Relations. Assuming they are one and the same I must reluctantly grant that he would know.

What I would love to hear now is how HFS+ will be re-engineered to actually deliver the data integrity and reliability implied by Time Machine.

And John, thanks for reading StorageMojo.com and commenting.

Adventures in the Byte Trade
I said I was going to wait, but I wouldn’t be a technophile if I did. Despite maxing out the old StorageMojo.com laptop mainframe with RAM, an 80 GB hard drive and an internal slot-loading DVD burner, the wait times grew burdensome and a couple of my newer apps were barely usable. The new mainframe: a 2.0 GHz Intel Core Duo MacBook, with 512MB RAM, 60 GB hard drive and slot-loading DVD burner.

RAM Tough
Ordered on a Thursday, it shipped from Suzhou, China via FedEx on Monday (Sunday here) and, after stops in Anchorage, Indianapolis and Phoenix, it was delivered to StorageMojo.com Global HQ Tuesday morning. Naturally I wasted no time booting up the beast. It came up fast, and after backing up my 30+ GB music collection and then tossing it off the 80 GB drive, I used the Mac’s very cool Migration Assistant to move from the old 12″ Powerbook to the new MacBook, which took about 90 minutes. I rebooted, brought up my favorite apps and my brand new machine was cra-a-a-wling. Time for a RAM upgrade!

RAMbo-liscious
I knew this would happen - MacBook reviews said so - so I was prepared. I’d gone on my favorite RAM site, dealram after ordering the MacBook and found 1GBx2 Mac-spec generic RAM for less than $80 per stick. It was here when the new mainframe arrived, so after assuring myself the reviews were right, I shut down, unplugged, flipped the ‘Book over, removed the battery, and then swore at the #00 Phillips-head screws until they loosened. Removing the L-bracket, two little levers popped out. Extending them made the SO-DIMMs pop out and 60 seconds later the brawny new 1 GB sticks were firmly seated. More swearing at the micro-Phillips screws and another 5 minutes and the now maxed-out RAM was working its magic.

It’s A Hard, Hard, Hard World
Were buying a disk drive as easy.

One of the best features of the MacBook - unlike the MacBook Pro - is the ease of replacing the hard drive. In most laptops HDD replacement is very delicate surgery of the warranty-voiding kind. My fine motor skills aren’t the best, so I’d never done it. But with the MacBook, once you’ve finished swearing at the #00 screws you are there. There is a little white plastic tab tucked around the end of the disk drive which is easily levered up. Pull it gently and the SATA drive slides out. Remove the four #8 Torx screws holding the carrier to the drive, mount the new drive, slide it in, mount the L-bracket and you’re done. A five minute process if the screws cooperate.

Google 2.5 SATA 120GB -ATA -USB
I’m a size king. I like a lot of drive capacity since I know it will fill up eventually. So I wanted one of the new 160GB 2.5″ drives. Sadly, for the usual reasons they are in short supply and expensive. You’ll find that most places advertising them do not have them in stock, and those that do are close to $2/GB after shipping. So I opted for a 120, whose best prices are on the order of $1.30/GB.

I used Google’s Froogle service to locate the best prices. Google/Froogle does a poor job of weeding out what you don’t want though, so use the minus (-) operator to exclude listings that contain ATA and USB, or anything else that shows up. Then select the low-to-high price listing option. I looked at some other sites, like Pricescan, and didn’t find their vendor selection very broad. Froogle gets you a lot of hole-in-the-wall vendors that can be trouble, so think about your comfort level with risk and choose accordingly.

Buy RAM, Not Disk Performance
I also don’t care about performance. Of course I looked at the drive reviews at Toms Hardware, The Tech Report and Storage Review. Yet I find disk drive benchmarks unpersuasive, personally as well as professionally. Especially on a laptop with sufficient memory, the need for disk swapping is minimal.

Currently, the MacBook as been up for 24 hours and has seven applications running, 71 tasks and 237 threads. It has had zero virtual memory page outs and is using zero on-disk swap space. So a more expensive and lower capacity 7200 RPM would have saved me at most a few seconds of wait time. With only 1 GB of RAM there’d be a lot of swapping and page outs where disk performance would start to matter. Since the additional gig of RAM was $80, and a faster drive meant lower capacity and a $50 premium, I know I am better off buying the RAM.

Application Benchmarks For Faster Laptop Drives?
I didn’t find any. I’d love to hear why you made different decisions - if you did.

James Barton has has a simple tweak to improve disk performance on Windows machines.

How to Check Current Transfer Mode

• Open Device Manager
• Expand IDE ATA/ATAPI Controllers
• Double click on Primary IDE Controller or Secondary IDE Controller
• Go to the Advanced Settings tab to see the current transfer modes. If you see anything besides UltraDMA-6, and especially if you see PIO Mode, then

Go to James’ site and follow his directions. It involves editing the registry which can be tricky. So follow the directions carefully.