StorageMojo

I moderate all comments on StorageMojo and it is one of my favorite jobs. Moderation has turned into an unlooked-for pleasure, as I get to see the many insightful comments before everyone else. This morning’s came from David Vellante, who was at IDC for years. (Is IDC having a problem retaining bright people? I have no hard numbers but it seems like I’m seeing a rising number of good people leaving.)

Wikibon link love
David’s comment on The consumerization of IT is interesting:

The costs of storing a minute of audio are about the same as dial tone today and folks like Amazon and X-Drive (and I would argue Google with Gmail) are ‘double-dipping’ today using thin-provisioning-like technologies and easy-to-understand monitoring, metering and ‘chargeback’ software to condition consumers that storage consumption can actually be transparent and utility-like.

IT shops and vendors alike had better understand this trend.

Frankly, I’m not sure I understand it either, yet it certainly is provocative. Want to know more?

David thoughtfully included this link to a Wikibon discussion. The technique they use is interesting:

Decision making usually requires decision makers to make certain assumptions about future events. For example, “the past is prologue” is a common technique for assessing the future. Sometimes this technique works great, such as when predicting whether or not the sun will rise in the morning or set at night. Sometimes it doesn’t, such as when predicting whether or not the winner of the Kentucky Derby will win the Preakness. . . .

Essentially, instead of discussing “What will happen,” Future Perfect Peer Incite research meetings will assume that an event has transpired at some moment in the future and “look back” to discuss, “What happened?”

Which leads to Wikibon, infinity and beyond
So what the heck is Wikibon? I’m pretty sure it is a work in process. Wikibon seems to be a platform for analysts and consultants to strut their stuff and, without any one person necessarily breaking a sweat, creating good content that will bring people who are looking for the kinds of answers that community members are happy to provide.

The StorageMojo take
The ground is shifting under the IT analyst industry. Whether the analyst is helping a CIO make better business decisions or churning out content-free lead-bait, individuals now have an opportunity to attract their own audience. As capital-goods IT evolves into consumer IT, the need for core infrastructure analysts shrinks, while the new applications broaden and mutate. The brand-name on the analyst’s card becomes less important than what the analyst demonstrates every day in the flat-earth niche they work.

Clients, vendors and consumers alike, now have the opportunity to assemble virtual analyst firms. Pick the people whose thinking inspires and/or infuriates and engage with them. Find out what they think about your problems. Use what you can and leave the rest. And when the market changes, find the people who seem to get that and do it again.

Wikibon also appears to be partnering with Storage Markets, the innovative storage market research group. It may soon come to pass that the web-based independents will surpass the Gartners and IDCs in range and depth. All at a lower price!

Comments welcome, of course. I probably won’t attend the meeting tomorrow as I am getting ready to leave for Seattle. If you do attend, feel free to report on it here on StorageMojo.

But we knew it all along
Infoworld published a follow-up clarifying Apple’s inclusion of ZFS in Mac OS X 10.5.0. And doing quite a bit of tap dancing.

Apple now says that the ZFS file system will be available in the forthcoming Mac operating system, sort of.
Seeking to clarify a statement made on Monday by Brian Croll, senior director of Mac OS X Product Marketing, to two InformationWeek reporters that Apple’s new “Leopard” operating system would not include the ZFS file system, an Apple spokesperson indicated that ZFS would be available as a limited option, but not as the default file system.

ZFS “is only available a read-only option from the command line,” according to an Apple spokesperson.

In a follow-up interview today, Croll explained, “ZFS is not the default file system for Leopard. We are exploring it as a file system option for high-end storage systems with really large storage. As a result, we have included ZFS — a read-only copy of ZFS — in Leopard.”

“Read-only means that at a later date, if there are ZFS volumes, those systems would be able to read ZFS volumes,” Croll added. “You cannot write data into the system. It will allow you to read ZFS volumes later.”

Asked whether ZFS might be implemented for Apple’s Xserve rack mountable server line, Croll said, “Where we head in the future, we’re not able to talk about.”

Apple omerta aside, the direction is clear even if the timetable is not.

Update: ZFS clone on Linux: Chris Mason announced that he’s releasing something that looks like a ZFS clone for Linux.

[ANNOUNCE] Btrfs: a copy on write, snapshotting FS

Maybe you could help him. [Thanks, Wes.]

Now back to our regularly scheduled programming.

I’m pumped!
Next week I’m flying to Seattle to attend a one day conference on scalability hosted by Google’s Kirkland office.

It is a great set of presentations with leading edge practitioners. Here’s the agenda, presenters and edited descriptions of the topics:

• Keynote I: MapReduce, BigTable, and Other Distributed System Abstractions for Handling Large Datasets - Jeff Dean
  Search is one of the most important applications used on the internet, but it also poses some of the most interesting challenges in computer science. Providing high-quality search requires understanding across a wide range of computer science disciplines, from lower-level systems issues like computer architecture and distributed systems to applied areas like information retrieval, machine learning, data mining, and user interface design. In this talk, I’ll highlight some of the behind-the-scenes pieces of infrastructure that we’ve built in order to operate Google’s services.
• Breakout I: Lustre File System - Peter Braam
  This lecture will explain the Lustre architecture and then focus on how scalability was achieved. We will address many aspects of scalability mostly from the field and some from future requirements, from having 25,000 clients in the Red Storm computer to offering exabytes of storage. Performance is an important focus and we will discuss how Lustre serves up over 100GB/sec today going to 100TB/sec in the coming years. It will deliver millions of metadata operations per second in a cluster and, write 10’s of thousands of small files per second on a single node. If you like big numbers (but less than a Gogol) please come to this talk.

• Breakout I: Building A Scalable Resource Management Layer for Grid Computing - Khalid Ahmed
  We will show how to build a centralized dynamic load information collection service that can handle up to 5000 nodes/20,000 cpus in a single cluster. The service is able to gather a variety of system level metrics and is extensible to collect up to 256 dynamic or static attributes of a node and actively feed them to a centralized master. A built-in election algorithm ensures timely failover of the master service ensuring high-availability without the need for specialized interconnects.

  This building block is extended to multiple clusters that can be organized hierarchically to support a single resource management domain that can span multiple data centers. We believe the current architecture could scale to 100,000 nodes/400,000 cpus. Additional services such as a distributed process execution service, and a policy-based resource allocation engine which leverage this core scale-out clustering service are described. The protocols, communication overheads, and various design tradeoffs that were made the development of these services will be presented along with experimental results from various tests, simulations and production environments.

• Breakout II: VeriSign’s Global DNS Infrastructure - Patrick Quaid, Scott Courtney
  VeriSign’s global network of nameservers for the .com and .net domains sees 500,000 DNS queries per second during its daily peak, and ten times that or more during attacks. By adding new servers and bandwidth, we’ve recently increased capacity to handle many times that query volume. Name and address changes are distributed to these nameservers every 15 seconds — from a provisioning system that routinely receives one million domain updates in an hour. In this presentation we describe VeriSign’s production DNS implementation as a context for discussing our approach to highly scalable, highly reliable architectures. We will talk about the underlying Advanced Transactional Lookup and Signaling software, which is used to handle database extraction, validation, distribution and name resolution. We also will show the central heads-up display that rolls up statistics reported from each component in the infrastructure.

• Breakout II: Using MapReduce on Large Geographic Datasets & Google Talk: Lessons in Building Scalable Systems - Barry Brumitt, Reza Behforooz

MapReduce is a programming model and library designed to simplify distributed processing of huge datasets on large clusters of computers. This is achieved by providing a general mechanism which largely relieves the programmer from having to handle challenging distributed computing problems such as data distribution, process coordination, fault tolerance, and scaling.

Since launching Google Talk in the summer of 2005, we have integrated the service with two large existing products: Gmail and orkut. Each of these integrations provided unique scalability challenges as we had to handle a sudden big increase in the number of users.

• Keynote II: Description TBD - Marissa Mayer
• Breakout III: Stream Control Transmission Protocol’s Additional Reliability and Fault Tolerance - Brad Penoff, Mike Tsai, and Alan Wagner
  The Stream Control Transmission Protocol (SCTP) is a newly standardized transport protocol that provides additional mechanisms for reliability beyond that of TCP. The added reliability and fault tolerance of SCTP may function better for MapReduce-like distributed applications on large commodity clusters.

  SCTP has the following features that provide additional levels of reliability and fault tolerance. Selective acknowledgment (SACK) is built-in to the protocol with the ability to express larger gaps than TCP; as a result, SCTP outperforms TCP under loss. For cluster nodes with multiple interfaces, SCTP supports multihoming, which transparently provides failover in the event of network path failure. SCTP has the stronger CRC32c checksum which is necessary with high data rates and large scale systems. SCTP also allows multiple streams within a single connection, providing a solution to the head- of-line blocking problem present in TCP-based farming applications like Google’s MapReduce. Like TCP, SCTP provides a reliable data stream by default, but unlike TCP, messages can optionally age or reliability can be disabled altogether. The SCTP API provides both a one-to-one (like TCP) and a one-to-many (like UDP) socket style; use of a one-to-many style socket can reduce the number of file descriptors required by an application, making it more scalable.

  The additional scalability and fault tolerance come at a cost. The CRC32c checksum calculation currently is not off-loaded to any NIC available on the market, so it must be performed by the host CPU. In high bandwidth environments with no loss, SACK processing may become a burden on the host CPU.

• Breakout III: Scalable Test Selection Using Source Code Deltas - Ryan Gerard
  As the number of automated regression tests increase, the ability to run all of them in a reasonable amount of time becomes more and more difficult, and simply doesn’t scale. Since we are looking for regressions, it is useful to hone in on the parts of the code that have changed from the last run to help select a small subset of tests that are likely to find the regression. In this way we are only running the tests that need to be run as your system gets larger and the number of possible tests scales outward. We have devised a method to select a subset of tests from an existing test set for scalable regression testing based on source code changes, or deltas.

• Breakout IV: YouTube Scalability - Cuong Do
  This talk will discuss some of the scalability challenges that have arisen during YouTube’s short but extraordinary history. YouTube has grown incredibly rapidly despite having had only a handful of people responsible for scaling the site. Topics of discussion will include hardware scalability, software scalability, and database scalability.

• Breakout IV: Challenges in Building an Infinite Scalable Datastore - Swami Sivasubramanian, Werner Vogels
  In this talk, we will present the design of one of our internal datastores, HASS. HASS is designed to be “always” available, i.e., it will always accept read/write requests even if disks are failing, routes are flapping or if datacenters are being destroyed by tornados. HASS is designed for incremental scalability where adding or removing nodes can be done easily and the load gets evenly distributed among the nodes uniformly without requiring any operator intervention. In this talk, we will focus on a single and one of the most crucial ideas in HASS’s design: its ability to partition data. HASS uses consistent hashing to partition its data across its storage nodes. The basic consistent hashing algorithm is well understood in the academic literature and several research systems have been designed using it. In this talk, we will discuss our experiences with using the basic consistent hashing algorithm and the optimizations we performed to achieve more uniform load distribution and ease of operation.

Which ones should I attend?
I’m torn between a couple of the breakout options. Lustre vs. scalable resource management. YouTube vs. infinitely scalable datastore.

I know some of you folks are intimately involved with these topics, so I’d appreciate your suggestions, not only for which to attend, but what questions you’d like to see addressed. If some of you are also going to be there I’d also be pleased to meet f2f as well.

That last breakout session is a really tough choice. How can I be in two places at once?

While I’m up there I’m also hoping to tour Isilon’s lab and see their gear in action.

Comments and suggestions welcome. Last I heard the conference was full with a waiting list.

Make HFS to ZFS conversions fast and painless
Let’s say you wanted to convert millions of computers to a new file system. How would you do it? Back up, reformat the disk, and then reinstall everything?

Or run a handy file system converter that changes the file system without touching the files?

What? You can do that?
According to a patent filed by a couple of smart Apple engineers, yes you can. The hardworking Greg Keizer mentioned it in this Computerworld article. I thought it was worth a deeper look.

What it does - and how
The patent says:

. . . a first file-system can be converted to a second file-system while . . . files stored on the computer . . . remain virtually undisturbed in the same location. . . . the second filey-system can be generated primarily based on the location of the file(s) already stored on the computer . . . . It will also be appreciated that there is no need for rewriting or storing the file(s) while the file system is being converted.

The converter reads the existing file system to find out where all the files are on disk. Then it creates a new set of data structures, such as a catalog and file extents, for the new file system. After verifying the new data structures, the converter can then replace the first file system by, for example, modifying the disk’s partition map and and overwriting the volume headers of the first file system.

Here’s the flowchart for the conversion:

HFS+ to ZFS conversion is obvious
But is that all there is to it? The patent makes the point that the converter could be embedded in iTunes and used to convert the file system used on a media player. I can see the dialogue box now:

You have attached a Zune media player. Would you like iTunes to make it compatible with your iTunes music collection and the iTunes Music Store? All existing Zune music will be saved.

Pop in an AACS decryption program - or maybe this happens after the labels give up on encryption - and away you rock.

Set Reality Distortion Field to “global domination”
Or how about this. Apple supports Mac OS X on non-Mac hardware. Change the file system, use Rosetta technology and the open-source Wine Windows emulator to run existing Windows apps and it is a whole new world.

Instead of shipping a few million copies of OS X every year on low-margin hardware, Apple suddenly has the opportunity to sell tens of millions of high-margin copies. The hardware business continues, because Mac hardware is good, and suddenly there is a huge new market for Apple to conquer: the world’s installed base of 700 million PCs.

The StorageMojo take
Touchless file system conversion is a very cool capability. It makes it plausible that ZFS will become the default file system for Mac OS 10.5.n. I still don’t think ZFS will make be the default file system in Leopard from day one, if only because of the Q&A advantages of rolling it out first on OS X Server.

As for the less grounded speculation: Apple is a profitable and cash-rich company with a strong brand and high-quality hardware. They are about as well-positioned to offer Mac OS X on non-Apple machines as they will ever be. If they partnered with IBM for support they would have a ready entree into a business market tired of the foul stew of viruses and malware in which Windows machines swim.

It is a charming thought, no? Real competition on the world’s desktops.

Comments welcome, of course.

All we need now is teh Steve to say it . . .
Thanks to alert reader Petieg, I’ve learned that according to Mac Rumors Sun CEO Jonathan Schwartz said today that

In fact, this week you’ll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the file system in Mac OS 10.

Jonathan is wrong, of course, but it was sweet of him to say it
Folks tell me that if ZFS is in Leopard it is pretty well hidden. I’ll stick to my prediction that Apple, as with HFS+, will put ZFS on OS X Server first before bringing it out later for the great unwashed.

For one thing it will fix a persistent problem Xserve RAID admins have: pulling out the wrong drive, or scrambling drives, and losing lots of bits. V cool.

Now I’m going to pat myself on the back
As I noted in Bring Me the Head of WinFS:

Can Apple Trump Vista With ZFS?
Apple now has a clear path to trump Vista’s aging data management with a port of ZFS. While not offering a relational database and the promise of a single cross-application data store, ZFS is a modern file/storage management system whose end-to-end data integrity and protection makes it a strong foundation for future innovation. NTFS and Apple’s HFS+ are no match for it. Let’s hope Apple says more at their World Wide Developer Conference in August.

Well, cough, cough, it looks like August 2006 is finally arriving next week.

The NEW news
I finally put two and two together and figured this out: ZFS will be great for flash disks. Unlike today’s Mac OS and Windows, ZFS bunches writes - kind of like NetApp’s WAFL - which is just what flash drives need since their random write performance is even worse than I’d realized.

In fact, it just occurs to me that it could be on the iPhone. Why? Because Bonwick, Moore, et. al. managed to write all this stuff in very little code.

More info coming on flash
I’ve been delving deep into flash disks. Can you say “weird”? My take now is that flash drives are to disk drives what quantum mechanics is to Newtonian physics. I’m planning to have something out next week.

The StorageMojo take
The real importance of ZFS on Mac is that it raises the bar for the entire industry. Journaled file systems are better than not, but as the consumer-driven IT market booms customers need better data protection and recovery tools. And flash drives need a compatible file system. ZFS goes a long way towards meeting both requirements.

Update II: No mention of ZFS in Steve’s keynote or on the Apple website. I doubt we’ll hear much about it until Apple includes it in a release of OS X Server. Maybe in October, maybe not.

Update: Want to know more about ZFS? I’ve been hot on it for over a year. See:

• ZFS: Threat or Menace? Pt I
• ZFS: Threat or Menace? Pt II
• Bring Me The Head Of WinFS
• ZFS On Leopard: How Cool Is That?
• Is Apple’s Time Machine Built On Sun’s ZFS?
• Means, Motive & Opportunity: Apple Kills the Media Center PC
• ZFS On Mac: Now All-But-Official

Comments welcome, of course.

How to botch an announcement
I’ve been thinking about email lately (see Email value management and What is email?). So it was interesting to see that Palm managed to bungle the announcement of their “next big thing” the email companion called Foleo.

They pre-announced it at the D conference and got hammered for their trouble. Funny hammered, too.

Palm’s long decline
I was a big fan of Palm for years. I owned several Palms and still have a Palm Vx gathering dust somewhere. I still use the Palm Mac Desktop for addresses. But the Mac synch got hinky at some point and it got to be too much bother. They’ve had some success with the Treo line of smartphones, which I’ve occasionally lusted after, but cell service out here in the boonies is so poor that I’ve never seen the point.

And June, of course, is the month the iPhone ships. I expect some rough edges, and the total eclipse of Palm’s aging OS. Collateral damage. C’est la vie.

Yet I like the idea of the Foleo
Which may not be their idea at all. A truly mobile device, small, light, with a bright screen and a real keyboard. With world-wide internet connectivity through your phone.

So why the negative reception?
Partly the snark comes from the techies, who tend not to get the real world of people use this stuff without loving it. Partly it is the legacy of a company that has done little right for the last 5 years, so derision is a reflex. But partly it is because Jeff Hawkins is a product guy, not a marketing guy. Look at the entire, expensively produced intro. Product as fetish, invoking coolness. The problem is that it looks like a laptop. We’ve seen laptops. A laptop can still be cool, but Foleo is a crippled laptop.

The use-case is key
Instead of focusing on the near-laptop, Palm should have focused on how this fits into the target market’s busy day. These people aren’t playing Halo, designing web pages or cutting code. They are on the phone, surfing the web, reading email and preparing and presenting spreadsheets, presentations and yes, email.

Breaking the telephony from the information processing makes sense for these people. They need a phone and they need an application appliance.

The StorageMojo take
Successful, wealthy people tend to develop egos. Sometimes, in the case of people who are long-term successful, like Richard Branson or Rupert Murdoch, those egos are justified. Other people win the lottery and confuse luck with brilliance. Jeff Hawkins falls somewhere in-between. He has a vision, and Palm isn’t able to articulate it, and he isn’t able to help.

Bringing new technology to market is often more difficult than the new technology. People are fickle and at the same time hopeful that maybe, this time, there is something meaningful for them. Exciting curiosity is easy. Exciting (techno) lust is hard. Foleo is a visionary product, without visionary marketing. It is all too likely to remain a vision, and never gain the acceptance it could.

Comments welcome, of course. I’m back from Boston, where I saw a couple of old friends, and re-acquainted myself with one of my favorite cities. Sadly, live music in the Hub has declined markedly from 12 years ago. The DJs, with their huge stores of recorded music, stand astride the city’s nightlife.

It is good to be home.

So why did Cisco buy social-networking software provider Tribe?
John Chambers, Cisco CEO, is quoted in the D/ blog on the Wall Street Journal website (it might be open to non-subscribers, but I’m in an airplane right now and can’t check)

“A lot of this social networking kids do is absolutely going to drive business and drive business in a big way,” said Mr. Chambers. . . . He envisions a world where Cisco provides the gear and software that corporations can use to maintain online communities for their employees and collaborate on projects.

I think he’s right. Social networking for business is the cyberspace instantiation of the flat organizations developed over the last 20 years.

On FaceBook no one knows you’re a dog
Imagine a business whose executives know they need a new direction for a key division. Set up an anonymous social networking site and give every employee an account. Throw some questions about what the possible opportunities could be and see what happens.

In an anonymous world the usual suck-ups won’t be blowing smoke at the boss, since they won’t be able to tell who it is. The bright kid in the mail room will have the same opportunity as the dead-ended middle-manager to show what they can do. And the senior execs will the opportunity - and the responsibility - to show their creativity and receptivity as new ideas develop.

OK, how about a more likely scenario?
Few executives have the humility to put themselves out like that. So a more likely example is manufacturer with lots of independent dealers hosts a site for the dealers to pass on experiences, photos, tips and requests. Lubricated by incentives, some will emerge as leaders of the on-line community whose know-how enables new dealers to ramp faster and veteran dealers to more easily adapt and expand in response to changing markets.

So who is going to provide the storage?
Social networking often, but not always, builds on user-contributed content. That content tends to be easy to create and high-growth: photos, videos, audio recordings and short text messages. Most of the data will be cool - accessed sporadically - and whatever gets popular should get cached. Cheap rules.

The StorageMojo take
Computer-based social networks go back to the days of 300 baud modems and BBS’s. Big markets ride on basic human desires. Certainly the desire to communicate is one of those. It won’t be easy, but businesses that integrate that desire into their businesses will build large and loyal audiences.

Comments welcome. I’m on the road so moderation will be a little slower than usual.

Mirrorworlds for the masses
David Gelernter’s company, Mirror Worlds Technologies, tried to put an interface on Windows that reflected how people actually remember and link their experiences, rather than some CompSci metadata accommodation.

The company ceased operations three years ago, and at the time I recall wondering why. Last week, on a concall with CEO Ray Bingham of Arcmail, I started thinking about what email *really* is. Arcmail makes email archive servers and they’re announcing something but there’s an embargo on it for some time - days? weeks? - so that’s probably the last you’ll hear of them from me.

Not what I thought
I always thought it was about communication. And it is. Email speaks to me. It isn’t the email. It is the by-product: an organized record of communication.

Why else do I keep 15,000 emails?

Email is my journal, my archive, my most used and reliable search tool. It tracks relationships. Helps keep cryptically named documents associated with something I do understand. It is easy to organize temporally, easy to search.

I’ve used email for 25 years. Yet I never thought about how I used it. Maybe its because I’m now also IM’ing and video chatting - methods with the immediacy that email seemed to have over snail mail - that I’m starting to get it.

I maintain several on-line identities - StorageMojo, Storage Bits, Data Mobility Group and several more email addresses.

My email client is where they all come together.

It is the original online social network
And we treat it like it is email. It is identity. In a very real sense it is who we are.

Email: your personal metadata generator
Email adds value because it adds context - metadata - to raw files and communications. Context that is human readable and human memorable. That fits the relational database in our brains, not our computers. That provides metadata that people use, like names, conversations, topics and words that mean something.

Ray made the point that email servers, like Exchange, are just email servers. They aren’t designed to handle multi-gigabyte mailboxes, frequent individual searches or company-wide searches. Arcmail is.

Fear trumps greed
I pointed out some of the business advantages of big mailboxes as a business tool to Ray. He responded that he agreed, and that he’d tried some of those messages. Yet the chief buying motivator is fear of lawsuits, not the business process advantages.

Thought leadership, anyone?
I’ve come to believe that innovation happens regularly where ever people confront problems. What changes is our individual and cultural receptiveness to innovation. I think we may be getting ready to accept that email is one of the most valuable organizational tools we use and that there could be new ways of extracting business value from it.

The StorageMojo take
Email isn’t electronic “mail,” any more than cars are “horseless carriages.” Email not only goes faster than snail mail, it also provides its own infrastructure that makes it uniquely accessible and valuable.

Instead of thinking of email “clients” how about “communication clients” where email, chat, downloads, uploads and VOIP contacts are logged and are reviewable and searchable. I’d love to be able to go to one place to search my email, Adium and Skype chats, review my downloads - Safari’s 20 download history is inadequate for me - uploads and surfing history. All the information exists, but only in stovepipes. I want it all, on my local machine, always available, with an easy archive function.

Comments welcome. I can’t be the first person to think of this, so has anyone done an open-source comm client?