I sat down with Nimble Storage co-founder and VP of engineering Varun Mehta to discuss their architecture – and shoot some video. Varun has been part of several Valley success stories – NetApp, Sun, Data Domain – and has a first hand perspective on disruptive technologies.

Varun and co-founder Umesh Maheshwari – a brilliant architect and a very nice guy – designed the Nimble product that he discusses. Take 4 minutes to learn more about Innovations in Storage Architecture at Nimble Storage:

Or you can see it in HD on YouTube.

The StorageMojo take
The Nimble guys have great technology, but they’ve also put together a compelling value proposition: collapse 3 time-consuming and complex workflows – primary storage, backup and archiving – into 1 appliance. Include all the needed software, price it well, target under-served mid-sized companies and you have a recipe for another Valley success.

The tech trends they’re riding will only get better. But the business trends are in their favor as well. SMB’s today have many TB of data and little staff to manage it – or capital to invest. With Congress ensuring that America operates well below capacity for years to come, the times favor thrifty solutions like Nimble’s.

Courteous comments welcome, of course.
Nimble bought my time for this video, but I made all editorial decisions.

Megastore handles over 3 billion writes and 20 billion reads daily on almost 8 PB of primary data across many global data centers.

In a paper by Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson, Jean-Michel Léon, Yawei Li, Alexander Lloyd, Vadim Yushprakh titled Megastore: Providing Scalable, Highly Available Storage for Interactive Services Google engineers describe how it works. From the abstract:

Megastore is a storage system developed to meet the requirements of today’s interactive online services. Megastore blends the scalability of a NoSQL data store with the convenience of a traditional RDBMS in a novel way, and provides both strong consistency guarantees and high-availability. We provide fully serializable ACID semantics within fine-grained partitions of data. This partitioning allows us to synchronously replicate each write across a wide area network with reasonable latency and support seamless failover between data centers.

The mission
Support Internet apps such as Google’s AppEngine.

• Scale to millions of users
• Responsive despite Internet latencies to impatient users
• Easy for developers
• Fault resilience from drive failures to data center loss and everything in between
• Low-latency synchronous replication to distant sites

The how
Scale by partitioning the data store and replicating each partition separately, providing full ACID semantics within partitions but limited consistency guarantees across them. Offer some traditional database features if they scale with tolerable latency.

The key assumptions are that data for many apps can be partitioned, for example by user, and that a selected set of DB features can make developers productive.

Availability and scale
To achieve availability and global scale the designers implemented two key architectural features:

• For availability, an asynchronous log replicator optimized for long-distance
• For scale, data partitioned into small databases each with its own replicated log

Rather than implement a master/slave or optimistic replication strategy, the team decided to use Paxos, a consensus algorithm that does not require a master, with a novel extension. A single Paxos log would soon become a bottleneck with millions of users so each partition gets its own replicated Paxos log.

Data is partitioned into entity groups which are synchronously replicated over a wide area while the data itself is stored in NoSQL storage. ACID transaction records within the entities are replicated using Paxos.

For transactions across entities, the synchronous replication requirement is relaxed and an asynchronous message queue is used. Thus it’s key that entity group boundaries reflect application usage and user expectations.

Entities
An e-mail account is a natural entity. But defining other entities is more complex.

Geographic data lacks natural granularity. For example, the globe is divided into non-overlapping entities. Changes across these geographic entities use (expensive) two-phase commits.

The design problem: entities large enough to make two-phase commits uncommon but small enough to keep transaction rates low.

Each entity has a root table and may have child tables. Each child table has a single root table. Example: a user’s root table may have each of the user’s photo collections as a child. Most applications find natural entity group boundaries.

API
The insight driving the API is that the big win is scalable performance rather than a rich query language. Thus a focus on controlling physical locality and hierarchical layouts.

For example, joins are implemented in application code. Queries specify scans or lookups against particular tables and indexes. Therefore, the application needs to understand the data schema to perform well.

Replication
Megastore uses Paxos to manage synchronous replication. But in order to make Paxos practical despite high latencies the team developed some optimizations:

• Fast reads. Current reads are usually from local replicas since most writes succeed on all replicas.
• Fast writes. Since most apps repeatedly write from the same region, the initial writer is granted priority for further replica writes. Using local replicas and reducing write contention for distant replicas minimizes latency.
• Replica types. In addition to full replicas Megastore has 2 other replica types:
  witness replicas. Witnesses vote in Paxos rounds and store the write-ahead log but do not store entity data or indexes to keep storage costs low. They are also tiebreakers when isn’t a quorum.
  Read-only replicas are the inverse: nonvoting replicas that contain full snapshots of the data. Their data may be slightly stale but they help disseminate the data over a wide area without slowing writes.

Architecture
What does Megastore look like in practice? Here’s an example.

A Megastore client library is installed on the app server. It implements Paxos and other algorithms such as read replica selection. The app server has a local replica written to a local BigTable instance.

A coordinator server tracks a set of entity groups and observes all Paxos writes. The coordinator is simpler than BigTable and serves local reads.

Concurrent with writing local data to BigTable and the coordinator the Megastore library is also writing to a second full replica: a replication server and a second coordinator. The stateless replication servers handle the writes to the remote big table while the lower latency coordinator handles any reads from the remote replica.

Failures may leave writes abandoned or in an uncertain state. The replication servers scan for incomplete writes and offer no op values via Paxos to complete the.

Availability
As coordinator servers do most local reads their availability is critical to maintaining Megastore’s performance. The coordinators use an out-of-band protocol to track other coordinators and use Google’s Chubby distributed lock service to obtain remote locks. If the coordinator loses a majority of its locks it will consider all entities in its purview to be out of date until the locks are regained and the coordinator is current.

There are a variety of network and race conditions that can affect coordinator availability. The team believes the simplicity of the coordinator architecture and their light network traffic makes the availability risks acceptable.

Performance
Because Megastore is geographically distributed, application servers in different locations may initiate writes to the same end entity group simultaneously. Only one of them will succeed and the other writers will have to retry.

Limiting writes to a few per second per entity group makes contention insignificant, e-mail for example.

For multiuser applications with higher write requirements developers can shard entity groups more finely or batch user operations into fewer transactions. Fine-grained advisory locks and sequencing transactions are other techniques to handle higher write loads.

The real world
Megastores been deployed for several years and more than 100 production applications using today. The paper provides these figures on availability and average latencies.

The high availability of the system architecture creates a nice-to-have problem: small transient errors on top of persistent uncorrected problems can cause much larger problems.

Fault tolerance makes finding underlying faults more difficult. The price of fault tolerance is eternal vigilance.

As the architecture diagram suggests Megastore doesn’t manage BigTable. Developers must optimize the storage for their app.

The StorageMojo take
As Brewer’s CAP theorem showed, a distributed system can’t provide consistency, availability and partition tolerance to all nodes at the same time. But this paper shows that by making smart choices we can get darn close as far as human users are concerned.

If Microsoft Office – or an open-source analog – could plug into a productized version of Megastore this could become popular for private cloud implementations: LAN performance in the office and global availability on the road. What’s not to like?

But whether that happens or not, the paper demonstrates again the value of Internet scale infrastructure thinking. Enterprise vendors would never have developed Megastore, but now that we’ve seen it work we can begin applying its principles to smaller scale problems.

Courteous comments welcome, of course. If this overview intrigues I urge you to read the entire paper as there are some interesting pieces I’ve left out.

Talked to a company last week whose cloud app handles several billion transactions per month on a cluster. Sounds like SSDs could help them but how?

In a paper from the latest 5th Biennial Conference on Innovative Data Systems Research (CIDR ’11) researchers Philip A. Bernstein and Colin W. Reid of Microsoft and Sudipto Das of UC Santa Barbara have a suggestion: Hyder – A Transactional Record Manager for Shared Flash (pdf).

As underlying hardware changes – faster networks, large memories, multi-core CPUs and SSDs – database software architectures may change too. Hyder architecture supports

. . . reads and writes on indexed records within classical multi-step transactions. It is designed to run on a cluster of servers that have shared access to a large pool of network-addressable raw flash chips. . . . Hyder uses a data-sharing architecture that scales out without partitioning the database or application.

No partition scale-out
Today, most popular database clusters partition the database across multiple servers. Done well this works, but at some cost. The database design is non-trivial – cross-partition transactions, cache coherence, load balancing, scaling and multi-server debugging – are knotty issues which translate into higher design and operation costs.

Hyder eliminates partitioning, distributed programming, layers of cache, remote procedure calls and load balancing. All servers can read and write the entire database – so any server can execute any transaction. Load-balancing is simple: direct new transactions to lightly-loaded servers.

Each update transaction runs on one machine and writes to a shared log – so there’s no 2-phase commit. And no 2-phase commit locking, which can force performance off a cliff when workloads spike.

The 3 main components of Hyder are the log, the index and the roll-forward algorithm.

Log
The log runs on multiple flash devices – chips, DIMMs or ??? – and writes multi-page log records across multiple devices with parity to enable log recovery after device failures.

Hyder uses a multi-versioned database – old record versions aren’t updated-in-place, only the most recent version of a record is used – which has a couple of useful properties:

• Server caches are inherently coherent since only the most recent versions of records are used.
• Data can be read while writes are in progress.
• Queries that can be decomposed can be run across multiple servers concurrently for a faster response time.

[This may seem like voodoo to ACIDheads. A good technical intro to multi-versioning concurrency control (MVCC) is Multi-core software: to gain speed, eliminate resource contention.]

Servers run a cache update process that keeps them current with updated records. Server caches don’t have to be identical and the cache invalidate messages that most clusters use for cache coherency aren’t needed.

All log writes are idempotent appends, so if a write fails the server can simply reissue the write. The authors describe several error modes and how Hyder handles them.

Index
The index stores the database as a search tree with each node a [key, payload] pair. The tree can store, for example, a relational database. The index tree is also represented in the log.

Tree nodes are not updated in place. When node n is updated, a new copy – n’is created. Then, of course, the parent node must be updated and so on up the tree.

A binary tree minimizes the number of node updates, but can be processor intensive. The optimal tree structure for Hyder is not yet resolved.

Garbage collection is an issue. Each node pointer includes the ID of the oldest reachable data element. An element older than any that is pointed to by a node is garbage.

Roll-forward algorithm
This is the key process of Hyder.

When a record update begins, one server executes the transaction. The server is given a copy of the latest database root, a static snapshot of the entire database.

The updates are stored in a local cache and after execution the after-images are gathered into an intention record, which is broadcast to all servers and appended to the log. The update’s readset is included in the intention record, to insure all intentions are properly ordered, none are lost, and the offset is made known to all servers.

Each server can assemble a local copy of the tail of the log, which is used to determine if there are conflicting updates. The meld procedure manages conflicting updates.

Appending the intention to the database log doesn’t commit the transaction. The intention references the static snapshot of the latest database root. The meld procedure determines if any committed transactions since the snapshot conflict with the intention.

If they don’t, all is good. If they do, the transaction is aborted.

All servers roll forward using meld and don’t message each other about committed and failed transactions. Therefore there is no lock manager and no 2-phase commit.

Contention
Losing the lock manager and 2-phase commit should help performance unless other points of contention throttle the system. Hyder’s points of contention include appending intentions to the log, melding the log at each server, and aborting transactions.

Intention appends are serial. The lower the write latency the more appends can be written. A 10us write latency means a 100k TPS.

Network latency adds to write latency. Faster switches improve append performance.

The abort rate depends on the number of concurrent transactions that conflict. Fast transactions reduce the probability of aborts by reducing the number of concurrent transactions.

The worst case is a record subject to multiple updates from different servers. Detecting high-conflict transactions and serializing them by forcing them onto 1 server would reduce the hot data performance hit.

Performance
The authors model Hyder’s performance with a focus on the high-contention corner cases. In general, the tests show linear scaling as servers are added.

The problems come when the underlying hardware limits are exceeded. Increasing execution times mean more aborts and performance falls off a cliff. From the paper:

The StorageMojo take
We’ve been building disk workarounds for for decades. We now tend to assume those workarounds are fundamental architectural requirements rather than hacks.

The Hyder paper asks us to imagine a world where non-volatile mass storage is fast and cheap – and how we could re-architect basic systems to be faster and cheaper too.

The authors conclusion is a fair assessment:

Many variations of the Hyder architecture and algorithms would be worth exploring. There may also be opportunities to use Hyder’s logging and meld algorithms with some modification in other contexts, such as file systems and middleware. We suggested a number of directions for future work throughout the paper. No doubt there are many other directions as well.

Courteous comments welcome, of course. I hope to get to some of the other CIDR papers before FAST ’11 snows me under. Update: Phil Bernstein was kind enough to scan the post and I’ve updated 1 minor error. He also mentioned that it won the Best Paper award at the conference. Those CIDR folks have great taste in papers, don’t they?

The official best paper winner at FAST ’10 isn’t one of the several I excerpted. I’m listening to the presentation as I write – trying live blogging – while following a fast talking presenter.

The winning paper is quFiles: The right file at the right time by Kaushik Veeraraghavan, Jason Flinn and Brian Noble of the University of Michigan and Edmund B. Nightingale of Microsoft Research.
From the abstract:

A quFile is a unifying abstraction that simplifies data management by encapsulating different physical representations of the same logical data. Similar to a quBit (quantum bit), the particular representation of the logical data displayed by a quFile is not determined until the moment it is needed. The representation returned by a quFile is specified by a data-specific policy that can take into account context such as the application requesting the data, the device on which data is accessed, screen size, and battery status.

One application is video files that may be played back on a variety of devices with differing resolutions, compute and graphics engines, codecs, editing capability and storage. There is one quFile that encapsulates several versions of the file – even different versions of the same file – and which is returned depends on the device requesting the file.

The key is that every device asks for the same file name, simplifying file management on the server and file distribution. quFiles are space efficient, adding little to file size, while their compute overhead is in the single-digit percents. And no application changes are required.

The StorageMojo take
FAST’s Best Paper isn’t necessarily StorageMojo’s Best Paper, yet this is a worthy candidate. Hiding the gory details of file types and network requirements from users is a Good Thing. I particularly like the support for file versioning, a feature I grew to love on the VMS operating system, but not widely appreciated today.

Congratulations to the team on their win.

Courteous comments welcome, of course. I’ll provide a link to the conference papers once the USENIX folks make them public.

MapReduce and its open source version, Hadoop, are parallel data analysis tools. A few lines of code can drive massive data reductions across thousands of nodes.

Cool.

Powerful though it is, Hadoop isn’t a database. Classic structured data analysis of the model/load/process type isn’t what it was designed for.

That’s where the paper HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads (pdf) comes in. Written by Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Avi Silberschatz and Alexander Rasin (the former 4 @Yale, and the latter @Brown) the paper proposes a method for building an open-source, commodity hardware-based massively scalable, shared-nothing, analytical parallel database.

What it is
HadoopDB coordinates SQL queries across multiple independent database nodes using Hadoop as the task coordinator and network communication layer. It uses the scheduling and job tracking of Hadoop while it intelligently pushes much of the query processing into the individual database nodes.

There are four components to HadoopDB.

• Database Connector. Each node has its own independent database. The connector is the interface between the database and Hadoop’s task trackers. A MapReduce jobs supplies the Connector with an SQL query and other parameters. The Connector executes a SQL query on the database and returns results as key value pairs. It can implemented to support a variety of databases.
• Catalog. The information needed to access the databases and metadata such as cluster data sets, replica locations and data partitions is kept in the catalog.
• Data loader. The data loader is responsible for two jobs. First executing a MapReduce job over Hadoop that reads the raw data files and partitions them into as many parts as the number of nodes in the cluster. Second, the partitions are loaded into the local file system of each node and chunked according the system-wide parameter.
• SQL to MapReduce to SQL planner. The planner provides a parallel database front end to enable SQL queries. The planner transforms the queries into map reduce jobs and optimizes the query plans for efficiency. This is where scratch that this is the secret sauce of HodoopDB.

HadoopDB complements the Hadoop infrastructure and does not replace it. Analysts have both available as needed.

Heterogeneity
A key issue for Internet-scale systems is the ability to run in a heterogenous environment where multi-year build-outs and rolling node replacement are the norm. That means that some nodes will be faster than others. HadoopDB breaks the work down into small tasks and moves them from slow to fast nodes automagically.

Results
The authors ran some benchmarks on Amazon’s EC to to test performance. The HadoopDB load times were about 10x that of Hadoop, but the higher performance of HadoopDB usually justified the longer set up time.

The authors found that HadoopDB was able to approach the performance of parallel database systems on much lower cost hardware and free software. Given the gift of the projects one can expect higher performance as improvements are made.

The killer app for private clouds?
MapReduce and Hadoop are already in wide use among Internet-scale datacenters. As companies begin to understand and correlate social media, web activity and ad response rates, the demand for large-scale parallel database processing will grow. But will they want to ship it out to Amazon?

Depending on the quantity and sensitivity of the data many organizations may prefer to keep the processing in-house. Private scale out Hadoop clusters may become the poor companies data warehouse of choice.

The StorageMojo take
HadoopDB is more science project than commercial tool today. Yet the project demonstrates the feasibility of using scale out compute/storage clusters for work that day typically requires proprietary high-end scale up system architectures.

If capital costs are reduced by two thirds with a commodity/FOSS architecture, companies could afford to hire the expertise required to make it work. The free software/paid support model will prove quite successful in this space.

Courteous comments welcome, of course.

While everyone else was watching the Apple iPad intro I was watching Oracle’s John Fowler talk about their systems and storage strategy. I like the iPad, but the O+S strategy could reshape the storage industry.

More details will emerge and many decisions still remain but the basic elements are clear:

• Focus on direct sales. In the mid-1990s, when I joined Sun, the tenacity and aggressiveness of their direct sales force was a welcome change. Direct sales forces are expensive, but losing touch with your customers is even costlier. The combo’s unique value propositions can’t be sold by channels today. In 5 years – maybe.
• A dedicated storage sales force. Generalist salespeople with millimeter deep storage product and application knowledge can’t compete with EMC and NetApp. Storage specialists aren’t easy to develop, so they’ll hire them – and they promise top commissions.
• Deep integration of ZFS into storage systems. A software company should like a software solution to many of the biggest storage problems? Putting real muscle behind ZFS will help thousands of enterprise customers to rethink their high-performance data protection strategies.
• Flash everywhere. Sun has done some creative things with flash already, such as Logzilla, and Oracle sees that much more can be done.

Not mentioned – not that it should have been – is the fate of ZFS on Mac OS X. That would be a boost for all concerned.

The StorageMojo take
Sun’s primary storage business has been a black smoking crater of disaster for over a decade. And it didn’t help StorageTek to have them answer to know-nothings.

Despite that Sun engineers outside the storage group developed innovative and game-changing technologies that the company couldn’t capitalize on. With Oracle’s investment now they can.

No database/systems company can be successful without a healthy and very competitive storage team — and the high gross margins don’t hurt. With a hard-nosed focus on application performance, marketing competence and continued innovation, the O+S storage group could be a fun place to work. They are hiring!

It will take Oracle 12 to 18 months to develop the kind of customer traction that will make other storage vendors set up and take notice. But Larry Ellison isn’t planning to lose and there is no reason he should.

Storage competition in the enterprise is about to get cranked up several notches. And that is a good thing for all customers.

Courteous comments welcome, of course.

In no particular order, cool stuff at Storage Visions 2010 and CES.

• Mobo-mounted SSD. Soligen has announced an SSD that mounts on motherboards. The drive mounts firmly, requires no special cooling and takes little board space.
• Tiny USB drive. Verbatim has announced a tiny USB thumb drive that is a fraction the size of most current thumb drives. Call it a thumbnail drive. Perfect for keychains.
• Super Talent is showing a 2 TB PCI-e SSD and claiming strong performance. At $6k gamers won’t buy it, but enterprises might.
• Raidon is showing a nice collection of 2.5″ drive enclosures, including 8 drive arrays. Not much larger than a 5.25″ drive. Can’t find them all on the web yet, though.
• A 32 GB Class 6 Micro SD is close to announcement. Micro.
• Supermicro showed a 48 drive JBOD/36 drive server chassis. The server is almost as dense of Sun’s Thumper – and drives are front and rear accessible.
• Eye-fi’s Wi-Fi enabled SD cards don’t handle AVCHD video files, but they’re working on it. With all the SD card using consumer, prosumer and even pro camcorders using SD, this will be a popular market for them.
• How about a double-ended flash drive: one end for personal; the other for work? Developed with the help of the social community at Quirky.com. They pay developers and influencers a percentage of the revenues. Cool!
• PoketyPoke is a con-call management service that reminds you of your concalls and optionally records them and provides transcripts for $9/hr. I like.

In other news
I moderated a too-short panel on Cloud storage at Storage Visions. Several technologies are out there that will change the current economics and application profiles of online storage. The field is young.

Got an update on USB 3.0 from Symwave, the fabless IC firm that makes USB 3.0 chips. Bottom line: unlike USB 2.0, whose marketing made promises the protocol could not keep, the new version can achieve over 400 MB/sec.

Here’s the 30 seconds over USB 3.0 video:

The StorageMojo take
No blockbuster, sector-defining new products. But many stepwise enhancements that move us forward.

USB 3.0 is going to push consumer storage as we can move gigabytes in seconds rather than minutes. But it looks like Apple is poised to miss this one – which could cost them a big chunk of their pro market.

Courteous comments welcome, of course. Fixed the pooched hyperlinks and a couple of other minor edits.

Virtual machines (VMs) solve the problem of many tiny servers on a big server. VMs are a logical outgrowth of Moore’s Law: server CPUs got bigger, faster, than the apps required. And Windows Server didn’t handle multiple apps well.

But the growth of 100 megawatt Internet-scale data centers has architects rethinking efficiency-at-scale. As James Hamilton put it in his presentation
Internet-Scale Service Infrastructure Efficiency (pdf):

Single dimensional performance measurements are not interesting at scale unless balanced against cost

Therefore: work done per $; per joule; and per rack.

Microslice server
Because CPU performance has grown so much faster than storage – disk and DRAM – over the last 30 years, powerful multicore CPUs are spending much of their time idling. The microslice server idea: build servers from slower, cheaper and much more power-efficient CPUs.

Amazon has done just that. A microslice prototype jointly developed with SGI – formerly Rackable – using a lower power Athlon 4850e CPU handled over 9x the requests per second (RPS) of a rack of conventional servers.

And the server cost just $500, used 1/5th the power and provided about 70% of the performance (RPS) of the much costlier server. Higher density – something like 6 servers per rack unit – provided the rack-level performance.

Disk Workload from Hell
At October’s 22nd ACM Symposium on Operating Systems Principles (SOSP) – David G. Andersen, Jason Franklin, Amar Phanishayee, Lawrence Tan, Vijay Vasudevan – all from Carnegie Mellon University – and Michael Kaminsky (Intel Research Pittsburgh) presented FAWN: A Fast Array of Wimpy Nodes, a Best Paper award winner.

FAWN’s goal: maximizing queries per Joule in a high performance key-value storage system. Key-value stores are seeing increasing use in Internet-scale systems – the key is a unique identifier for the associated value.

The paper explains:

The workloads these systems support share several characteristics: they are I/O, not computation, intensive, requiring random access over large datasets; they are massively parallel, with thousands of concurrent, mostly-independent operations; their high load requires large clusters to support them; and the size of objects stored is typically small, e.g., 1 KB values for thumbnail images, 100s of bytes for wall posts, twitter messages, etc.

The paper describes both the hardware – which uses 500 MHz embedded processors, 256 MB DRAM and 4 GB CF flash – and the software – a log-structured per-node datastore that optimizes flash performance. The net/net: FAWN is over 6x more efficient – on queries per second – than conventional systems.

At 1/5th the cost. And 1/8th the power.

The StorageMojo take
This is more important than it looks. The Internet guys are optimizing for power, something most businesses ignore. But the low cost and performance of these nodes is attractive to everyone else.

Back in the day, DEC sold a lot of 3 node DSSI VAXclusters. Why? They were cheap(er) and if you lost a node you still had 2/3rds of your system.

In 2010 I expect to see low-end, cluster-based storage systems that offer multi-node resilience at low cost. Not just purchase price either, but service costs as well. A node went down? We’ll overnight you a new one.

The low-end is about to get a lot more interesting.

Courteous comments welcome, of course. The other SOSP best paper is fascinating too: RouteBricks: Exploiting Parallelism to Scale Software Routers. I hope I have time to post on it.

And BTW, Intel is also showing a microslice proto.

Ding, dong.
PC file system progress took a giant step back this week with the news on MacOSforge that Apple’s ZFS project has been discontinued.

ZFS Project Shutdown 2009-10-23
The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

Apple announced in June ’08 that Snow Leopard server would support ZFS. But things came apart early this year.

What happened?
Jeff Bonwick, ZFS architect, posted Saturday on an earlier quoted comment:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a “private
> license” from Sun (with appropriate technical support and
> indemnification), and the two entities couldn’t come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Indemnification?
Sun is being sued by NetApp claiming that ZFS infringes on NetApp patents. If NetApp won, Apple would find itself in a tough position unless Sun shouldered the financial damage. That’s indemnification.

IMHO Sun has a good case that NetApp’s patents will be invalidated by prior art. But with all their other problems and the Oracle purchase it was a headache they, Oracle and Apple didn’t need.

Where does Apple go from here?
Apple has hired some smart file system engineers and wants to hire more to work on “state-of-the-art file system technologies for Mac OS X.”

I’m not convinced: it sounds like standard HR boilerplate and a snare for the unwary. But hey! it could happen.

But writing new file systems isn’t easy. It takes 5-7 years for a new file system to achieve the maturity needed to support large-scale deployment. Even replacing QuickTime is non-trivial.

So if Apple is starting from scratch we have a long wait for real innovation to appear. Like Mac OS XII.

What about Microsoft?
Meanwhile Redmond’s file system gurus are well aware of NTFS issues. They’re making stepwise enhancements.

But as the NTFS and HFS+ architectures age and the pace of storage innovation increases the gap between what is and what could be grows. It’s like putting a 1001 hp Bugatti engine in a Model T: the power is there but you can’t use it.

The StorageMojo take
I already hate software patents – but that’s another post. As long as law allows companies will try to enforce them.

Why didn’t Apple cut a deal with NetApp directly? Probably for the same reason Sun didn’t: money. Apple has a lot more of it than Sun, but Steve is a tightwad, especially when it comes to storage.

NetApp could have raised their visibility in the consumer market by cutting a deal with Apple, but NetApp’s management isn’t thinking strategically about the low-end of the market, as the rapidity of StoreVault’s entrance and exit demonstrated. True, they have bigger issues, but multi-tasking is supposed to be a corporate strength.

Consumers are generating masses of video and photos at an accelerating pace – and they’ll need reliable, available and dirt-easy storage. Lots of it.

Let EMC supply it!

Until the Next New Thing in file systems rolls out of Cupertino, Redmond or, maybe, Redwood City, consumers will stuck with too many BSODs, missing or corrupted files and app crashes. Let’s hope we don’t have to wait too many more years.

Comments welcome, of course. An earlier version of this was posted on Storage Bits. Can you spot the dozen or so differences?

And there is a Google code page for MacZFS for you diehards out there.

High-end big iron storage arrays have long owned the transaction processing market. The big relational database systems need all the I/O and availability you can give them.

But what if we didn’t need big relational databases? What then?

RDBMS – RIP?
On his ACM blog, Michael Stonebraker, a database guru, says that relational databases may be nearing the end of the line. He says that their one-size-fits-all philosophy and 1980s code base is at the end of its useful life.

Why? Quoting Stonebraker:

If we examine the non-trivial sized DBMS markets, it turns out that the current relational DBMSs can be beaten by approximately a factor of 50 in most any market I can think of.

In the data warehouse market, a column store beats a row store by approximately a factor of 50 on typical business intelligence queries. . . .

In the online transaction processing (OLTP) market, a lightweight main memory DBMS needs a row store by a factor of 50. . . .

In the science DBMS market, users have never liked relational DBMSs and want a non-relational model and query facility. . . .

Text applications have never used relational DBMSs. This was pointed out to me most clearly by Eric Brewer nearly 15 years ago in the early days of Inktomi. He wanted to use a relational DBMS to store the results of web crawling, but found relational DBMSs to be two orders of magnitude slower than a homebrew system. . . .

Even in XML, where the current major vendors have spent a great deal of energy extending their and engines, it is claimed that specialized engines, such as Mark Logic or Tamino, run circles around the major vendors according to a private communication by Dave Kellogg.

In summary, one can leverage at least the following ideas to get superior performance:

A non-relational data model. . . . .

A different implementation of tables. . . .

A different implementation of transactions. . . .

Mr. Stonebraker’s comments have interesting storage implications. First, big iron storage arrays may not have the relational database management market to rely on much longer.

Second, what happens to storage system engineering when we no longer have one basic data management model to design for? And that is without considering the effect of a 50 times faster database on applications.

In the hardware world a 50 times speed up has 2 major effects: existing problems increase their resolution to absorb the additional compute cycles; and new applications – both low and high end – become economically feasible.

Is there 50 times more data we would collect from existing applications if we had a 50 times faster database? Or will we be running enterprise data management applications on hardware with the power of a netbook? Great power savings. Not so great for hardware vendors.

Mr. Stonebraker theorizes that the DBMS replacement will be a collection of vertical market specific engines. Each, no doubt, with its own storage I/O profile.

The StorageMojo take
Just as the ground has shifted under storage vendors in the last decade, it may be that DBMS vendors face the same problem opportunity in the coming decade.

If past experience is any guide, the storage industry will face multiple challenges supporting these new data management models, even as their high performance and lower (relative) costs drive new waves of application invention and adoption.

Only one thing is certain: much more data will be collected and, therefore, stored. The opportunities keep on coming, whether we are ready for them or not.

Courteous comments welcome, of course. For an interesting dissent, check out Daniel Lemire’s blog.

With the release of Snow Leopard it is now official: no ZFS – anywhere – in Mac OS 10.6. Given that Apple went to the trouble of announcing it last year as part of Snow Leopard Server this is quite a reversal.

The question is why?

Many theories
I wrote this up on ZDnet Friday. At the time my theory was that the integration schedule or migration issues turned out to be less manageable than once thought. Or maybe NIH reared its parochial head.

ZDnet readers wrote in with ideas as well, the most popular that technical issues with ZFS itself forced the issue. I discounted this because, after all, ZFS is in production in large, I/O intense environments. If it is fundamentally broken we’d know by now.

I follow the ZFS discussion list and while there are issues, they aren’t show stopper bugs.

A new narrative
But then a couple of sources came in with a new angle: that Sun’s licensing demands killed the deal. Sun prefers the CDDL and may have asked for some extra protections, including Apple’s promise not seek damages should Sun lose the ZFS patent infringement suits initiated by NetApp, that caused Apple to reconsider the business risk of ZFS.

Sun could, of course, GPL ZFS, but it may also be that the ZFS engineering team – like other Sun engineers – rejected GPL. I’d love to get some comment from the ZFS team – very bright guys all – because this reminds me of the late ’80s at DEC when senior people begged DEC founder and CEO Ken Olsen to essentially open source some of DEC’s advanced software, like VMS, VMSclusters and DECnet.

Ken, a very smart engineer who shepherded DEC from a $70,000 startup to a $14 billion company, couldn’t see the business sense in giving away what the company had spent millions developing. So that leadership technology withered as DEC cratered.

The NetApp lawsuit may have come into play, making patent risk pertinent and potentially costly. Given that and the other CDDL-related risks, plus engineering opposition to GPL, Apple must have reluctantly stepped away. Apple would like bragging rights over Windows 7 that ZFS would give it, but in this narrative Sun’s pre-acquisition turmoil and tougher-than-expected licensing terms killed the deal.

Going forward
Now that Oracle is acquiring Sun things look brighter. Oracle is already bankrolling a GPL’d ZFS clone – btrfs – that will take years to reach the level of maturity that ZFS now enjoys. Once they own ZFS why wouldn’t they GPL it and call it good?

Update: Also, Oracle is in a stronger position to negotiate a settlement with NetApp over the ZFS/WAFL patent suits. After all, why would a storage company want the world’s largest database company as an enemy? End update.

The StorageMojo take
This is speculation of course and no doubt missing many specifics. But what is public – that Apple announced ZFS in June 2008, included a read-only CLI version in Leopard Server and is not shipping it in August 2009 – is evidence enough that things went awry. What other than a license issue would cause Apple to step away from even the read-only CLI version in Snow Leopard Server?

The ZFS team has produced a game-changing file system/volume manager. The chance to get it into the hands of 10s of millions of Mac users – and to influence Redmond’s file system strategy – seem to this outsider an opportunity of a lifetime.

If the ZFS engineering team opposed this – and I’d love to hear their take – I encourage them to reconsider. Marketers often ask the question “would you prefer 100% of nothing or 40% of something huge?”

Once the acquisition of Sun is complete, I hope Oracle quickly GPLs ZFS and cuts a deal with Apple. It will be good for them, for ZFS and for the entire industry.

Courteous comments welcome, of course. I worked for Sun for 3 years in the mid-90s and despite the many problems in the storage group I remain impressed by much of the company’s culture and accomplishments.

Update: I got the indemnification issue backwards in the original post and I thank those readers who deciphered my intent. For those who didn’t, I corrected it. While I was at it I made some other edits for clarity.End update.

Bigtable to the rescue (sort of)
In Part 1, Sean Quinlan, a Google engineer, related how the original GFS single master architecture became a bottleneck. But since Google controls its entire software stack from OS to apps, it could compensate by tweaking the apps and the infrastructure, like Bigtable.

Bigtable is Google’s structured storage system. If you need a refresher – I did – check out Google’s Bigtable Distributed Storage System.

The short version from 3 years ago is:

Google’s Bigtable is essentially a massive, distributed 3-D spreadsheet. It doesn’t do SQL, there is limited support for atomic transactions, nor does it support the full relational database model. In short, in these and other areas, the Google team made design trade-offs to enable the scalability and fault-tolerance Google apps require. . . .Bigtable today supports almost 400 Google apps with data stores ranging up to several hundred terabytes.

Bigtable has a distributed lock server, Chubby, that coordinates the several thousand nodes in large Bigtable clusters. Presumably that is why Bigtable has been able to scale to handle many of the problems the single-master GFS has created.

But – and there’s always a but – Quinlan says that Bigtable isn’t an optimal solution to the many files/small files problem:

. . . [U]sing BigTable . . . as a way of fighting the file-count problem where you might have otherwise used a file system to handle that — then you would not end up employing all of BigTable’s functionality . . . . BigTable isn’t really ideal . . . in that it requires resources for its own operations that are nontrivial. Also, it has a garbage-collection policy that’s not super-aggressive, so that might not be the most efficient way to use your space. . . . people who have been using BigTable purely to deal with the file-count problem probably haven’t been terribly happy, but . . . it is one way for people to handle that problem.

GFS was designed to maximize bandwidth to disk as the crawlers sluiced data back to Google. Low latency was a non-goal. But as Google offered more user-facing apps, latency became important. Sean notes:

. . . if you’re writing a file, it will typically be written in triplicate—meaning you’ll actually be writing to three chunkservers. Should one of those chunkservers die . . . the GFS master will notice the problem and schedule what we call a pullchunk, which means it will basically replicate one of those chunks. That will get you back up to three copies, and then the system will pass control back to the client, which will continue writing.

When we do a pullchunk we limit it to something on the order of 5-10 MB a second. So, for 64 MB, you’re talking about 10 seconds. . . . If you are working on Gmail, however, and you’re trying to write a mutation that represents some user action, then getting stuck for a minute is really going to mess you up.

Consistency, consistency, cOnsiStencY
Since GFS and BigTable were designed to run on massive pools of commodity hardware failures and faults were a given.

For example, disk drives would tell Linux they supported some IDE versions when they really didn’t, leading to silent data corruption when drives and kernels disagreed about the drive’s state. GFS includes rigorous end-to-end check-summing to protect data from network and storage corruption, but other decisions compromised data consistency.

GFS simply assumes that there will be times that stale data is returned to applications. Data appended to an open file won’t be seen until the file is reopened.

Given that the Google owned GFS, Bigtable and the apps, it seemed acceptable to ask the apps to handle some problems. But some of the inherent problems are hard ones.

If a client crashes in the middle of a write, data could be left in an indeterminate state. The RecordAppend operation supported multiple writers to a single file, so if a primary writer failed you could end up with multiple inconsistent copies of the data in a single file – with different versions of the file in different chunks.

These things may not happen all that often, but it’s Murphy’s Law: if they can happen, they will. With several million servers and dozens of data centers, it is a continuing headache.

Snapshot?
Sean makes an interesting comment about the GFS snapshot feature, which he calls “the most general-purpose snapshot capability you can imagine.”

I also think it’s interesting that the snapshot feature hasn’t been used more since it’s actually a very powerful feature. . . from a file-system point of view, it really offers a pretty nice piece of functionality.

I’d like to hear from Google app developers why they didn’t use the snapshot feature. I suspect it is an interesting set of reasons.

The StorageMojo take
Google engineers have been hard at work for the last 2 years building a distributed master system that will work better with Bigtable to fix many of the current problems.

Still, it is amazing that in 1 year 4 or 5 people could put together a file system critical to Google’s success for almost 10 years. It looks creaky now, but it has also scaled far beyond what it’s developers expected.

“Scalability” is one of the most abused words in the IT marketing lexicon. It is often used where “expandability” is more appropriate.

That GFS has scaled 1,000x or more is a benchmark for Internet data center infrastructure. With billions of people still not on the web and the growth of sensor networks, machine translation and other scale intensive apps, 1,000x is the new normal.

Get used to it. Plan for it.

Courteous comments welcome, of course.

Jeff Darcy has a good post on key data stores, like Amazon’s Dynamo, and how they differ from filesystems and databases. He relates his transition from a filesystem purist to a more flexible perspective.

The thing that really changed my mind about this was an observation in the Dynamo paper: strong consistency reduces availability. I’ve always thought of data availability in terms of data not being lost or stranded on the other side of a failed network connection. The Dynamo insight is that many applications have to do a lot of work within a small acceptable-response-time window, and to make sure that they fit into that window they have to impose deadlines on all sub-operations including data access. If consistency issues make data unavailable within that deadline then they’ve made it unavailable period, with practically the same effect as if the data were unavailable in any other sense.

In short, while there is a class of applications where traditional consistency is important, there is an emerging class where strong consistency isn’t affordable or necessary. Good stuff.

Another point
Many of the features that make up these non-FS/non-DB stores seem to have a lot in common with object storage. In a highly mobile world the whole idea of placing a file in cyberspace by a path name is anachronistic at best: it could be, physically, almost anywhere and is most likely in several places at once.

The StorageMojo take
While the name “object” is problematic for market acceptance, the concept of managing objects in a flat address space – like the web itself – is a better fit for a mobile networked world. There is a major opportunity to move file management infrastructure forward to reflect the world we now live in rather than a 35 year old server environment.

Courteous comments welcome, of course. Thanks to Wes Felter’s Hack the Planet blog for the link to Jeff’s post.

Am @ Interop today – a nice, relaxing 250 mile drive from home – so this isn’t a standard StorageMojo post. Think of it as an expanded tweet.

Part of what Oracle gets with Sun is ZFS. And part of what Chris Mason of Oracle is working on is Btrfs – B-Tree or “butter” FS – seen as a Linux answer to ZFS. With a GPL license.

With many of the same features – such as parent-stored checksums and snapshots – Btrfs provides important new functionality to Linux. But if ZFS is an Oracle property, how hard could it be to change the licensing and open it up to the Linux community?

The StorageMojo take
I’m asking the question, not answering it. License T&C’s are important, but if the bottom line is that CDDL is incompatible with GPL, will Oracle be able to fix that? Will they want to?

Or does Linux really need AZFS – Almost ZFS?

Courteous comments welcome, of course.