The official best paper winner at FAST ’10 isn’t one of the several I excerpted. I’m listening to the presentation as I write – trying live blogging – while following a fast talking presenter.

The winning paper is quFiles: The right file at the right time by Kaushik Veeraraghavan, Jason Flinn and Brian Noble of the University of Michigan and Edmund B. Nightingale of Microsoft Research.
From the abstract:

A quFile is a unifying abstraction that simplifies data management by encapsulating different physical representations of the same logical data. Similar to a quBit (quantum bit), the particular representation of the logical data displayed by a quFile is not determined until the moment it is needed. The representation returned by a quFile is specified by a data-specific policy that can take into account context such as the application requesting the data, the device on which data is accessed, screen size, and battery status.

One application is video files that may be played back on a variety of devices with differing resolutions, compute and graphics engines, codecs, editing capability and storage. There is one quFile that encapsulates several versions of the file – even different versions of the same file – and which is returned depends on the device requesting the file.

The key is that every device asks for the same file name, simplifying file management on the server and file distribution. quFiles are space efficient, adding little to file size, while their compute overhead is in the single-digit percents. And no application changes are required.

The StorageMojo take
FAST’s Best Paper isn’t necessarily StorageMojo’s Best Paper, yet this is a worthy candidate. Hiding the gory details of file types and network requirements from users is a Good Thing. I particularly like the support for file versioning, a feature I grew to love on the VMS operating system, but not widely appreciated today.

Congratulations to the team on their win.

Courteous comments welcome, of course. I’ll provide a link to the conference papers once the USENIX folks make them public.

Related posts:

1. StorageMojo @FAST ’10 next week The USENIX conference on File And Storage Technologies is in...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo @FAST ’10 next week The USENIX conference on File And Storage Technologies is in...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

MapReduce and its open source version, Hadoop, are parallel data analysis tools. A few lines of code can drive massive data reductions across thousands of nodes.

Cool.

Powerful though it is, Hadoop isn’t a database. Classic structured data analysis of the model/load/process type isn’t what it was designed for.

That’s where the paper HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads (pdf) comes in. Written by Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Avi Silberschatz and Alexander Rasin (the former 4 @Yale, and the latter @Brown) the paper proposes a method for building an open-source, commodity hardware-based massively scalable, shared-nothing, analytical parallel database.

What it is
HadoopDB coordinates SQL queries across multiple independent database nodes using Hadoop as the task coordinator and network communication layer. It uses the scheduling and job tracking of Hadoop while it intelligently pushes much of the query processing into the individual database nodes.

There are four components to HadoopDB.

• Database Connector. Each node has its own independent database. The connector is the interface between the database and Hadoop’s task trackers. A MapReduce jobs supplies the Connector with an SQL query and other parameters. The Connector executes a SQL query on the database and returns results as key value pairs. It can implemented to support a variety of databases.
• Catalog. The information needed to access the databases and metadata such as cluster data sets, replica locations and data partitions is kept in the catalog.
• Data loader. The data loader is responsible for two jobs. First executing a MapReduce job over Hadoop that reads the raw data files and partitions them into as many parts as the number of nodes in the cluster. Second, the partitions are loaded into the local file system of each node and chunked according the system-wide parameter.
• SQL to MapReduce to SQL planner. The planner provides a parallel database front end to enable SQL queries. The planner transforms the queries into map reduce jobs and optimizes the query plans for efficiency. This is where scratch that this is the secret sauce of HodoopDB.

HadoopDB complements the Hadoop infrastructure and does not replace it. Analysts have both available as needed.

Heterogeneity
A key issue for Internet-scale systems is the ability to run in a heterogenous environment where multi-year build-outs and rolling node replacement are the norm. That means that some nodes will be faster than others. HadoopDB breaks the work down into small tasks and moves them from slow to fast nodes automagically.

Results
The authors ran some benchmarks on Amazon’s EC to to test performance. The HadoopDB load times were about 10x that of Hadoop, but the higher performance of HadoopDB usually justified the longer set up time.

The authors found that HadoopDB was able to approach the performance of parallel database systems on much lower cost hardware and free software. Given the gift of the projects one can expect higher performance as improvements are made.

The killer app for private clouds?
MapReduce and Hadoop are already in wide use among Internet-scale datacenters. As companies begin to understand and correlate social media, web activity and ad response rates, the demand for large-scale parallel database processing will grow. But will they want to ship it out to Amazon?

Depending on the quantity and sensitivity of the data many organizations may prefer to keep the processing in-house. Private scale out Hadoop clusters may become the poor companies data warehouse of choice.

The StorageMojo take
HadoopDB is more science project than commercial tool today. Yet the project demonstrates the feasibility of using scale out compute/storage clusters for work that day typically requires proprietary high-end scale up system architectures.

If capital costs are reduced by two thirds with a commodity/FOSS architecture, companies could afford to hire the expertise required to make it work. The free software/paid support model will prove quite successful in this space.

Courteous comments welcome, of course.

Related posts:

1. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...
2. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...

While everyone else was watching the Apple iPad intro I was watching Oracle’s John Fowler talk about their systems and storage strategy. I like the iPad, but the O+S strategy could reshape the storage industry.

More details will emerge and many decisions still remain but the basic elements are clear:

• Focus on direct sales. In the mid-1990s, when I joined Sun, the tenacity and aggressiveness of their direct sales force was a welcome change. Direct sales forces are expensive, but losing touch with your customers is even costlier. The combo’s unique value propositions can’t be sold by channels today. In 5 years – maybe.
• A dedicated storage sales force. Generalist salespeople with millimeter deep storage product and application knowledge can’t compete with EMC and NetApp. Storage specialists aren’t easy to develop, so they’ll hire them – and they promise top commissions.
• Deep integration of ZFS into storage systems. A software company should like a software solution to many of the biggest storage problems? Putting real muscle behind ZFS will help thousands of enterprise customers to rethink their high-performance data protection strategies.
• Flash everywhere. Sun has done some creative things with flash already, such as Logzilla, and Oracle sees that much more can be done.

Not mentioned – not that it should have been – is the fate of ZFS on Mac OS X. That would be a boost for all concerned.

The StorageMojo take
Sun’s primary storage business has been a black smoking crater of disaster for over a decade. And it didn’t help StorageTek to have them answer to know-nothings.

Despite that Sun engineers outside the storage group developed innovative and game-changing technologies that the company couldn’t capitalize on. With Oracle’s investment now they can.

No database/systems company can be successful without a healthy and very competitive storage team — and the high gross margins don’t hurt. With a hard-nosed focus on application performance, marketing competence and continued innovation, the O+S storage group could be a fun place to work. They are hiring!

It will take Oracle 12 to 18 months to develop the kind of customer traction that will make other storage vendors set up and take notice. But Larry Ellison isn’t planning to lose and there is no reason he should.

Storage competition in the enterprise is about to get cranked up several notches. And that is a good thing for all customers.

Courteous comments welcome, of course.

Related posts:

1. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
2. Storage weather forecast: much coolness Spending the week in Silicon Valley catching up on storage...
3. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...

Related posts brought to you by Yet Another Related Posts Plugin.

1. HP’s unified storage/compute strategy HP’s Tech Days this week in Colorado Springs impressed on...
2. Storage weather forecast: much coolness Spending the week in Silicon Valley catching up on storage...
3. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...

In no particular order, cool stuff at Storage Visions 2010 and CES.

• Mobo-mounted SSD. Soligen has announced an SSD that mounts on motherboards. The drive mounts firmly, requires no special cooling and takes little board space.
• Tiny USB drive. Verbatim has announced a tiny USB thumb drive that is a fraction the size of most current thumb drives. Call it a thumbnail drive. Perfect for keychains.
• Super Talent is showing a 2 TB PCI-e SSD and claiming strong performance. At $6k gamers won’t buy it, but enterprises might.
• Raidon is showing a nice collection of 2.5″ drive enclosures, including 8 drive arrays. Not much larger than a 5.25″ drive. Can’t find them all on the web yet, though.
• A 32 GB Class 6 Micro SD is close to announcement. Micro.
• Supermicro showed a 48 drive JBOD/36 drive server chassis. The server is almost as dense of Sun’s Thumper – and drives are front and rear accessible.
• Eye-fi’s Wi-Fi enabled SD cards don’t handle AVCHD video files, but they’re working on it. With all the SD card using consumer, prosumer and even pro camcorders using SD, this will be a popular market for them.
• How about a double-ended flash drive: one end for personal; the other for work? Developed with the help of the social community at Quirky.com. They pay developers and influencers a percentage of the revenues. Cool!
• PoketyPoke is a con-call management service that reminds you of your concalls and optionally records them and provides transcripts for $9/hr. I like.

In other news
I moderated a too-short panel on Cloud storage at Storage Visions. Several technologies are out there that will change the current economics and application profiles of online storage. The field is young.

Got an update on USB 3.0 from Symwave, the fabless IC firm that makes USB 3.0 chips. Bottom line: unlike USB 2.0, whose marketing made promises the protocol could not keep, the new version can achieve over 400 MB/sec.

Here’s the 30 seconds over USB 3.0 video:

The StorageMojo take
No blockbuster, sector-defining new products. But many stepwise enhancements that move us forward.

USB 3.0 is going to push consumer storage as we can move gigabytes in seconds rather than minutes. But it looks like Apple is poised to miss this one – which could cost them a big chunk of their pro market.

Courteous comments welcome, of course. Fixed the pooched hyperlinks and a couple of other minor edits.

Related posts:

1. StorageMojo @ Storage Visions & CES this week 2010 is off to a fast start: I’ll be at...
2. Storage weather forecast: much coolness Spending the week in Silicon Valley catching up on storage...

Related posts brought to you by Yet Another Related Posts Plugin.

1. StorageMojo @ Storage Visions & CES this week 2010 is off to a fast start: I’ll be at...
2. Storage weather forecast: much coolness Spending the week in Silicon Valley catching up on storage...

Virtual machines (VMs) solve the problem of many tiny servers on a big server. VMs are a logical outgrowth of Moore’s Law: server CPUs got bigger, faster, than the apps required. And Windows Server didn’t handle multiple apps well.

But the growth of 100 megawatt Internet-scale data centers has architects rethinking efficiency-at-scale. As James Hamilton put it in his presentation
Internet-Scale Service Infrastructure Efficiency (pdf):

Single dimensional performance measurements are not interesting at scale unless balanced against cost

Therefore: work done per $; per joule; and per rack.

Microslice server
Because CPU performance has grown so much faster than storage – disk and DRAM – over the last 30 years, powerful multicore CPUs are spending much of their time idling. The microslice server idea: build servers from slower, cheaper and much more power-efficient CPUs.

Amazon has done just that. A microslice prototype jointly developed with SGI – formerly Rackable – using a lower power Athlon 4850e CPU handled over 9x the requests per second (RPS) of a rack of conventional servers.

And the server cost just $500, used 1/5th the power and provided about 70% of the performance (RPS) of the much costlier server. Higher density – something like 6 servers per rack unit – provided the rack-level performance.

Disk Workload from Hell
At October’s 22nd ACM Symposium on Operating Systems Principles (SOSP) – David G. Andersen, Jason Franklin, Amar Phanishayee, Lawrence Tan, Vijay Vasudevan – all from Carnegie Mellon University – and Michael Kaminsky (Intel Research Pittsburgh) presented FAWN: A Fast Array of Wimpy Nodes, a Best Paper award winner.

FAWN’s goal: maximizing queries per Joule in a high performance key-value storage system. Key-value stores are seeing increasing use in Internet-scale systems – the key is a unique identifier for the associated value.

The paper explains:

The workloads these systems support share several characteristics: they are I/O, not computation, intensive, requiring random access over large datasets; they are massively parallel, with thousands of concurrent, mostly-independent operations; their high load requires large clusters to support them; and the size of objects stored is typically small, e.g., 1 KB values for thumbnail images, 100s of bytes for wall posts, twitter messages, etc.

The paper describes both the hardware – which uses 500 MHz embedded processors, 256 MB DRAM and 4 GB CF flash – and the software – a log-structured per-node datastore that optimizes flash performance. The net/net: FAWN is over 6x more efficient – on queries per second – than conventional systems.

At 1/5th the cost. And 1/8th the power.

The StorageMojo take
This is more important than it looks. The Internet guys are optimizing for power, something most businesses ignore. But the low cost and performance of these nodes is attractive to everyone else.

Back in the day, DEC sold a lot of 3 node DSSI VAXclusters. Why? They were cheap(er) and if you lost a node you still had 2/3rds of your system.

In 2010 I expect to see low-end, cluster-based storage systems that offer multi-node resilience at low cost. Not just purchase price either, but service costs as well. A node went down? We’ll overnight you a new one.

The low-end is about to get a lot more interesting.

Courteous comments welcome, of course. The other SOSP best paper is fascinating too: RouteBricks: Exploiting Parallelism to Scale Software Routers. I hope I have time to post on it.

And BTW, Intel is also showing a microslice proto.

Related posts:

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. Hot data, smart cache Okay, we’ve figured out how to produce protected storage for...

Related posts brought to you by Yet Another Related Posts Plugin.

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. Hot data, smart cache Okay, we’ve figured out how to produce protected storage for...

Ding, dong.
PC file system progress took a giant step back this week with the news on MacOSforge that Apple’s ZFS project has been discontinued.

ZFS Project Shutdown 2009-10-23
The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

Apple announced in June ’08 that Snow Leopard server would support ZFS. But things came apart early this year.

What happened?
Jeff Bonwick, ZFS architect, posted Saturday on an earlier quoted comment:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a “private
> license” from Sun (with appropriate technical support and
> indemnification), and the two entities couldn’t come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Indemnification?
Sun is being sued by NetApp claiming that ZFS infringes on NetApp patents. If NetApp won, Apple would find itself in a tough position unless Sun shouldered the financial damage. That’s indemnification.

IMHO Sun has a good case that NetApp’s patents will be invalidated by prior art. But with all their other problems and the Oracle purchase it was a headache they, Oracle and Apple didn’t need.

Where does Apple go from here?
Apple has hired some smart file system engineers and wants to hire more to work on “state-of-the-art file system technologies for Mac OS X.”

I’m not convinced: it sounds like standard HR boilerplate and a snare for the unwary. But hey! it could happen.

But writing new file systems isn’t easy. It takes 5-7 years for a new file system to achieve the maturity needed to support large-scale deployment. Even replacing QuickTime is non-trivial.

So if Apple is starting from scratch we have a long wait for real innovation to appear. Like Mac OS XII.

What about Microsoft?
Meanwhile Redmond’s file system gurus are well aware of NTFS issues. They’re making stepwise enhancements.

But as the NTFS and HFS+ architectures age and the pace of storage innovation increases the gap between what is and what could be grows. It’s like putting a 1001 hp Bugatti engine in a Model T: the power is there but you can’t use it.

The StorageMojo take
I already hate software patents – but that’s another post. As long as law allows companies will try to enforce them.

Why didn’t Apple cut a deal with NetApp directly? Probably for the same reason Sun didn’t: money. Apple has a lot more of it than Sun, but Steve is a tightwad, especially when it comes to storage.

NetApp could have raised their visibility in the consumer market by cutting a deal with Apple, but NetApp’s management isn’t thinking strategically about the low-end of the market, as the rapidity of StoreVault’s entrance and exit demonstrated. True, they have bigger issues, but multi-tasking is supposed to be a corporate strength.

Consumers are generating masses of video and photos at an accelerating pace – and they’ll need reliable, available and dirt-easy storage. Lots of it.

Let EMC supply it!

Until the Next New Thing in file systems rolls out of Cupertino, Redmond or, maybe, Redwood City, consumers will stuck with too many BSODs, missing or corrupted files and app crashes. Let’s hope we don’t have to wait too many more years.

Comments welcome, of course. An earlier version of this was posted on Storage Bits. Can you spot the dozen or so differences?

And there is a Google code page for MacZFS for you diehards out there.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

High-end big iron storage arrays have long owned the transaction processing market. The big relational database systems need all the I/O and availability you can give them.

But what if we didn’t need big relational databases? What then?

RDBMS – RIP?
On his ACM blog, Michael Stonebraker, a database guru, says that relational databases may be nearing the end of the line. He says that their one-size-fits-all philosophy and 1980s code base is at the end of its useful life.

Why? Quoting Stonebraker:

If we examine the non-trivial sized DBMS markets, it turns out that the current relational DBMSs can be beaten by approximately a factor of 50 in most any market I can think of.

In the data warehouse market, a column store beats a row store by approximately a factor of 50 on typical business intelligence queries. . . .

In the online transaction processing (OLTP) market, a lightweight main memory DBMS needs a row store by a factor of 50. . . .

In the science DBMS market, users have never liked relational DBMSs and want a non-relational model and query facility. . . .

Text applications have never used relational DBMSs. This was pointed out to me most clearly by Eric Brewer nearly 15 years ago in the early days of Inktomi. He wanted to use a relational DBMS to store the results of web crawling, but found relational DBMSs to be two orders of magnitude slower than a homebrew system. . . .

Even in XML, where the current major vendors have spent a great deal of energy extending their and engines, it is claimed that specialized engines, such as Mark Logic or Tamino, run circles around the major vendors according to a private communication by Dave Kellogg.

In summary, one can leverage at least the following ideas to get superior performance:

A non-relational data model. . . . .

A different implementation of tables. . . .

A different implementation of transactions. . . .

Mr. Stonebraker’s comments have interesting storage implications. First, big iron storage arrays may not have the relational database management market to rely on much longer.

Second, what happens to storage system engineering when we no longer have one basic data management model to design for? And that is without considering the effect of a 50 times faster database on applications.

In the hardware world a 50 times speed up has 2 major effects: existing problems increase their resolution to absorb the additional compute cycles; and new applications – both low and high end – become economically feasible.

Is there 50 times more data we would collect from existing applications if we had a 50 times faster database? Or will we be running enterprise data management applications on hardware with the power of a netbook? Great power savings. Not so great for hardware vendors.

Mr. Stonebraker theorizes that the DBMS replacement will be a collection of vertical market specific engines. Each, no doubt, with its own storage I/O profile.

The StorageMojo take
Just as the ground has shifted under storage vendors in the last decade, it may be that DBMS vendors face the same problem opportunity in the coming decade.

If past experience is any guide, the storage industry will face multiple challenges supporting these new data management models, even as their high performance and lower (relative) costs drive new waves of application invention and adoption.

Only one thing is certain: much more data will be collected and, therefore, stored. The opportunities keep on coming, whether we are ready for them or not.

Courteous comments welcome, of course. For an interesting dissent, check out Daniel Lemire’s blog.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

With the release of Snow Leopard it is now official: no ZFS – anywhere – in Mac OS 10.6. Given that Apple went to the trouble of announcing it last year as part of Snow Leopard Server this is quite a reversal.

The question is why?

Many theories
I wrote this up on ZDnet Friday. At the time my theory was that the integration schedule or migration issues turned out to be less manageable than once thought. Or maybe NIH reared its parochial head.

ZDnet readers wrote in with ideas as well, the most popular that technical issues with ZFS itself forced the issue. I discounted this because, after all, ZFS is in production in large, I/O intense environments. If it is fundamentally broken we’d know by now.

I follow the ZFS discussion list and while there are issues, they aren’t show stopper bugs.

A new narrative
But then a couple of sources came in with a new angle: that Sun’s licensing demands killed the deal. Sun prefers the CDDL and may have asked for some extra protections, including Apple’s promise not seek damages should Sun lose the ZFS patent infringement suits initiated by NetApp, that caused Apple to reconsider the business risk of ZFS.

Sun could, of course, GPL ZFS, but it may also be that the ZFS engineering team – like other Sun engineers – rejected GPL. I’d love to get some comment from the ZFS team – very bright guys all – because this reminds me of the late ’80s at DEC when senior people begged DEC founder and CEO Ken Olsen to essentially open source some of DEC’s advanced software, like VMS, VMSclusters and DECnet.

Ken, a very smart engineer who shepherded DEC from a $70,000 startup to a $14 billion company, couldn’t see the business sense in giving away what the company had spent millions developing. So that leadership technology withered as DEC cratered.

The NetApp lawsuit may have come into play, making patent risk pertinent and potentially costly. Given that and the other CDDL-related risks, plus engineering opposition to GPL, Apple must have reluctantly stepped away. Apple would like bragging rights over Windows 7 that ZFS would give it, but in this narrative Sun’s pre-acquisition turmoil and tougher-than-expected licensing terms killed the deal.

Going forward
Now that Oracle is acquiring Sun things look brighter. Oracle is already bankrolling a GPL’d ZFS clone – btrfs – that will take years to reach the level of maturity that ZFS now enjoys. Once they own ZFS why wouldn’t they GPL it and call it good?

Update: Also, Oracle is in a stronger position to negotiate a settlement with NetApp over the ZFS/WAFL patent suits. After all, why would a storage company want the world’s largest database company as an enemy? End update.

The StorageMojo take
This is speculation of course and no doubt missing many specifics. But what is public – that Apple announced ZFS in June 2008, included a read-only CLI version in Leopard Server and is not shipping it in August 2009 – is evidence enough that things went awry. What other than a license issue would cause Apple to step away from even the read-only CLI version in Snow Leopard Server?

The ZFS team has produced a game-changing file system/volume manager. The chance to get it into the hands of 10s of millions of Mac users – and to influence Redmond’s file system strategy – seem to this outsider an opportunity of a lifetime.

If the ZFS engineering team opposed this – and I’d love to hear their take – I encourage them to reconsider. Marketers often ask the question “would you prefer 100% of nothing or 40% of something huge?”

Once the acquisition of Sun is complete, I hope Oracle quickly GPLs ZFS and cuts a deal with Apple. It will be good for them, for ZFS and for the entire industry.

Courteous comments welcome, of course. I worked for Sun for 3 years in the mid-90s and despite the many problems in the storage group I remain impressed by much of the company’s culture and accomplishments.

Update: I got the indemnification issue backwards in the original post and I thank those readers who deciphered my intent. For those who didn’t, I corrected it. While I was at it I made some other edits for clarity.End update.

Related posts:

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...
2. Oracle+Sun storage: wiser & brighter While everyone else was watching the Apple iPad intro I...
3. 2009′s big STORies 2009 has been an eventful year: the Great Recession has...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...
2. Oracle+Sun storage: wiser & brighter While everyone else was watching the Apple iPad intro I...
3. 2009′s big STORies 2009 has been an eventful year: the Great Recession has...

Bigtable to the rescue (sort of)
In Part 1, Sean Quinlan, a Google engineer, related how the original GFS single master architecture became a bottleneck. But since Google controls its entire software stack from OS to apps, it could compensate by tweaking the apps and the infrastructure, like Bigtable.

Bigtable is Google’s structured storage system. If you need a refresher – I did – check out Google’s Bigtable Distributed Storage System.

The short version from 3 years ago is:

Google’s Bigtable is essentially a massive, distributed 3-D spreadsheet. It doesn’t do SQL, there is limited support for atomic transactions, nor does it support the full relational database model. In short, in these and other areas, the Google team made design trade-offs to enable the scalability and fault-tolerance Google apps require. . . .Bigtable today supports almost 400 Google apps with data stores ranging up to several hundred terabytes.

Bigtable has a distributed lock server, Chubby, that coordinates the several thousand nodes in large Bigtable clusters. Presumably that is why Bigtable has been able to scale to handle many of the problems the single-master GFS has created.

But – and there’s always a but – Quinlan says that Bigtable isn’t an optimal solution to the many files/small files problem:

. . . [U]sing BigTable . . . as a way of fighting the file-count problem where you might have otherwise used a file system to handle that — then you would not end up employing all of BigTable’s functionality . . . . BigTable isn’t really ideal . . . in that it requires resources for its own operations that are nontrivial. Also, it has a garbage-collection policy that’s not super-aggressive, so that might not be the most efficient way to use your space. . . . people who have been using BigTable purely to deal with the file-count problem probably haven’t been terribly happy, but . . . it is one way for people to handle that problem.

GFS was designed to maximize bandwidth to disk as the crawlers sluiced data back to Google. Low latency was a non-goal. But as Google offered more user-facing apps, latency became important. Sean notes:

. . . if you’re writing a file, it will typically be written in triplicate—meaning you’ll actually be writing to three chunkservers. Should one of those chunkservers die . . . the GFS master will notice the problem and schedule what we call a pullchunk, which means it will basically replicate one of those chunks. That will get you back up to three copies, and then the system will pass control back to the client, which will continue writing.

When we do a pullchunk we limit it to something on the order of 5-10 MB a second. So, for 64 MB, you’re talking about 10 seconds. . . . If you are working on Gmail, however, and you’re trying to write a mutation that represents some user action, then getting stuck for a minute is really going to mess you up.

Consistency, consistency, cOnsiStencY
Since GFS and BigTable were designed to run on massive pools of commodity hardware failures and faults were a given.

For example, disk drives would tell Linux they supported some IDE versions when they really didn’t, leading to silent data corruption when drives and kernels disagreed about the drive’s state. GFS includes rigorous end-to-end check-summing to protect data from network and storage corruption, but other decisions compromised data consistency.

GFS simply assumes that there will be times that stale data is returned to applications. Data appended to an open file won’t be seen until the file is reopened.

Given that the Google owned GFS, Bigtable and the apps, it seemed acceptable to ask the apps to handle some problems. But some of the inherent problems are hard ones.

If a client crashes in the middle of a write, data could be left in an indeterminate state. The RecordAppend operation supported multiple writers to a single file, so if a primary writer failed you could end up with multiple inconsistent copies of the data in a single file – with different versions of the file in different chunks.

These things may not happen all that often, but it’s Murphy’s Law: if they can happen, they will. With several million servers and dozens of data centers, it is a continuing headache.

Snapshot?
Sean makes an interesting comment about the GFS snapshot feature, which he calls “the most general-purpose snapshot capability you can imagine.”

I also think it’s interesting that the snapshot feature hasn’t been used more since it’s actually a very powerful feature. . . from a file-system point of view, it really offers a pretty nice piece of functionality.

I’d like to hear from Google app developers why they didn’t use the snapshot feature. I suspect it is an interesting set of reasons.

The StorageMojo take
Google engineers have been hard at work for the last 2 years building a distributed master system that will work better with Bigtable to fix many of the current problems.

Still, it is amazing that in 1 year 4 or 5 people could put together a file system critical to Google’s success for almost 10 years. It looks creaky now, but it has also scaled far beyond what it’s developers expected.

“Scalability” is one of the most abused words in the IT marketing lexicon. It is often used where “expandability” is more appropriate.

That GFS has scaled 1,000x or more is a benchmark for Internet data center infrastructure. With billions of people still not on the web and the growth of sensor networks, machine translation and other scale intensive apps, 1,000x is the new normal.

Get used to it. Plan for it.

Courteous comments welcome, of course.

Related posts:

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. quFiles: The right file at the right time The official best paper winner at FAST ’10 isn’t one...
3. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...

Related posts brought to you by Yet Another Related Posts Plugin.

1. MaxiScale’s Web-scale file system A new web scale – they claim linear scaling to...
2. quFiles: The right file at the right time The official best paper winner at FAST ’10 isn’t one...
3. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...

Jeff Darcy has a good post on key data stores, like Amazon’s Dynamo, and how they differ from filesystems and databases. He relates his transition from a filesystem purist to a more flexible perspective.

The thing that really changed my mind about this was an observation in the Dynamo paper: strong consistency reduces availability. I’ve always thought of data availability in terms of data not being lost or stranded on the other side of a failed network connection. The Dynamo insight is that many applications have to do a lot of work within a small acceptable-response-time window, and to make sure that they fit into that window they have to impose deadlines on all sub-operations including data access. If consistency issues make data unavailable within that deadline then they’ve made it unavailable period, with practically the same effect as if the data were unavailable in any other sense.

In short, while there is a class of applications where traditional consistency is important, there is an emerging class where strong consistency isn’t affordable or necessary. Good stuff.

Another point
Many of the features that make up these non-FS/non-DB stores seem to have a lot in common with object storage. In a highly mobile world the whole idea of placing a file in cyberspace by a path name is anachronistic at best: it could be, physically, almost anywhere and is most likely in several places at once.

The StorageMojo take
While the name “object” is problematic for market acceptance, the concept of managing objects in a flat address space – like the web itself – is a better fit for a mobile networked world. There is a major opportunity to move file management infrastructure forward to reflect the world we now live in rather than a 35 year old server environment.

Courteous comments welcome, of course. Thanks to Wes Felter’s Hack the Planet blog for the link to Jeff’s post.

Related posts:

1. A petascale parallel database MapReduce and its open source version, Hadoop, are parallel data...

Related posts brought to you by Yet Another Related Posts Plugin.

1. A petascale parallel database MapReduce and its open source version, Hadoop, are parallel data...

Am @ Interop today – a nice, relaxing 250 mile drive from home – so this isn’t a standard StorageMojo post. Think of it as an expanded tweet.

Part of what Oracle gets with Sun is ZFS. And part of what Chris Mason of Oracle is working on is Btrfs – B-Tree or “butter” FS – seen as a Linux answer to ZFS. With a GPL license.

With many of the same features – such as parent-stored checksums and snapshots – Btrfs provides important new functionality to Linux. But if ZFS is an Oracle property, how hard could it be to change the licensing and open it up to the Linux community?

The StorageMojo take
I’m asking the question, not answering it. License T&C’s are important, but if the bottom line is that CDDL is incompatible with GPL, will Oracle be able to fix that? Will they want to?

Or does Linux really need AZFS – Almost ZFS?

Courteous comments welcome, of course.

Related posts:

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...

The Wall Street Journal (subscription probably required) reports that e-books appear to be taking off:

Barnes & Noble Inc. has launched a free electronic-reader application for Research In Motion Ltd.’s BlackBerry as general book sales flag and the e-book market heats up. . . .

Consumers spent about $100 million on e-books in 2008, according to one estimate. German media concern Bertelsmann AG noted in its annual results Tuesday that its e-book sales “increased tremendously” in 2008 and that its Random House publishing arm will have 15,000 e-book titles in the U.S. by year-end. . . .

Sony Corp., which also sells a dedicated e-book reader, recently struck a partnership with Google Inc. that provides users of the Sony Reader with free access to more than 500,000 public-domain titles that Google has digitized.

A practical lesson in consumer technology diffusion
For 20 years E-books have been a so-obvious-it-hurts application:

• Books are heavy and expensive to ship and store – and returns kill profits
• Many books are words only: easy to convert to ASCII
• Shop at home convenience
• Instant download gratification

But despite all the Obvious Goodness ebooks haven’t left the building – until now.

A confluence of factors is behind the take-off:

• A growing supply of ebooks
• Wider adoption of readable large screen handheld devices
• Simplified purchase and wireless download – thanks to improved infrastructure
• Popular awareness of ebook advantages

Ebooks could have taken off on notebook computers – but they didn’t. The unseen need for a paperback-sized reader, along with much simpler access, has catalyzed a market.

The StorageMojo take
Emerging and evolving technologies are StorageMojo’s stock-in-trade. Comments back sometimes betray the notion that because it hasn’t happened yet, it will never happen. Which is always true – until it isn’t.

Apple’s Newton was a great idea hampered by technology limitations. The less-ambitious Palm Pilot got the form factor and consumer pricing right, and the handheld device market took off. But not the Palm ebook market.

Fast forward 10 years and iPhone has more power than the Palm Pilot ever dreamed of and it’s an acceptable – though not ideal – ebook reader. As well as a gaming platform, browser, PIM and more.

In prospect technology change always takes longer than we think it should. In retrospect it is amazing how quickly the technology evolves.

Courteous comments welcome, of course.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Ms. Mojo persuaded me – she can be very persuasive – that tweeting could be cool. I’m skeptical, but game.

Not about me
For those of you coming in even later, Twitter is the trendy new revenue-free social web phenom. Users communicate through messages of no more than 140 characters dubbed “tweets.”

Some call it “micro-blogging.” Expect storage-focused tweets. Click here to follow StorageMojo .

What do 140 character StorageMojo posts look like?
Here are the 3 most recent – and only – StorageMojo tweets, presented in their entirety:

Fusion-io’s ioDrive.”. . .scary fast in a database server, especially when it comes to tiny random write IO’s.” http://tr.im/hDVv

Latest LTO tape is spec’d at 200 entire read/writes. What if disks had that spec? http://tr.im/hDUU

Cisco to Buy Video Camera Maker Pure Digital – are they nuts? Kodak’s mtk cap is $1B – with real assets & a great brand.

The weird URLs are compressed – like TinyURLs – and clickable. The latest 5 tweets are also on the StorageMojo home page, as is a “follow” link. Look below the tower ad in the far right sidebar.

The StorageMojo take
I hope this makes Ms. Mojo happy. And who knows, the 21st century telegram may grow on us.

Courteous comments welcome, of course.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Can single-user OS X be far behind?
Here’s the official Apple announcement:

For business-critical server deployments, Snow Leopard Server adds read and write support for the high-performance, 128-bit ZFS file system, which includes advanced features such as storage pooling, data redundancy, automatic error correction, dynamic volume expansion, and snapshots.

The StorageMojo take
Cool! And only 2 years later than I’d predicted. I’m an optimist.

As I noted almost 2 years ago:

StorageMojo.com has devoted time to this issue because today’s computer business is largely driven by consumer computing, not enterprise computing. Putting a really modern integrated file and storage management system on a consumer OS would raise the bar for everyone else.

I stand by that.

Comments welcome, of course.
For more on ZFS see:
Want to know more about ZFS? I’ve been hot on it for over a year. See:

• ZFS: Threat or Menace? Pt I
• ZFS: Threat or Menace? Pt II
• Bring Me The Head Of WinFS
• ZFS On Leopard: How Cool Is That?
• Means, Motive & Opportunity: Apple Kills the Media Center PC

Related posts:

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...
2. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...

Related posts brought to you by Yet Another Related Posts Plugin.

1. Mac ZFS is dead Ding, dong. PC file system progress took a giant step...
2. Tiny server clusters Virtual machines (VMs) solve the problem of many tiny servers...