Amazon Web Services architect James Hamilton has been posting on network issues for over a year and researching them much longer. As Ethernet becomes the de facto SAN technology, his views become more relevant to the larger storage market.

Critique
Part of Mr. Hamilton’s concern is the structure of the networking industry: the high margins; the dominance of a single player, Cisco; the closed technology; and the heavy vertical integration. All antithetical to the dynamics that have driven server costs down so successfully in the last 20 years.

These are issues the storage industry knows too well. But Mr. Hamilton is more concerned about the waste the current high-cost industry structure causes.

Waste?

Workload placement
The cost of network bandwidth leads to network over-subscription. Networks are configured as tree topologies: the further you move from end nodes the worse the over subscription.

As described in the 2009 Microsoft Research paper VL2: A Scalable and Flexible Data Center Network:

. . . the capacity between different branches of the tree is typically over- subscribed by factors of 1:5 or more, with paths through the highest levels of the tree oversubscribed by factors of 1:80 to 1:240. This limits communication between servers to the point that it fragments the server pool — congestion and computation hot-spots are prevalent even when spare capacity is available elsewhere.

This throttles data center performance by limiting server-to-server bandwidth, fragmenting resources and reducing network utilization. The latter reflects the redundant paths needed in case of switch failure: ≈50% or more of costly data center bandwidth goes unused.

As might be expected, big Internet data centers like Amazon’s have complex and unpredictable workloads. They need lots of bandwidth between all servers all the time.

A solution
The VL2 paper describes an experimental solution to these problems that includes location-specific and application-specific addressing, multi-path traffic load balancing and a novel directory design that efficiently handles lookups and updates to network mappings.

In an 75-node test cluster the design moved 2.75TB of data in 395 seconds – 94% of maximum network bandwidth – at a fraction of the cost of current enterprise networks. The paper calculates that a cloud-service scale network with no over-subscription could be built with commodity switches at 1/14th the cost of a traditional data center Ethernet.

Whoa!

The StorageMojo take
VC and engineering dollars follow high-growth markets. What Google, Amazon and Microsoft want, they get. With the rapid growth of public cloud services the network over-subscription problem will get solved.

Merchant silicon from Broadcom, Intel and Marvell is making a tried-and-true Moore’s Law attack on hardware cost. The protocol stack is tougher, but several open-source industry initiatives are under way with support from major companies. Progress will be slower than hoped, but within 3 years we’ll have a viable stack to build on.

Where does this leave the networking industry? That depends on where you sit.

Cisco will be the biggest loser, because they’ve been the biggest winner with the current model. They may need to pull an IBM and move big into services if they want to stick around. Ironically, Cisco’s UCS product line – which bakes in the tree-structured network – has further motivated broader industry action.

The rest of the industry can go after this emerging market with a lower-GM business model. Not all of them will, but it will be a critical success factor.

The big winner will be storage. Scale-out storage relies on spraying data across multiple racks for maximum availability, utilization and performance. Cheaper, faster, better scale-out networks will only drive storage demand.

For most of us this is an academic problem today. Lightly used systems – such as for backup and archiving – don’t see Amazon’s problems. But in 5 years this will be common even outside the public cloud providers.

Just as IT users have benefited from Google’s push on energy efficiency and much more, they will also benefit from much lower cost and more scalable networks.

Courteous comments welcome, of course. I can’t help but continue to marvel at how dumb Cisco’s UCS has turned out to be. It’s a gift that keeps on giving.

One more datapoint and we’ll have a trend: NoSQL databases managing metadata. It’s obvious in retrospect: use a scalable big data tool to handle scale-out metadata. Maybe not a requirement today, but surely will be with even bigger data tomorrow.

Metadata is a fraction of the user data set, but it gets hammered much more. As more metadata is found useful the hammering will get more insistent.

Nutanix
Nutanix, whose CTO and co-founder, Mohit Aron, was a developer of the Google File System, uses MapReduce. Nutanix achieves it scale due to its distributed metadata, masterless architecture – powered by MapReduce jobs that run in the background.

Druva
Druva, a backup company for mobile devices, also uses a NoSQL database to manage storage metadata. They say they’ve found that NoSQL scales over an order of magnitude better than relational in similar applications.

Somebody else
A company that shall remain nameless is porting Hadoop to their backend. The customer won’t be able to access Hadoop for their work – it is strictly for the system’s internal use.

It is a proof of concept so it isn’t a 3rd data point, but they see the potential advantages. Call it data point 2½.

The StorageMojo take
Small advances are the building blocks of disruption. RAID made it possible to build available storage using cheap disks. Consumer adoption of PCs made disks even cheaper. Moore’s Law made RAID controllers cheaper and faster, or faster and more capable.

A virtuous circle of disruption.

The basic architecture of scale-out storage systems – purpose-built software on clustered commodity hardware – has been stable. But this is the beginning of scale-out storage 2.0: taking scale-out technology developed for users and incorporating it into the storage infrastructure itself.

These ideas are bubbling up among the latest startups and among the establishment players. At some point the old RAID architectures will be well and truly broken, able to compete in smaller and smaller niches until the revenue can’t justify more investment.

Of course vendors have been making RAID controllers out of servers for years now, and those servers can run any software they want. But at some point the explicit and implicit assumptions in the old architecture crash into current realities – either in cost, development time, feature completeness or management overhead – and then we move on.

Courteous comments welcome, of course. I learned about Nutanix at the last Tech Field Day “The Independent IT Influencer Event” which paid for my travel expenses to Silicon Valley.

CES is a lot of fun, but my favorite toy trade show is the National Association of Broadcasters (NAB) convention. I arrive tomorrow and return Wednesday and hope – on the way back – to walk across the new bridge that is 900 feet above the Colorado River at Hoover Dam.

I do video work today and did FM broadcasting decades ago. Today we’re all narrowcasting, but digital has made the tech now so much better – and cheaper! – than what broadcasters had 10 years ago that the possibilities are endless.

Pre-show expectations
Rumor has it that Apple will announce the newest version of Final Cut Studio – my preferred editing platform – on Tuesday. I’m also looking for any and all Thunderbolt peripherals, although the pre-show PR hasn’t mentioned it. I hope Promise and LaCie have something to show, and perhaps Sony as well.

Object storage should be more visible this year as well. And where will USB 3.0 show up – if it shows up anywhere – in pro gear?

The StorageMojo take
There’s something about pro gear – even though I can’t afford 98% of it and wouldn’t use it to best advantage if I could – that fascinates. Built into all the features and specs is deep knowledge of the technology and its limitations.

There are people who spend $5k for a microphone to record a single instrument. Their ears can discern the differences between equally high-end mics and they know how to mix them to get the sound they want.

Then we listen to all that painstaking work through crummy little earbuds. Oh well!

If you have something to share please contact me through the comments. I’m looking for cool stuff.

Courteous comments welcome, of course.

The younger StorageMojo analysts have heard that New York is a den of sin and depravity and can’t wait to try some. With new Wrangler jeans and polished silver bolos they are ready to par-tay with some big city gals. Yee-ha!

Should I tell them that it will be 95% male? Nah.

The event is GigaOm’s Structure Big Data 2011 conference on Wednesday the 23rd. Haven’t been to it before, but I was overdue to see what Mr. Malik has cooked up.

The StorageMojo take
With the gradual slowing of improvement in hardware and big plans for massive data collection, the world of storage is looking at accelerating change. Drive and CPU vendors won’t be doing all the heavy lifting.

That means that architecture will become even more critical to successful products. Lots of room for creativity and innovation because there is a ready audience that needs something better than what they have today.

Looking forward to meeting new people and learning about new technologies and markets. I’ll have some time Thursday morning too, if anyone is eager to bend my ear. Leave a comment with your preferred means of contact.

Courteous comments welcome, of course.

Update: The survey is now closed. Thanks to everyone who responded!

I’ll have more later on the results. End update.

StorageMojo would like your help.

TechnoQWAN LLC, StorageMojo’s publisher, is a research and analysis firm. An IT supplier has retained us to help plan their internal storage strategy .

You’ll be anonymous, unless you choose not to be. This is a research project, not a marketing Trojan horse.

How you can help
Please donate 5 to 8 minutes to complete a survey. You’ll help keep StorageMojo the independent source of storage coolness it has always strived to be.

We’d like to get a couple of hundred respondents RSN. Can you do it now?

Here’s the link to the Internal Storage Survey.

And please pass the link on to likely friends and colleagues. The more the merrier!

The StorageMojo take
After we’ve looked at the data I plan to write about what I’ve found interesting in the results. You’ll get to learn something about the rest of the StorageMojo community.

And I’ll get to learn some more about you.

If, perchance, you’re passionate about the topic, you’ll be able to volunteer for a more in-depth discussion. Trust me: the sponsor can make a real difference in the servers you buy in a couple of years.

Courteous comments welcome, of course. Completed surveys even more so! BTW, QWAN = Quality Without A Name.

After a quick scan of the paper titles I wasn’t impressed. But after seeing presentations and posters I am.

Here’s some I found interesting. I’ll be posting longer pieces on some of these.

• A Study of Practical Deduplication Full paper *Best Paper Winner*
• Tradeoffs in Scalable Data Routing for Deduplication Clusters Full paper
• Exploiting Half-Wits: Smarter Storage for Low-Power Devices Full paper
• Reliably Erasing Data from Flash-Based Solid State Drives Full paper
• Scale and Concurrency of GIGA+: File System Directories with Millions of Files Full paper
• Emulating Goliath Storage Systems with David Full paper *Best Paper Winner*

An excellent conference. NetApp, EMC, Microsoft and IBM were recruiting.

The StorageMojo take
We’re still learning about flash, and the research presented here is a substantial addition to our meager knowledge.

Microsoft tells me they’re delivering major improvements to NTFS and Windows Server later this year. I’m looking forward to that briefing.

And it’s always a pleasure catching up with the people who, for some reason, never come to Sedona.

Courteous comments welcome, as always.

It’s that time of the year again: the Usenix File And Storage Technology conference in San Jose next Tuesday thru Thursday.

The elite StorageMojo analyst SWAT unit will be there, in color coordinated Kevlar and Spandex, rappelling into the Marriott ballroom. So come by and say hello.

The StorageMojo take
No obvious must-reads among the papers this year, so we’ll have to wait and see if lightning strikes.

Courteous comments welcome, of course.

The performance increase in individual CPUs is slowing to a crawl. All the easy wins – higher clock speeds, wider datapaths, more DRAM, larger registers and caches, 2-4 cores – have been exploited.

Doctor, is there any hope?
In the recent PCAST report on Federal technological initiatives (see Fed funding for our digital future) (pdf) one sidebar suggested “Progress in Algorithms Beats Moore’s Law.”

. . . in many areas, performance gains due to improvements in algorithms have vastly exceeded even the dramatic performance gains due to increased processor speed.

The algorithms that we use today for speech recognition, for natural language translation, for chess playing, for logistics planning, have evolved remarkably in the past decade. It’s difficult to quantify the improvement, though, because it is as much in the realm of quality as of execution time.

In the field of numerical algorithms, however, the improvement can be quantified. Here is just one example . . . a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later – in 2003 – this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms!

The StorageMojo take
Let’s file this one under “Wishful thinking” along with “US housing prices will never decline.” Piecemeal enhancements of specific application areas cannot replace the generalized performance improvements we’ve seen for decades.

No doubt there are important algorithmic improvements to be made. And that in certain problem spaces those speedups will far exceed Moore’s Law – even though the Law is about transistor count, not performance.

That doesn’t change the fact of computation today: the era of predictable and rapid performance improvement is over. Like a vein of rich ore that thins out, our computers will still improve, but the effort needed to do so is rising fast.

Cheap(er) SSDs, larger memories and caches are helping mask the performance plateau by increasing system performance, but reduced I/O latency and increased bandwidth will only take us so far. The way forward is a game of wringing out single-digit percent improvements, not the 2-3 year doubling of the last 60 years.

Courteous comments welcome, of course. The professor whose work the PCAST quote refers to is Martin Grötschel of Konrad-Zuse-Zentrum in Berlin. He’s been doing brilliant work on optimization problems- including the traveling salesman problem and data network design – for decades.

Talked to a company last week whose cloud app handles several billion transactions per month on a cluster. Sounds like SSDs could help them but how?

In a paper from the latest 5th Biennial Conference on Innovative Data Systems Research (CIDR ’11) researchers Philip A. Bernstein and Colin W. Reid of Microsoft and Sudipto Das of UC Santa Barbara have a suggestion: Hyder – A Transactional Record Manager for Shared Flash (pdf).

As underlying hardware changes – faster networks, large memories, multi-core CPUs and SSDs – database software architectures may change too. Hyder architecture supports

. . . reads and writes on indexed records within classical multi-step transactions. It is designed to run on a cluster of servers that have shared access to a large pool of network-addressable raw flash chips. . . . Hyder uses a data-sharing architecture that scales out without partitioning the database or application.

No partition scale-out
Today, most popular database clusters partition the database across multiple servers. Done well this works, but at some cost. The database design is non-trivial – cross-partition transactions, cache coherence, load balancing, scaling and multi-server debugging – are knotty issues which translate into higher design and operation costs.

Hyder eliminates partitioning, distributed programming, layers of cache, remote procedure calls and load balancing. All servers can read and write the entire database – so any server can execute any transaction. Load-balancing is simple: direct new transactions to lightly-loaded servers.

Each update transaction runs on one machine and writes to a shared log – so there’s no 2-phase commit. And no 2-phase commit locking, which can force performance off a cliff when workloads spike.

The 3 main components of Hyder are the log, the index and the roll-forward algorithm.

Log
The log runs on multiple flash devices – chips, DIMMs or ??? – and writes multi-page log records across multiple devices with parity to enable log recovery after device failures.

Hyder uses a multi-versioned database – old record versions aren’t updated-in-place, only the most recent version of a record is used – which has a couple of useful properties:

• Server caches are inherently coherent since only the most recent versions of records are used.
• Data can be read while writes are in progress.
• Queries that can be decomposed can be run across multiple servers concurrently for a faster response time.

[This may seem like voodoo to ACIDheads. A good technical intro to multi-versioning concurrency control (MVCC) is Multi-core software: to gain speed, eliminate resource contention.]

Servers run a cache update process that keeps them current with updated records. Server caches don’t have to be identical and the cache invalidate messages that most clusters use for cache coherency aren’t needed.

All log writes are idempotent appends, so if a write fails the server can simply reissue the write. The authors describe several error modes and how Hyder handles them.

Index
The index stores the database as a search tree with each node a [key, payload] pair. The tree can store, for example, a relational database. The index tree is also represented in the log.

Tree nodes are not updated in place. When node n is updated, a new copy – n’is created. Then, of course, the parent node must be updated and so on up the tree.

A binary tree minimizes the number of node updates, but can be processor intensive. The optimal tree structure for Hyder is not yet resolved.

Garbage collection is an issue. Each node pointer includes the ID of the oldest reachable data element. An element older than any that is pointed to by a node is garbage.

Roll-forward algorithm
This is the key process of Hyder.

When a record update begins, one server executes the transaction. The server is given a copy of the latest database root, a static snapshot of the entire database.

The updates are stored in a local cache and after execution the after-images are gathered into an intention record, which is broadcast to all servers and appended to the log. The update’s readset is included in the intention record, to insure all intentions are properly ordered, none are lost, and the offset is made known to all servers.

Each server can assemble a local copy of the tail of the log, which is used to determine if there are conflicting updates. The meld procedure manages conflicting updates.

Appending the intention to the database log doesn’t commit the transaction. The intention references the static snapshot of the latest database root. The meld procedure determines if any committed transactions since the snapshot conflict with the intention.

If they don’t, all is good. If they do, the transaction is aborted.

All servers roll forward using meld and don’t message each other about committed and failed transactions. Therefore there is no lock manager and no 2-phase commit.

Contention
Losing the lock manager and 2-phase commit should help performance unless other points of contention throttle the system. Hyder’s points of contention include appending intentions to the log, melding the log at each server, and aborting transactions.

Intention appends are serial. The lower the write latency the more appends can be written. A 10us write latency means a 100k TPS.

Network latency adds to write latency. Faster switches improve append performance.

The abort rate depends on the number of concurrent transactions that conflict. Fast transactions reduce the probability of aborts by reducing the number of concurrent transactions.

The worst case is a record subject to multiple updates from different servers. Detecting high-conflict transactions and serializing them by forcing them onto 1 server would reduce the hot data performance hit.

Performance
The authors model Hyder’s performance with a focus on the high-contention corner cases. In general, the tests show linear scaling as servers are added.

The problems come when the underlying hardware limits are exceeded. Increasing execution times mean more aborts and performance falls off a cliff. From the paper:

The StorageMojo take
We’ve been building disk workarounds for for decades. We now tend to assume those workarounds are fundamental architectural requirements rather than hacks.

The Hyder paper asks us to imagine a world where non-volatile mass storage is fast and cheap – and how we could re-architect basic systems to be faster and cheaper too.

The authors conclusion is a fair assessment:

Many variations of the Hyder architecture and algorithms would be worth exploring. There may also be opportunities to use Hyder’s logging and meld algorithms with some modification in other contexts, such as file systems and middleware. We suggested a number of directions for future work throughout the paper. No doubt there are many other directions as well.

Courteous comments welcome, of course. I hope to get to some of the other CIDR papers before FAST ’11 snows me under. Update: Phil Bernstein was kind enough to scan the post and I’ve updated 1 minor error. He also mentioned that it won the Best Paper award at the conference. Those CIDR folks have great taste in papers, don’t they?

CPU performance and clock speed have leveled out over the last several years. What does this mean for the industry?

Moore’s law
Strictly speaking, Moore’s law says that the number of transistors on a chip will double every 18 to 24 months. And that’s been true for the last 40 years. And it appears set to continue for another decade.

But Moore’s observation has been simplified to mean a doubling of performance every 18 to 24 months. And that too has been true. But not anymore.

Transistors and performance do not have a one-to-one relationship. Yes, clock speeds have improved from the 1 MHz 6502 processor in the original Apple II to over 3 GHz in the latest and greatest. But we’ve reached the end of the line in clock speed improvements: in a third of a nanosecond light moves about 4 inches or 10 cm. Not much distance when chips have miles of internal wiring.

But clock speed isn’t the whole story. Chips now move data in 64 and 128-bit chunks, rather than the 6502′s 8-bit chunks. While there are experiments with Very Long Word computer architectures, as a practical matter we were also at the end of the line for wider data paths as well: 256 bits is as wide as personal and commercial processors can reasonably use.

More RAM? We’ve also been adding ever-larger on-chip caches that improve performance. But inevitably cache-hit ratios decline with size and so do the performance benefits.

Multicore
We can’t make processors go faster. We can’t process more data per clock cycle. So how do we put twice as many transistors to work?

Stuffing more processors on a chip. And right now many of the brightest minds in computer science are struggling with the problem of getting usable work out of 8, 12 or 16 core CPUs.

Dual and quad core processors work pretty well because our multitasking operating systems run a lot of background threads. Spreading those threads across multiple cores improves performance for everyone.

But outside video, image, voice and scientific apps, most of what we do today – word processing, e-mail, web surfing, spreadsheets and presentations – don’t need multicore architectures. Certainly not 8 or more cores. Humans aren’t good multi-taskers.

The wall
We’ve hit a technology wall. We can still double the number of transistors on a chip every couple of years. We can still double disk drive capacity every 2 to 3 years. We can build faster interconnects, such as QuickPath, Light Peak and 10 Gb Ethernet.

But the easy wins are over. Going forward performance gains will be measured in single digit percents each year.

Implications
Information technology, like most of the US economy, is driven by consumer spending. So what happens when a new PC is only 20% faster than your fully paid for three-year-old PC?

Digital Equipment Corporation, who pioneered the minicomputer in the 1960s, had a simple model for product improvements. A successful product would add functionality and performance at a constant cost. And they would offer the same functionality and performance at a declining cost. Here’s a graph of their model:

At some point the cost of producing a given level of functionality would be so low that distribution and marketing costs would dominate. Then volumes would migrate to the price performance sweet spot and lower volume products died.

Today, we can no longer count on performance increases to open up new application territory. Therefore we will see differentiation move to what were once considered secondary characteristics.

• Power. The server space is grappling with the implications of greater power efficiency, but the mobile space has been pushing this metric for the last 15 years. That will continue for years to come.
• Integration. Open up in iPad or a MacBook Air and what do you see? A tiny PC board, a few chips and a huge set of batteries. Long battery life is what makes the product so convenient that they become part everyday life.
• Functionality. Creatively integrating multiple applications, each with their own dedicated core, may enable consumer devices to collapse multistep workflows into a single handy device. Combine image capture, voice recognition, editing and compression into a single device that would enable consumers to capture, edit and post video from a single candy bar sized device, editing on-the-fly with spoken commands.
• Cost. The first low-res digital cameras cost hundreds of dollars, but today we build them into cheap cell phones.

The StorageMojo take
The days of the Moore’s Law driven application growth are over. The next step is to use our still growing technical capabilities to refine what we already do.

The good news for the storage industry is that new data production will continue to grow rapidly. Always on, always available consumer data systems will create ever more demand for storage.

This is also another nail in the coffin of the RAID controller paradigm. Distributed multicore processing power requires distributed data protection and storage architectures.

When you can’t scale up, you have to scale out. Decomposable storage architectures will inevitably come to the fore.

Courteous comments welcome, of course. Oddly enough, the Apple ][ motherboard's style was the same as today's MacBook Air: a few chips on a PC board. Friends were always startled to see empty my Apple ]['s case was.

StorageMojo’s Global HQ is pulling up stakes and traveling to wilds of Palo Alto for the OpenStorage Summit 2010. I’m hoping to hear more about the future of ZFS and other storage stacks.

I’ll arrive Tuesday afternoon and will leave Thursday. If you’re in the neighborhood please stop by and say hello.

The StorageMojo take
The rapidly growing cloud storage market is slowly but surely changing the dynamics of the storage market. Private clouds will never be as flexible and cheap as public clouds, but the closer they get the more their economics will improve. It’s a game played at the margins.

That bodes well for enterprise adoption of open storage for private cloud infrastructure. Not everyone, but those with the scale and the moxie to make it work will find a significant competitive advantage.

Courteous comments welcome, of course.

One infrastructure to rule them all discussed the emerging enterprise need for a single, scalable file storage infrastructure. But what infrastructure?

Some background to this is last year’s Cloud Quadrant and this year’s Why private clouds are part of the future.

Block and file
For decades direct-attached block-based storage was the only option. The ’80s introduced file-based storage. Much of storage systems growth in the last 15 years has been in file servers.

New systems, be they video, sensor or social, are producing massive collections of files at an accelerating rate. The rapid development of lower cost mobile computing devices – smartphones, iPad’s, netbooks and Android tablets – mean that content consumption and production will be a major source of file growth. The long tail of content demand means that the variety of online content will grow – especially as the cost of storage declines.

Private cloud
The larger issue is the need to keep this fast-growing information online for years, despite rapid change in the underlying storage, network and computing infrastructures. File data must become independent of our storage and server choices.

As stores grow data migration becomes less feasible. Rip ‘n replace gives way to in-place upgrades.

Achieving that means moving to an object storage paradigm. How do we know this will happen? Because it already has.

Object stores at Google and Amazon Web Services are already among the largest storage infrastructures in the world. AWS alone stores over 100 billion objects today. Hundreds of millions of people use object storage every day – and don’t even know it.

What is object storage?
Object storage instantiations vary in detail and supported features. However, all object storage has two key characteristics:
–Individual objects are accessed by a global handle. The handle may, for example, be a hash, a key or a something like a URL.
–Extended metadata. The extended metadata content goes beyond that of traditional file systems and may include additional security and content validation as well as presentation, decompression or other information relating to the content, production or value of the enclosed file.

Like files, objects contain data. But they lack key features that would make them files. They don’t have:
-Hierarchy. Not only are all objects created equal, they all remain at the same level. You can’t put one object inside another.
-Names. At least, not human-type names like Claudia_Schiffer or 2006_Taxes.

A user-facing component provides those missing elements. You decide which files belong in which folders. You give the files names. You decide which users have access to which files and what those users can do with those files.

Those choices are embedded in the object metadata so they can be presented as you have organized them. But if you have the object’s handle you can access it directly.

All objects look alike. Some are bigger and some are smaller, but until we get them dressed and named, they aren’t files. Yet they are a lot closer to files than blocks are. Which means that if you choose to manage objects you no longer have to worry about blocks.

Essentially then, objects are files with an address – instead of a pathname – and extra metadata. Unlike distributed file systems – where the metadata is stored in a metadata server. The metadata server keeps track the location of the data on the storage servers.

Some file storage systems are built on object storage repositories. Legacy APIs make it a requirement for many applications, but URL-style access through HTTP is more flexible in the long run.

Crossing the implementation chasm
While the economics of objects are obvious at scale, they are less compelling at the beginning of a typical enterprise project. It is easier to buy another file server than to worry about long-term architecture.

Here’s a rough diagram of the relative scalability of storage options:

When under-12-month paybacks are expected, who will buy an object storage infrastructure? The simple answer is that as object stores become better known and startup costs are reduced, more companies will buy them. Archives will be the first market. The longer answer is that as public cloud projects are brought inside, object stores will receive them.

The StorageMojo take
As organizations amass large file collections, the economies of scale and management for object storage will become apparent. Savvy architects will add commodity-based scale-out object storage to their tool kit.

HDS, NetApp and HP have recently added modern object stores to their product lines. And rumor has it EMC will too, either by getting Atmos to work or by buying Isilon.

Courteous comments welcome, of course. Still don’t like the name object, but I’ll get over it.

The friendly folks at Scality have put up $100,000 to encourage open source development of useful cloud storage bits. It’s open to anyone, not just grad students.

Yup, it’s corporate self-interest at work – Scality sells object-based cloud storage software – but they’re taking an enlightened approach. The resulting code will be open source and is intended to work with a variety of object-based cloud storage services – such as S3 – not just theirs.

Real money
The defined projects have bonuses of $2,000 to $10k. The projects include a Gallery plugin for the

. . . full replacement of the underlying filesystem based storage of content/objects with object storage using the REST interface . . . .

There’s a WordPress plugin project to add an object storage backend to the popular CMD. The $10,000 prize goes for a KVM virtualization storage engine for the Linux Kernel Volume Manager that provides:

. . . block level storage volumes that can be attached to KVM virtual machines. The solution should not require any central node, for example, no central meta-data server and provide a completely stateless operation model.

Here’s the list of defined projects.

Even better
And almost ⅔ of the money remains uncommitted. If you have an idea for domesticating object storage in the cloud – propose it.

The StorageMojo take
Objects are the future of large-scale storage. If cutting edge stuff gets your heart pumping, this is a good place to start.

Or just collect some cash and move on. Your choice.

Feel free to ask questions in the comments. I’ll ping the Scality guys to get answers.

Courteous comments welcome, of course. I’ve done some work for Scality and like the team. I’m also wondering why Amazon hasn’t done something like this.

Concall today with Bryan Cantrill, the smart guy behind Dtrace. Dtrace was the engine behind Sun’s Oracle’s Fishworks server and application monitor. Dtrace has also been incorporated into OS X.

Bryan left Oracle last week and started Monday at Joyent the cloud infrastructure provider, as VP of engineering. Why?

Bryan is an instrumentation geek. He really wants to know what’s going on. Instrumentation in the cloud is the next big challenge.

That makes sense: there are so many moving parts that understanding and resolving performance and availability issues will be critical to the widespread adoption of cloud.

Tech epiphanies
Bryan described 3 technology epiphanies that he’s enjoyed. The 1st was when he saw Java for the first time back in 1995. The 2nd was when he saw a Ruby on Rails video about deploying a web app.

His 3rd epiphany came recently when he saw something called node.js. Developed by Ryan Dahl it turns the JavaScript paradigm on its head: node.js runs on the server, not the client.

Latency bubbles
We know that server I/O latency can kill performance. It’s even worse in the cloud.

A single bad drive can hose a server if the app is holding locks. What if you have a webpage that relies on five different Web services, or as many Amazon pages do, 150 services?

You need an infrastructure that is resilient in the face of long latency while maintaining high throughput. Bryan says that most failures are not hard failures but are latency bubbles that cascade out and lock up the rest of the infrastructure.

Ryan took Google’s of V8 JavaScript engine and extended it so you can handle long latency events. Without locking up the server.

Ryan does a fine job introducing node.js in a 1 hour Google Tech Talk last week. He outlined how to build a server that can handle 10,000 or more users. His goal with node.js was to make it easy to write high-performance servers.

There is an arms race out there for performance – Google, Apple, Mozilla, Opera, Microsoft – to win the hearts and eyeballs of hundreds of millions of consumers. Fickle consumers.

Node.js only exposes nonblocking asynchronous interfaces to the programmer. It has very few abstractions. Its power lies in the fact that it moves you away from certain interfaces like synchronous I/O that you shouldn’t do.

You don’t have to worry about some event completing and taking over while you’re in the middle of something else. Each node.js is a single thread. If you want to do more work you start multiple node.js instances and let the kernel do the load balancing.

Memory isolation is enforced at the process boundary. The kernel manages it, not the coder. That’s a good thing.

The StorageMojo take
Latency is the app killer of the cloud. The current cloud focus on write once/read never apps reflects that.

The fight against latency proceeds on many fronts: storage; network; CPU; and software. Asankya and others have good ideas for reducing Internet latency. Flash architectures are undergoing rapid evolution. Multicore and multiprocessor servers are attacking throughput.

Node.js is a big step in the right direction. Removing the dependency is that synchronous I/O create means any more resilient and higher performance infrastructure. Ryan reports that a Japanese website is already running several hundred thousand users on node.js instances.

As for Bryan, he’ll bring the same intelligence and energy to Joyent that he brought to Dtrace and Fishworks. Expect more great things.

Courteous comments welcome, of course. Update: The other smart guys behind Dtrace are the redoubtable Adam Leventhaland Mike Shapiro.