Spoke to Chris Gladwin, founder and CEO of Cleversafe at NAB 2012. Cleversafe had stopped communicating a few years ago – usually a bad sign – so an update was long overdue.

When last heard from, Cleversafe had an ISP/MSP target market, offered an open-source version of their software, and focused on safely archiving confidential data on public networks. No more.

A 50PB order
CEOs are professional optimists. But Chris’s story was good.

Their 1st order was 100TB. The 2nd, 50 petabytes. That is a lot of boxes to rack.

Now Cleversafe focuses on multi-petabyte orders. They can handle 5-7 such orders a year.

Patents
But when they aren’t installing petabytes of disk, they’re writing patents. Hundreds of them.

Chris thought they were up to 268 patent applications. The USPTO shows 21 granted patents, including 7,904,475 Virtualized data storage vaults on a dispersed data storage network, 7,853,710 Methods and devices for controlling the rate of a pull protocol, 7,844,712 Hybrid open-loop and closed-loop erasure-coded fragment retrieval process, 7,818,518 System for rebuilding dispersed data,” 7,818,430, Methods and systems for fast segment reconstruction, and 7,574,579 Metadata management system for an information dispersed storage system along with another 181 applications yet to be granted.

At a conservative $25k per patent in legal and filing fees and lost engineering time, that’s $6.7 million. If there’s another startup as aggressive on patents, I haven’t heard of it.

The StorageMojo take
It’s seems that most, if not all, of Cleversafe’s business comes from the US intelligence community, not commercial users. Otherwise we’d see reference sites and more interest from top-tier VCs.

Regardless, Cleversafe’s strategy of massive orders, massive patents and limited fulfillment is unlike any other in the industry. Is the limited fulfillment due to a complex product – the GPFS of scale-out storage – or a limited market?

As many HPC-focused firms have found, it can be difficult to shift from extremely specialized high-end government customers to commercial users. Companies that have, like Panasas, have had to work to keep their products general purpose, avoiding the honey-trap of fascinating but one-off designs.

Cleversafe’s pivot from the commercial market and open-source may reflect a 1st mover disadvantage: too early to the commercial market, they’ve been co-opted by the government market. But the bigger concern is whether or not that massive patent portfolio will stall development of better high-scale storage systems.

Cleversafe’s exit strategy seems to have almost as much to do with patents as it does with building a business. Are they the first storage company hoping for a buyout by a patent troll?

Courteous comments welcome, of course. I’ve recently done work for Panasas and am working with a company – Amplidata – being sued by Cleversafe.

Amazon Web Services architect James Hamilton has been posting on network issues for over a year and researching them much longer. As Ethernet becomes the de facto SAN technology, his views become more relevant to the larger storage market.

Critique
Part of Mr. Hamilton’s concern is the structure of the networking industry: the high margins; the dominance of a single player, Cisco; the closed technology; and the heavy vertical integration. All antithetical to the dynamics that have driven server costs down so successfully in the last 20 years.

These are issues the storage industry knows too well. But Mr. Hamilton is more concerned about the waste the current high-cost industry structure causes.

Waste?

Workload placement
The cost of network bandwidth leads to network over-subscription. Networks are configured as tree topologies: the further you move from end nodes the worse the over subscription.

As described in the 2009 Microsoft Research paper VL2: A Scalable and Flexible Data Center Network:

. . . the capacity between different branches of the tree is typically over- subscribed by factors of 1:5 or more, with paths through the highest levels of the tree oversubscribed by factors of 1:80 to 1:240. This limits communication between servers to the point that it fragments the server pool — congestion and computation hot-spots are prevalent even when spare capacity is available elsewhere.

This throttles data center performance by limiting server-to-server bandwidth, fragmenting resources and reducing network utilization. The latter reflects the redundant paths needed in case of switch failure: ≈50% or more of costly data center bandwidth goes unused.

As might be expected, big Internet data centers like Amazon’s have complex and unpredictable workloads. They need lots of bandwidth between all servers all the time.

A solution
The VL2 paper describes an experimental solution to these problems that includes location-specific and application-specific addressing, multi-path traffic load balancing and a novel directory design that efficiently handles lookups and updates to network mappings.

In an 75-node test cluster the design moved 2.75TB of data in 395 seconds – 94% of maximum network bandwidth – at a fraction of the cost of current enterprise networks. The paper calculates that a cloud-service scale network with no over-subscription could be built with commodity switches at 1/14th the cost of a traditional data center Ethernet.

Whoa!

The StorageMojo take
VC and engineering dollars follow high-growth markets. What Google, Amazon and Microsoft want, they get. With the rapid growth of public cloud services the network over-subscription problem will get solved.

Merchant silicon from Broadcom, Intel and Marvell is making a tried-and-true Moore’s Law attack on hardware cost. The protocol stack is tougher, but several open-source industry initiatives are under way with support from major companies. Progress will be slower than hoped, but within 3 years we’ll have a viable stack to build on.

Where does this leave the networking industry? That depends on where you sit.

Cisco will be the biggest loser, because they’ve been the biggest winner with the current model. They may need to pull an IBM and move big into services if they want to stick around. Ironically, Cisco’s UCS product line – which bakes in the tree-structured network – has further motivated broader industry action.

The rest of the industry can go after this emerging market with a lower-GM business model. Not all of them will, but it will be a critical success factor.

The big winner will be storage. Scale-out storage relies on spraying data across multiple racks for maximum availability, utilization and performance. Cheaper, faster, better scale-out networks will only drive storage demand.

For most of us this is an academic problem today. Lightly used systems – such as for backup and archiving – don’t see Amazon’s problems. But in 5 years this will be common even outside the public cloud providers.

Just as IT users have benefited from Google’s push on energy efficiency and much more, they will also benefit from much lower cost and more scalable networks.

Courteous comments welcome, of course. I can’t help but continue to marvel at how dumb Cisco’s UCS has turned out to be. It’s a gift that keeps on giving.

VMworld is the best storage show I’ve seen in years. VMware’s severe storage problems leave users hungry for solutions – and your friendly neighborhood storage industry is happy to oblige.

It’s almost as if VMware were owned by a storage company.

Flash everywhere
Fusion-io, Nimble Storage, Nimbus Data, Avere, Pure and more were talking about how well flash supports VMware. Fixes VDI boot storms, deduped VMDKs, I/O bound servers and much more.

Pure Storage
Here is Pure’s Matt Kixmoeller giving a nifty demo in this 50 second video:

Not exactly sure what those thousand VMs were doing. Maybe Pure will comment.

Falconstor
I lost track of Falconstor due to their OEM focus and sprawling product line. New CEO Jim McNiel has refocused the company – with the help of former Cheyenne teammates – on backup, business continuity/DR, dedup and virtualization.

Their clustered Network Storage Server turns all of Fstor’s products into tin-wrapped software suitable for channel partners. Takeaway: forget what you knew about them; they are a new company.

Virsto
While the release of their storage hypervisor for VMware makes them seem like a new company, Virsto has been shipping product for over a year, but on Hyper-V, not VMware. Microsoft lost interest in server virtualization and Virsto moved on.

Their product is a virtual appliance that:

. . . runs in each host, creating a transparent virtual storage layer that is thin provisioned, fully cluster-aware, supports very rapid snapshot and clone creation, and scales to support tens of thousands of high performance snapshots and clones.

Virsto . . . decouple[s] application performance from any dependence on the rotational latencies and seek times of underlying disk associated with random writes. All random writes are sequentialized and written directly to a transparent logging device . . . and then asynchronously de-staged to primary storage. . . .

Net/net: high performance virtual storage regardless of underlying physical storage. Virsto offers a free trial – if you try it, let me know how it works.

But wait! There’s more!
Cloud-related products from StorSimple, AMAX and Raidundant continue to pick at the problem of how/when/where cloud integrates with the enterprise.

The StorageMojo take
Many cool products and ideas. The storage problems of many virtual machines are not unlike those of earlier time-shared virtual memory systems, but the scale is much greater.

And when the scale is greater the problem is fundamentally different. As virtualization grows we’ll need to see more creative answers beyond deduplication and flash.

Courteous comments welcome, of course. Message to SNIA: storage networking is passé. Time to retool for the world of virtual machines, noSQL databases, scale-out storage and flash-enabled architectures. New name would be a start.

Running – or planning to run – some bigtime Java apps? A Silicon Valley startup named Cumulogic is looking for a few good beta testers to help them wring out their Java Paas.

Their ideal tester can use a Java PaaS running on either vSphere, Eucalyptus or Cloud.com private clouds or Amazon’s EC2. They currently support Ngnix 0.8, Apache 2.x, Tomcat 7.x, JBoss 4.x-6.x, and MySQL 5.x, MongoDB as well as connections to an Oracle database.

They know that a lot of people run WebLogic or Websphere and have those on their roadmap, but this is where they are today. If you have a Websphere cloud license they can support that.

Their control panel seemed reasonably complete in the demo I saw, but that’s the point of demos, isn’t it? If everything worked perfectly it wouldn’t be a beta.

The StorageMojo take
PaaS is a sweet spot: apps + data + computes all in one place. Is Java on PaaS an even sweeter spot? You tell me.

But the beta tester has an important role because development teams don’t know what they don’t know. The model user they have in mind is, in my experience, rarely seen in the wild.

It is that rawest market feedback dev teams get. If Java is important to you please register for the beta. Click around and you can find out more about them

If you do beta it I’d love to hear about your experience in the comments.

Courteous comments welcome, of course.

Commenters on the last post – Open source storage array – helped crystallize an idea that’s been lurking for years: comparing disk storage hardware on per-slot price. The Backblaze box, which costs about $50/slot, got a comment that said, in effect, “it doesn’t have the features of a $200/slot box.” Good!

But the comment raised an interesting point: since we all use the same disks from the same few – and soon to be fewer – manufacturers, isn’t the cost of the tin we wrap them in a key metric? Let’s call it PSC – Per Slot Cost.

Some advantages:

• Focus on value-add. We know how many disk slots there are in a storage system. We know how much disks cost. Therefore, the per-slot price tells us what the vendor’s value-add per disk is – or what we’re supposed to think it is.
• Increases pricing contrast. Disk costs are typically 10-15% of the price of a mid-to-high end array. The number of disk slots in those arrays vary, as do individual disk capacities. These variables obscure what the vendor is asking for their value add.
• Cleaner comparisons. As a corollary to the previous point, PSC makes it easier to compare architecturally similar systems – SAS vs SAS, hybrid SSD/SATA systems, RAID 6 systems – whose hardware cost structures should be similar.
• Focus on software value. Since most storage systems – even high-end systems – run on commodity hardware, the biggest price variable is in software. Isn’t that where we should focus?

The cloud storage angle
PSC should be useful for market segmentation. Instead of dumping arrays into entry-level price buckets – such as $75-$100k or $/GB – the PSC should track with the value of the stored data.

Expect to see segments range from Bulk (the Backblaze segment) to Heavy Transactional (traditional big iron) with yet-to-be-named segments between. But the most important use for PSC is in highly-scalable architectures in the public vs private cloud storage arena.

Cloud architectures are distinguished by the fact that the larger they scale, the lower their PSC. This is partly a function of economic necessity – who can afford 2 dozen PB of Symm? – and largely due to their use of software-based object replication instead of RAID.

When your storage is cheap, you can afford triple replication. And when you have massive numbers of boxes – and at least 2 data centers – you can have strong disaster tolerance. So large-scale cloud suppliers have motive and opportunity to reduce PSC.

The private cloud space is where the calculus gets interesting. Many observers dismiss the private cloud concept because they can’t possibly compete with Amazon, Microsoft and Google on scale or cost, including PSC.

The StorageMojo take
There is a private cloud market because there are other issues, such as network latency, and the commercialization of high-scale software such as Hadoop, that make it possible for any focused billion-dollar company to build a competitive cloud infrastructure. The hardware is already a commodity, and many of the improvements that Google 1st pushed, such as more efficient power supplies, are now widely available.

The bigger issue for competitive private clouds is the enterprise IT mindset that lacks the skills to specify and manage them. This is where PSC comes in: it allows CFOs to compare their costs to best-in-breed cloud providers in a simple way.

PSC is just a metric, not the metric. The big guys are optimizing things – like power distribution – that won’t move the needle for smaller players.

But if you use commodity hardware then you should focus on the software. And since every big player is already running on commodity hardware – a Good Thing, BTW – let’s focus on getting software that delivers business value. To the extent that PSC helps decision-makers do that, it will help the industry shift the focus from things like $/GB to a higher-level discussion.

Courteous comments welcome, of course. I just paid $250 per slot for an array with 1 controller, 1 fan and 1 Thunderbolt connection to my 1 desktop. Yes, I could have done better – if I didn’t want Thunderbolt. So PSC doesn’t trump all.

Most business files are only opened a few times, yet remain valuable enough to keep on line, just in case. That cold data is normally stored on high-performance, high-price NAS boxes at $$/GB.

Why?

2 years ago Backblaze, an online backup provider, open-sourced their storage pod design: 45 drives in a box (see Build a RAID 6 array for $100/TB). Now they’re back with v2: 45 3TB drives in a box with higher performance.

Backblaze now has over 16PB of storage pods in production.

Now for the good news
Backblaze isn’t in the box building business. They designed the storage pod for their backup business and released the plans out of the goodness of their hearts and for the free publicity.

I’ve thought that this could be a viable business for someone who doesn’t want to be the next NetApp or Isilon. Someone happy to build and ship boxes on a cost-plus basis to people who understand and can support a fault-tolerant software layer above the box, but who don’t have time to chase down miscellaneous hardware from vendors who prefer to sell in bulk.

That vendor has emerged: Protocase, the quick-turn enclosure shop that builds Backblaze’s enclosures.

I spoke to Protocase co-founder Doug Milburn – a PhD in mechanical engineering – today. Protocase will announce a complete just-add-drives storage pod: assembled, tested and software loaded box. Look for it in 2-4 weeks, priced at ≈$6k. With another $5500 for 3TB drives, it will come in at less than $90 per raw TB.

Why no drives? That’s the lion’s share of the cost and also the fastest to decline in price. They don’t need the inventory exposure and tech savvy shoppers can probably do better anyway. BTW, Backblaze has had good experience with the Hitachi HDS5C3030ALA630 drive.

The StorageMojo take
This will help energize the private cloud market by reducing the entry price. Amazon and Google don’t use NetApp or EMC. Why should you?

And the savings over renting cloud storage can be substantial as this Backblaze chart suggests:

True, Amazon provides many more services, but if you need petabytes for mini-bucks, this is hard to beat.

Courteous comments welcome, of course. Read about the v2 storage pod at the Backblaze blog post. Or get the shorter version in my ZDnet post “Build a 135TB array for $7,384.

Iron Mountain plans to either sell or shutter its digital archiving, eDiscovery and online backup and recovery solutions. Why?

Investors driving management
IM has been under pressure from a couple of funds that invested in the company and want board representation. The investors see a company that has built a national brand in a traditionally fragmented industry, but whose results have been hurt by – among other projects – its digital storage business.

As one them said in an SEC filing:

Management has done a commendable job building the Company’s North American Physical business, which currently represents 69% of 2010 Revenues and 86% of 2010 OIBDA. This business has an industry-leading market position, 44% margins and a sustainable, recurring revenue stream. . . .

Over the last decade, the Company has spent $2.7bn on its weak International Physical and ailing Worldwide Digital businesses, neither of which has come close to earning the margins or returns on capital of the North American Physical business. . . .

To us it hardly seems surprising that IRM is not competitively positioned against technology titans like EMC or Google in the digital marketplace.

(emphasis added) (OIBDA=operating income before depreciation & adjustments)

Update: Well, that didn’t take long. According to wide-awake reader Dave – see comments – IM has reached a “definitive agreement” to sell its digital business. Note to new owners: good luck! End update.

A long and winding road
IM acquired a developer of PC data protection, Connected, in November 2004. Since then they’ve bought LiveVault, DigiGuard, Anamnis, Accutrac, RMS Services, Stratify and, a year ago, Mimosa Systems.

In all, IM spent hundreds of millions on software companies. And, evidently, they spent a lot more on hardware: in a 2002 press release from EMC, IM’s CIO notes

. . . the advanced architecture of EMC systems and software makes it cost-efficient to grow our information infrastructure. Our EMC information storage capacity has nearly tripled in the past three years, yet the costs associated with managing this information have essentially stayed the same. . . .

For their Digital Archives division, EMC Celerra networked-attached storage systems will provide a high availability environment for Iron Mountain’s File Shares and Home Services.

Today’s commodity scale-out storage was only in the planning stages in 2002. But selecting EMC – a premium-priced provider – suggests that IM got off on the wrong foot and never recovered.

Pricing tells a tale
Looking at IM’s pricing is suggestive. They were charging a minimum of $0.35/GB/mo for Connected BackUp – 2-3x Amazon – plus the cost of initial setup and other server and client software charges.

The StorageMojo take
At those rates IM should have been profitable using anything. Therefore they were losing business to more cost-effective suppliers and earning anemic profits on the – heavily discounted? – business they did win.

It is no accident that Amazon and Google engineered their own storage. IM lacked the expertise to do it themselves and ended up in a cost bind.

It is a cautionary tale for would-be cloud storage providers: either get your costs competitive with AWS or get a clear differentiation. Most, like Nirvanix, Nexenta, Zetta and Atmos are going with the latter.

Courteous comments welcome, of course.

Amazon’s outage was caused by a failure of the underlying storage – the Elastic Block Storage. Here’s what they learned.

EBS
The Elastic Block Store (EBS) is a distributed and replicated storage optimized for consistent and low latency I/O from EC2 instances. EBS runs on clusters that store data and serve requests and a set of control services that coordinate and propagate I/Os.

Each EBS cluster consists of EBS nodes where data is replicated and I/Os are served. Nodes are connected by 2 networks: a primary high-bandwidth network for traffic between the EBS nodes and EC2 server instances; and a slower replication network intended as a backup and for reliable internode communication.

Newly written data is replicated ASAP. An EBS node searches the cluster for a node with enough capacity, connects to it and replicates the data, usually in milliseconds.

If connectivity to a node it is replicating to is lost the node assumes the other node failed and tries to find another node to replicate the data. In the meantime it holds onto all data until it can confirm the data is replicated.

The outage
During a network change on April 21 to upgrade primary network capacity a mistake occurred: the primary network data traffic was shifted to the slower secondary network.

The secondary network couldn’t handle the traffic which isolated many nodes in the cluster. Losing contact with nodes they were replicating to the remaining EBS nodes sought new nodes, but the few remaining nodes were quickly overwhelmed in a retry storm.

The now degraded secondary network then slammed the coordinating control services. Configured with a long timeout the retry requests backed up and the control services suffered thread starvation.

Once a large number of I/O requests were backed up the control services had no ability to service I/O requests and began to fail I/O requests from other Amazon availability zones. Within two hours the Amazon team had identified this issue and disabled all new create volume requests in the cluster.

But then another bug kicked in.

A race condition in EBS caused them to fail when closing a large number of replication requests. Because there were so many replication requests the race condition caused even more EBS notes to fail, re-creating the need to replicate even more data and again the control services were overwhelmed.

Recovery
The Amazon team get control of the replication storms in about 12 hours. Then the problem was recovering customer data.

Amazon optimizes its systems to protect customer data. When a node fails it is not reused until its data is replicated.

But since so many nodes were failed the only way to ensure no customer data was lost was by adding more physical capacity – no easy chore – but that wasn’t all.

The replication mechanisms had been throttled to control the storm, so adding physical capacity also meant delicate management of the many queued replication requests. It took the team 2 days to implement a process.

Amazon Relational Database Service
The Amazon Relational Database Service (RDS) uses EBS for database and log storage. RDS can be configured to operate within a single Amazon zone or replicated across multiple zones. Customers with a single zone RDS were quite likely to be affected, but a 2.5% of multi-zone RDS customers were affected as well due to another bug.

Lessons learned
The network upgrade process will be further automated to prevent a similar mistake. But the more important issue is to keep a cluster from entering a replication storm. One factor is to increase the amount of free capacity in each EBS cluster.

Retry logic will be changed as well to back off faster to focus on reestablishing connections first before more retries. And of course, the race condition bug will be fixed.

Finally, Amazon has learned it must improve the isolation between zones. They will tune timeout logic to prevent thread exhaustion, increase control services awareness of zone loads and, finally, move more control services into each EBS cluster.

The StorageMojo take
Data center opponents of cloud computing will point with alarm to this incident to make the case that they are still needed. But they forget that today’s enterprise gear is reliable only because of the many failures that led to better error handling.

While painful for the affected, the Amazon team’s response shows a level of openness and transparency that few enterprise infrastructure vendors ever display. Of course, that is due to the public nature of these large cloud failures; nevertheless the outcome is commendable.

But the battle is not only between large public clouds and private enterprise infrastructures, but between architectures. Traditionally, enterprise infrastructures have focused on increasing MTBF. Cloud architectures, on the other hand, have focused on fast MTTR – Mean Time To Repair.

What can be scaled up can also be scaled down. Not every application is suitable for public cloud hosting. But small-scale, commodity-based, self managing infrastructures are very doable. They are the bigger threat to the large proprietary hardware vendors of today.

Courteous comments welcome, of course. I speculated in Amazon’s experience: fault tolerance and fault finding about the cause of the failure, but I was wrong. A failure precipitated by a network upgrade? Way-y-y too simple.

Megastore handles over 3 billion writes and 20 billion reads daily on almost 8 PB of primary data across many global data centers.

In a paper by Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson, Jean-Michel Léon, Yawei Li, Alexander Lloyd, Vadim Yushprakh titled Megastore: Providing Scalable, Highly Available Storage for Interactive Services Google engineers describe how it works. From the abstract:

Megastore is a storage system developed to meet the requirements of today’s interactive online services. Megastore blends the scalability of a NoSQL data store with the convenience of a traditional RDBMS in a novel way, and provides both strong consistency guarantees and high-availability. We provide fully serializable ACID semantics within fine-grained partitions of data. This partitioning allows us to synchronously replicate each write across a wide area network with reasonable latency and support seamless failover between data centers.

The mission
Support Internet apps such as Google’s AppEngine.

• Scale to millions of users
• Responsive despite Internet latencies to impatient users
• Easy for developers
• Fault resilience from drive failures to data center loss and everything in between
• Low-latency synchronous replication to distant sites

The how
Scale by partitioning the data store and replicating each partition separately, providing full ACID semantics within partitions but limited consistency guarantees across them. Offer some traditional database features if they scale with tolerable latency.

The key assumptions are that data for many apps can be partitioned, for example by user, and that a selected set of DB features can make developers productive.

Availability and scale
To achieve availability and global scale the designers implemented two key architectural features:

• For availability, an asynchronous log replicator optimized for long-distance
• For scale, data partitioned into small databases each with its own replicated log

Rather than implement a master/slave or optimistic replication strategy, the team decided to use Paxos, a consensus algorithm that does not require a master, with a novel extension. A single Paxos log would soon become a bottleneck with millions of users so each partition gets its own replicated Paxos log.

Data is partitioned into entity groups which are synchronously replicated over a wide area while the data itself is stored in NoSQL storage. ACID transaction records within the entities are replicated using Paxos.

For transactions across entities, the synchronous replication requirement is relaxed and an asynchronous message queue is used. Thus it’s key that entity group boundaries reflect application usage and user expectations.

Entities
An e-mail account is a natural entity. But defining other entities is more complex.

Geographic data lacks natural granularity. For example, the globe is divided into non-overlapping entities. Changes across these geographic entities use (expensive) two-phase commits.

The design problem: entities large enough to make two-phase commits uncommon but small enough to keep transaction rates low.

Each entity has a root table and may have child tables. Each child table has a single root table. Example: a user’s root table may have each of the user’s photo collections as a child. Most applications find natural entity group boundaries.

API
The insight driving the API is that the big win is scalable performance rather than a rich query language. Thus a focus on controlling physical locality and hierarchical layouts.

For example, joins are implemented in application code. Queries specify scans or lookups against particular tables and indexes. Therefore, the application needs to understand the data schema to perform well.

Replication
Megastore uses Paxos to manage synchronous replication. But in order to make Paxos practical despite high latencies the team developed some optimizations:

• Fast reads. Current reads are usually from local replicas since most writes succeed on all replicas.
• Fast writes. Since most apps repeatedly write from the same region, the initial writer is granted priority for further replica writes. Using local replicas and reducing write contention for distant replicas minimizes latency.
• Replica types. In addition to full replicas Megastore has 2 other replica types:
  witness replicas. Witnesses vote in Paxos rounds and store the write-ahead log but do not store entity data or indexes to keep storage costs low. They are also tiebreakers when isn’t a quorum.
  Read-only replicas are the inverse: nonvoting replicas that contain full snapshots of the data. Their data may be slightly stale but they help disseminate the data over a wide area without slowing writes.

Architecture
What does Megastore look like in practice? Here’s an example.

A Megastore client library is installed on the app server. It implements Paxos and other algorithms such as read replica selection. The app server has a local replica written to a local BigTable instance.

A coordinator server tracks a set of entity groups and observes all Paxos writes. The coordinator is simpler than BigTable and serves local reads.

Concurrent with writing local data to BigTable and the coordinator the Megastore library is also writing to a second full replica: a replication server and a second coordinator. The stateless replication servers handle the writes to the remote big table while the lower latency coordinator handles any reads from the remote replica.

Failures may leave writes abandoned or in an uncertain state. The replication servers scan for incomplete writes and offer no op values via Paxos to complete the.

Availability
As coordinator servers do most local reads their availability is critical to maintaining Megastore’s performance. The coordinators use an out-of-band protocol to track other coordinators and use Google’s Chubby distributed lock service to obtain remote locks. If the coordinator loses a majority of its locks it will consider all entities in its purview to be out of date until the locks are regained and the coordinator is current.

There are a variety of network and race conditions that can affect coordinator availability. The team believes the simplicity of the coordinator architecture and their light network traffic makes the availability risks acceptable.

Performance
Because Megastore is geographically distributed, application servers in different locations may initiate writes to the same end entity group simultaneously. Only one of them will succeed and the other writers will have to retry.

Limiting writes to a few per second per entity group makes contention insignificant, e-mail for example.

For multiuser applications with higher write requirements developers can shard entity groups more finely or batch user operations into fewer transactions. Fine-grained advisory locks and sequencing transactions are other techniques to handle higher write loads.

The real world
Megastores been deployed for several years and more than 100 production applications using today. The paper provides these figures on availability and average latencies.

The high availability of the system architecture creates a nice-to-have problem: small transient errors on top of persistent uncorrected problems can cause much larger problems.

Fault tolerance makes finding underlying faults more difficult. The price of fault tolerance is eternal vigilance.

As the architecture diagram suggests Megastore doesn’t manage BigTable. Developers must optimize the storage for their app.

The StorageMojo take
As Brewer’s CAP theorem showed, a distributed system can’t provide consistency, availability and partition tolerance to all nodes at the same time. But this paper shows that by making smart choices we can get darn close as far as human users are concerned.

If Microsoft Office – or an open-source analog – could plug into a productized version of Megastore this could become popular for private cloud implementations: LAN performance in the office and global availability on the road. What’s not to like?

But whether that happens or not, the paper demonstrates again the value of Internet scale infrastructure thinking. Enterprise vendors would never have developed Megastore, but now that we’ve seen it work we can begin applying its principles to smaller scale problems.

Courteous comments welcome, of course. If this overview intrigues I urge you to read the entire paper as there are some interesting pieces I’ve left out.

Talked to a startup the other day that looks interesting – Zettar. It wasn’t the name that caught my attention.

Object storage is big in clouds. But objects aren’t compatible with standard apps: Powerpoint expects files, not objects.

Cloudstores like Amazon and Azure aren’t compatible with each other. Lock-in.

And you can’t run an internal cloud that is compatible with them either – they won’t sell you their software.

Double-secret lock-in.

Zettar founder Chin Fang’s idea is 2-fold.

• A virtual file system that front-ends several cloud services.
• Software that duplicates server-side cloud storage services.

The pitch:

The Zettar ZCloud Virtual Appliance (ZCloud) enables you to setup an Amazon S3 sandbox instantly on any computer, even a tiny netbook. Developers and QA engineers can use it to prototype, analyze, test, and stage a cloud application locally, before rolling it out onto Amazon Web Services (AWS). Thus, ZCloud can improve, simplify and speed up your development, and minimize AWS S3 development costs.

The StorageMojo take
Seems like a good idea. And Chin’s team did a good job on their web site.

Domesticating cloud storage will be a continuing process for the next 10 years. Zettar’s Z-cloud is a good start.

If you try out their software please tell us how it goes, I’m sure others would be interested.

Courteous comments welcome, of course. Don’t confuse Zettar with cloud NAS provider Zetta a cloud enterprise NAS provider who just raised another $11.5 million. I like them too.

One infrastructure to rule them all discussed the emerging enterprise need for a single, scalable file storage infrastructure. But what infrastructure?

Some background to this is last year’s Cloud Quadrant and this year’s Why private clouds are part of the future.

Block and file
For decades direct-attached block-based storage was the only option. The ’80s introduced file-based storage. Much of storage systems growth in the last 15 years has been in file servers.

New systems, be they video, sensor or social, are producing massive collections of files at an accelerating rate. The rapid development of lower cost mobile computing devices – smartphones, iPad’s, netbooks and Android tablets – mean that content consumption and production will be a major source of file growth. The long tail of content demand means that the variety of online content will grow – especially as the cost of storage declines.

Private cloud
The larger issue is the need to keep this fast-growing information online for years, despite rapid change in the underlying storage, network and computing infrastructures. File data must become independent of our storage and server choices.

As stores grow data migration becomes less feasible. Rip ‘n replace gives way to in-place upgrades.

Achieving that means moving to an object storage paradigm. How do we know this will happen? Because it already has.

Object stores at Google and Amazon Web Services are already among the largest storage infrastructures in the world. AWS alone stores over 100 billion objects today. Hundreds of millions of people use object storage every day – and don’t even know it.

What is object storage?
Object storage instantiations vary in detail and supported features. However, all object storage has two key characteristics:
–Individual objects are accessed by a global handle. The handle may, for example, be a hash, a key or a something like a URL.
–Extended metadata. The extended metadata content goes beyond that of traditional file systems and may include additional security and content validation as well as presentation, decompression or other information relating to the content, production or value of the enclosed file.

Like files, objects contain data. But they lack key features that would make them files. They don’t have:
-Hierarchy. Not only are all objects created equal, they all remain at the same level. You can’t put one object inside another.
-Names. At least, not human-type names like Claudia_Schiffer or 2006_Taxes.

A user-facing component provides those missing elements. You decide which files belong in which folders. You give the files names. You decide which users have access to which files and what those users can do with those files.

Those choices are embedded in the object metadata so they can be presented as you have organized them. But if you have the object’s handle you can access it directly.

All objects look alike. Some are bigger and some are smaller, but until we get them dressed and named, they aren’t files. Yet they are a lot closer to files than blocks are. Which means that if you choose to manage objects you no longer have to worry about blocks.

Essentially then, objects are files with an address – instead of a pathname – and extra metadata. Unlike distributed file systems – where the metadata is stored in a metadata server. The metadata server keeps track the location of the data on the storage servers.

Some file storage systems are built on object storage repositories. Legacy APIs make it a requirement for many applications, but URL-style access through HTTP is more flexible in the long run.

Crossing the implementation chasm
While the economics of objects are obvious at scale, they are less compelling at the beginning of a typical enterprise project. It is easier to buy another file server than to worry about long-term architecture.

Here’s a rough diagram of the relative scalability of storage options:

When under-12-month paybacks are expected, who will buy an object storage infrastructure? The simple answer is that as object stores become better known and startup costs are reduced, more companies will buy them. Archives will be the first market. The longer answer is that as public cloud projects are brought inside, object stores will receive them.

The StorageMojo take
As organizations amass large file collections, the economies of scale and management for object storage will become apparent. Savvy architects will add commodity-based scale-out object storage to their tool kit.

HDS, NetApp and HP have recently added modern object stores to their product lines. And rumor has it EMC will too, either by getting Atmos to work or by buying Isilon.

Courteous comments welcome, of course. Still don’t like the name object, but I’ll get over it.

The friendly folks at Scality have put up $100,000 to encourage open source development of useful cloud storage bits. It’s open to anyone, not just grad students.

Yup, it’s corporate self-interest at work – Scality sells object-based cloud storage software – but they’re taking an enlightened approach. The resulting code will be open source and is intended to work with a variety of object-based cloud storage services – such as S3 – not just theirs.

Real money
The defined projects have bonuses of $2,000 to $10k. The projects include a Gallery plugin for the

. . . full replacement of the underlying filesystem based storage of content/objects with object storage using the REST interface . . . .

There’s a WordPress plugin project to add an object storage backend to the popular CMD. The $10,000 prize goes for a KVM virtualization storage engine for the Linux Kernel Volume Manager that provides:

. . . block level storage volumes that can be attached to KVM virtual machines. The solution should not require any central node, for example, no central meta-data server and provide a completely stateless operation model.

Here’s the list of defined projects.

Even better
And almost ⅔ of the money remains uncommitted. If you have an idea for domesticating object storage in the cloud – propose it.

The StorageMojo take
Objects are the future of large-scale storage. If cutting edge stuff gets your heart pumping, this is a good place to start.

Or just collect some cash and move on. Your choice.

Feel free to ask questions in the comments. I’ll ping the Scality guys to get answers.

Courteous comments welcome, of course. I’ve done some work for Scality and like the team. I’m also wondering why Amazon hasn’t done something like this.

Concall today with Bryan Cantrill, the smart guy behind Dtrace. Dtrace was the engine behind Sun’s Oracle’s Fishworks server and application monitor. Dtrace has also been incorporated into OS X.

Bryan left Oracle last week and started Monday at Joyent the cloud infrastructure provider, as VP of engineering. Why?

Bryan is an instrumentation geek. He really wants to know what’s going on. Instrumentation in the cloud is the next big challenge.

That makes sense: there are so many moving parts that understanding and resolving performance and availability issues will be critical to the widespread adoption of cloud.

Tech epiphanies
Bryan described 3 technology epiphanies that he’s enjoyed. The 1st was when he saw Java for the first time back in 1995. The 2nd was when he saw a Ruby on Rails video about deploying a web app.

His 3rd epiphany came recently when he saw something called node.js. Developed by Ryan Dahl it turns the JavaScript paradigm on its head: node.js runs on the server, not the client.

Latency bubbles
We know that server I/O latency can kill performance. It’s even worse in the cloud.

A single bad drive can hose a server if the app is holding locks. What if you have a webpage that relies on five different Web services, or as many Amazon pages do, 150 services?

You need an infrastructure that is resilient in the face of long latency while maintaining high throughput. Bryan says that most failures are not hard failures but are latency bubbles that cascade out and lock up the rest of the infrastructure.

Ryan took Google’s of V8 JavaScript engine and extended it so you can handle long latency events. Without locking up the server.

Ryan does a fine job introducing node.js in a 1 hour Google Tech Talk last week. He outlined how to build a server that can handle 10,000 or more users. His goal with node.js was to make it easy to write high-performance servers.

There is an arms race out there for performance – Google, Apple, Mozilla, Opera, Microsoft – to win the hearts and eyeballs of hundreds of millions of consumers. Fickle consumers.

Node.js only exposes nonblocking asynchronous interfaces to the programmer. It has very few abstractions. Its power lies in the fact that it moves you away from certain interfaces like synchronous I/O that you shouldn’t do.

You don’t have to worry about some event completing and taking over while you’re in the middle of something else. Each node.js is a single thread. If you want to do more work you start multiple node.js instances and let the kernel do the load balancing.

Memory isolation is enforced at the process boundary. The kernel manages it, not the coder. That’s a good thing.

The StorageMojo take
Latency is the app killer of the cloud. The current cloud focus on write once/read never apps reflects that.

The fight against latency proceeds on many fronts: storage; network; CPU; and software. Asankya and others have good ideas for reducing Internet latency. Flash architectures are undergoing rapid evolution. Multicore and multiprocessor servers are attacking throughput.

Node.js is a big step in the right direction. Removing the dependency is that synchronous I/O create means any more resilient and higher performance infrastructure. Ryan reports that a Japanese website is already running several hundred thousand users on node.js instances.

As for Bryan, he’ll bring the same intelligence and energy to Joyent that he brought to Dtrace and Fishworks. Expect more great things.

Courteous comments welcome, of course. Update: The other smart guys behind Dtrace are the redoubtable Adam Leventhaland Mike Shapiro.

Cloud computing gets a bad rap because it can’t replace corporate data centers for mission critical apps. But new computing paradigms never do that: it is the new capabilities they enable that drive adoption. Case in point: transcoding.

Why?
Anyone who shoots video soon discovers that changing from, say, AVCHD to an editing-friendly codec and then to H.264 for distribution takes a lot of compute cycles. Conversion from one codec to another is called transcoding. It is the price we pay for high quality compressed content.

Compression and format conversion are necessary because highly compressed video – the kind most camcorders shoot – isn’t easy to edit. And the stuff that’s easy to edit has large files that chew up bandwidth and storage.

So we transcode. Add to that the number of formats we use – ranging from iPhones to flash to SD and 1080p – and transcoding is a major CPU cycle sink.

Fortunately, transcoding can be a highly parallel operation. A frame – or a series of frames – can be divided and split among multiple cores and CPUs.

Where?
Where can you find a lot of CPUs for a quick job? Right, the cloud. Which is why there are a number of online services that front-end Amazon Web Services to provide transcoding.

I spoke to the CEO of startup Zencoder, Jon Dahl to learn more.

Zencoder
Zencoder is a transcoding service provider that uses Amazon as a cloud provider. The Zencoder team has developed transcoding infrastructure for several startups and finally decided to build a general-purpose service.

While they use open source software in their stack – as do most transcoding providers – their major value-add is in a high-performance scalable interface. Handling 100,000 concurrent transcodes is non-trivial.

They also look out for problems common in transcoding such as audio/video getting out of sync and aspect ratio distortion. They can transcode 1080p faster than real time. And they’ve licensed the proprietary formats as well.

Amazon offers Linux as a service and a file service. S3′s files are limited to 5 GB, but that isn’t a problem for Zencoder: customers can specify input and output locations, bypassing Amazon storage.

Also they don’t transcode Mac ProRes – Final Cut Pro’s preferred editing format – today. But they do handle QuickTime movies.

The StorageMojo take
So the glass house doesn’t want to outsource cloud infrastructure. Who cares? They’re the last to adopt new technology anyway.

It is apps like transcoding that drive the business. In 5 years much, perhaps most, transcoding will be cloud-based.

Before the digital video craze in the last 5 years there wasn’t much demand for transcoding. But today, with HD video smartphones, millions are producing videos that they want to share and save.

Your smartphone won’t have the cycles to do it, but the cloud does. Expect transcoding vendors to add new features, such as noise-reduction or sharpening.

Business units are discovering the power of short videos to inform, train, persuade and excite. All at a fraction of the cost of 4-color brochures.

The outlook for storage vendors is mixed. Yes, much more storage will be sold – but cost-conscious cloud managers will be buying it. And as more new services develop on the cloud, consumers will be as hazy about “local” and “cloud” as they are about “memory” and “disk” today. Branding nightmare, but that’s where those petabytes will be.

Courteous comments welcome, of course.