Think high-end enterprise requirements

  • Mission critical reliability
  • High performance & scalability
  • “One throat to choke” support

Would you expect to find free open-source software crowding out closed proprietary solutions? It is at JPMorgan and other financial services companies.

Enterprise message bus
No, this isn’t storage software, but it is close.

I came across this article Toward a Commodity Enterprise Middleware in ACM Queue (ACM membership required, I think) by John O’Hara, a senior architect and distinguished engineer at JPMorgan. In it he describes the forces and the process that led to the development of AMQP (Advanced Message Queuing Protocol), an open spec EMB with several open-source instantiations which I list later.

I found it interesting because of the parallels I see between EMBs and data storage, especially high-end data storage. The business models of the companies that support AMQP are interesting as well, since enterprises must have good support even for free software.

RAID is middleware between servers and disks
EMB’s are middleware that enable independent applications to work together through message passing. Common protocols include store-and-forward, publish and subscribe and others. They commonly run over Ethernet and Infiniband and are designed to be usable on many operating systems and programming languages.

Big banks are pushing EMB performance thanks to automated trading systems that can generate as many as 500,000 events per second. The reliability of the EMB under heavy load is paramount.

Why an OSS EMB?
The usual reasons. Proprietary EMBs are costly. They don’t play well with other EMBs. And the banks found that their needs often outstripped what the closed vendors could provide. John got tired of waiting for an open source version and led the effort himself.

He notes that to make AMQP happen he felt he had to meet some some requirements. I’ll quote John at length:

This had to be an industry initiative. Home-grown middleware could not thrive in the small market available within a host organization, even the largest host.

It is also notable that pervasive networking standards such as Ethernet, the Internet Protocol, e-mail, and the Web share some traits. They are all royalty-free and unencumbered by patents, they are all publicly specified, and they all shipped with a useful early implementation for free. The combination of freedom and usefulness drives their adoption when predicated on fitness for purpose.

To succeed, AMQP needed to adopt these same characteristics:

  • It needed to be a fully defined, open, royalty-free, unpatented specification to enable anyone to implement a compatible service. Furthermore, the standard specification had to be clearly separate from the implementations; otherwise, it would not be a fair market for commercial entities to enter. AMQP had to be appealing for commercial implementation and exploitation or it would not succeed.
  • AMQP needed to have real implementations of the specification; otherwise, the specification would not be immediately useful or interesting to front-line developers with pressing needs. Ideally, it should have more than one implementation to qualify as a potential IETF draft standard. So, there are real implementations you can run today (as detailed later in the article).
  • AMQP software had to be proven in live systems. Middleware is a critical piece of any system and must be trusted. That trust has to be earned. To this extent, it was clear we would have to deploy an implementation in a high-profile, mission-critical application to assuage the fears of other early adopters. So, a combination of OpenAMQ and Qpid are live at JPMorgan, supporting 2,000 users on five continents and processing 300 million messages per day.
  • Finally, and most importantly, AMQP needed to be a collective effort. Openness to partnership and the ideas of others had to be there from the beginning. To this end, we carefully selected a partner to co-develop the specification and implement the software. We chose iMatix, a boutique European development house that had clearly demonstrated a commitment to open source and sound ethics, and had a strong engineering background and excellent writing abilities.

Because the project was sponsored by a bank, it also had to “wash its own face,” as they say. This was not a research project. Through sheer good luck, there was a need to refresh some large systems with very specific requirements. This provided a tangible return for AMQP investment, so I was able to convince a forward-looking CIO that AMQP was the way to go.

John goes on to make valuable points about other issues such as governance (where Java stumbled), user-driven implementation and architecture, which are important for any product development effort, with governance the special issue for OSS.

Here’s a list of suppliers and resources on AQMP

The StorageMojo take
The emergence of commodity cluster-based storage – as epitomized by the Google FS – has demonstrated the value of an open implementation of a similar product. We have a venture-backed OSS router company, but no storage companies. Are VC’s clueless about storage? Or is storage doomed to remain riven by proprietary products with poor interoperability?

Maybe some of the Google engineers whose options have vested will start the company to commoditize cluster storage. Storage is a conservative realm, but there are enough high-growth, high-capacity apps to create an opening for the right product. And even storage people will change if they have good reason to.

Comments welcome, of course. And what is the name of that GFS-like OSS project? Keywords are no good if you can’t remember them! Update: the first commenter, Jim, got the one I was thinking of: Hadoop DFS. Thanks!