Oh, and QLogic switches most definitely do NOT have mellanox stamped on the chips inside their switches. They DO however have QLogic stamped on them.

Uhhh… RHEL runs on cell processors…

http://www.linuxdevices.com/news/NS5481356092.html

Yellow Dog Linux has been running on the PS3 since release.

Well. Point to you.

When, and not until, pNFS is ratified and a primary part of their product, then it will be a commodity… to the extent that there are substitutes. That’s realistically 12-18 months away.

Just don’t kid yourself. The TCO of a hand-rolled Lustre cluster is astronomical… for those of us who distinguish CAPEX vs OPEX, those “open source” clusters are looking not so attractive.

While I understand the conversation was introduced as pertaining to an HPC product, some of us have production “five nine” interests in these technologies.

Joe Kraska
San Diego, CA
USA

As for commoditization, I think it all misses the point. The real commodities are DRAM die and silicon wafers. I know, from having performed the experiment, that “custom” silicon can compete in HPC against mass produced processors.

The problem with building things out of “PC” components often boils down to things that LINPACK won’t show: reliability, memory bandwidth, and communication performance. Only the last of these is a strong lever against LINPACK. There’s a big difference between a component or node designed for a desktop and one designed for a 3000 node cluster. In the first case, an MTBF of longer than two years is irrelevant, in the latter it is woefully inadequate.

RoadRunner is a really interesting experiment, and it will surely succeed in its special purpose deployment. However, its balance is a little odd. With a peak compute rate of 100GF per chip, and “only” 24GB/s of memory bandwidth, the FP to MEM ratio is better than many x86 solutions, but a little light for many HPTC applications.

Disclosure: I’m Chief Engineer at SiCortex and blog at http://www.bigncomputing.org

I guess we just disagree on what “commodity” means, then. To me, a commodity is something that you can buy on an open market without needing to consider the source, like wheat or pork bellies. Secondarily, competition between practically-anonymous sources means reduced prices and trivial replacement of one supplier with another. *None* of that happens if there’s a single source, and “turnkey” has absolutely nothing to do with it. The old “pre-commoditization” HPC systems that Robin mentions were often presented as “turnkey” too. There’s nothing wrong with Panasas, I was an architect for one of their direct technological predecessors, but what they produce is not a commodity. It’s a specialized product that even HPC buyers consider high-priced, and if you knew HPC buyers like I know HPC buyers you’d find that pretty noteworthy.

“f you compare where HPC was 20 years ago - hand-tooled from the CPU to the OS and compilers to low-volume networks like FDDI - today’s HPC is a commodity play.”

FDDI was more of a commodity than IB is. No matter whose namepate is on an IB switch, the chips inside will all have “Mellanox” stamped on the top. At least FDDI had multiple suppliers from sand to switches. Yes, you can mix and match CPUs and operating systems and interconnects and storage more now in HPC. There’s less vendor lock-in across the entire system, and that’s one aspect of commoditization. If you look at the markets individually, though, only some of them have been commoditized. The OS on the Opterons is a case in point, and parallel filesystems are heading that way because there are already two significant open-source competitors (Lustre and PVFS) and open-source pNFS coming Real Soon Now. On the other hand, a custom processor running a specialized OS on a custom blade connected via a single-source interconnect to a home-grown I/O node bridging to a proprietary storage system doesn’t say “commoditization” to me.

True, auto engines aren’t as commoditized as CPUs.

But if you compare where HPC was 20 years ago - hand-tooled from the CPU to the OS and compilers to low-volume networks like FDDI - today’s HPC is a commodity play.

Not as commoditized as the data center, which is behind the consumer, but the trend is clear.

And who knows, maybe the PowerXCell 8i will show up in the PS4?

Robin

I don’t know what the I/O nodes do, but I cannot agree with the above sentence. Panasas is a turnkey appliance. One can hardly be more commoditized than that. “Proprietary” does not make something not a commodity.

Are the I/O nodes Infiniband gateways? Seems likely.

Joe.

There’s some commoditization going on, there always has been and always will be, but I think it’s less than you make out. Once you factor out the changes that are actually side effects of the shift from shared-memory systems (the most complex and expensive piece of custom stuff in the old days) to clusters, I’d say the custom/commodity balance hasn’t changed all that much. If there is a commoditization trend, Roadrunner is bucking it on the storage side by using a proprietary solution and home-grown I/O nodes instead of an open-source filesystem and COTS storage servers directly on the IB fabric.