Comments on: Fat trees and skinny switches http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/ Data storage info & analysis Sun, 20 May 2012 13:26:14 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Ethernet Fabric « Pronto Systems http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-215020 Ethernet Fabric « Pronto Systems Sat, 12 Feb 2011 18:37:19 +0000 http://storagemojo.com/?p=895#comment-215020 [...] have seen a lot of discussions on the benefit of fabric architecture, including higher bandwidth, lower latency, lower cost, and [...] [...] have seen a lot of discussions on the benefit of fabric architecture, including higher bandwidth, lower latency, lower cost, and [...]

]]> By: Steve Jones http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197276 Steve Jones Tue, 26 Aug 2008 08:02:49 +0000 http://storagemojo.com/?p=895#comment-197276 The price parity between 10 bytes of network traffic, and 1 MB of disk bandwidth refers to WAN network, not LAN. As the article refers to data centre articles, I'm not sure of the relevance. Indeed if you read the distributed computing economics paper, then it has price parity between LAN and disk bandwidth (about 1TB per $). For data centres that's not suprising - big iron SAN switches and LAN switches are both expensive items. In many cases, it's even the same basic frames. That's not to say that there isn't some potential value in using commodity switches for some types of connection, but the reality of many data centres is that extensive use is made of network virtualisation in SAN and Ethernet connections. commodity switches generally can't do those sort of things. As for the energy consumption per GBps and trhe heat dissipation, the chart shows an approximate 3:1 reduction for both. That's hardly suprising - unless something has gone seriously wrong with the law of conservation of energy (or somehow a very large proportion of the "big iron" switches' power is being disipated in the form of electro-magnetic radiation outside the data centre (hardly likely) then pretty well all that electrical energy eventually comes out as heat. (Yes, there are a few little niceties about power factors, reactive loads and the like, but generally electrical power in = heat out to a very close order in a closed environment). The price parity between 10 bytes of network traffic, and 1 MB of disk bandwidth refers to WAN network, not LAN. As the article refers to data centre articles, I’m not sure of the relevance. Indeed if you read the distributed computing economics paper, then it has price parity between LAN and disk bandwidth (about 1TB per $). For data centres that’s not suprising – big iron SAN switches and LAN switches are both expensive items. In many cases, it’s even the same basic frames. That’s not to say that there isn’t some potential value in using commodity switches for some types of connection, but the reality of many data centres is that extensive use is made of network virtualisation in SAN and Ethernet connections. commodity switches generally can’t do those sort of things.

As for the energy consumption per GBps and trhe heat dissipation, the chart shows an approximate 3:1 reduction for both. That’s hardly suprising – unless something has gone seriously wrong with the law of conservation of energy (or somehow a very large proportion of the “big iron” switches’ power is being disipated in the form of electro-magnetic radiation outside the data centre (hardly likely) then pretty well all that electrical energy eventually comes out as heat. (Yes, there are a few little niceties about power factors, reactive loads and the like, but generally electrical power in = heat out to a very close order in a closed environment).

]]> By: bofkentucky http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197272 bofkentucky Mon, 25 Aug 2008 22:55:59 +0000 http://storagemojo.com/?p=895#comment-197272 Check out http://aggregate.org/FNN/ for more research on maximizing bisection bandwidth while controlling per-port costs for clusters. Check out http://aggregate.org/FNN/ for more research on maximizing bisection bandwidth while controlling per-port costs for clusters.

]]> By: Som http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197270 Som Mon, 25 Aug 2008 17:59:59 +0000 http://storagemojo.com/?p=895#comment-197270 You can achieve the same using existing switches using ECMP. The first level switches do not even need to run IP - they can be L2 switches. HPC clusters have been using two and multi-layer CLOS configuration for at least seven/eight years. What's the benefit of the two level routing table scheme? ( I have not gone through the full paper yet - I hope to do that shortly - maybe the answer will be obvious) Som Sikdar You can achieve the same using existing switches using ECMP. The first level switches do not even need to run IP – they can be L2 switches.

HPC clusters have been using two and multi-layer CLOS configuration for at least seven/eight years.

What’s the benefit of the two level routing table scheme?
( I have not gone through the full paper yet – I hope to do that shortly – maybe the answer will be obvious)

Som Sikdar

]]> By: Amin Vahdat http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197269 Amin Vahdat Mon, 25 Aug 2008 14:03:21 +0000 http://storagemojo.com/?p=895#comment-197269 Thank you very much for your note and for the article. I think that the article is well done. As co-author of the original article, I have a few comments: - You mention that "The power efficiency argument is not going to be that persuasive. The real win is in the dollars." I would argue that, if you care to, you can reduce the power efficiency argument to dollars. For PC's in a data center, the cost to cool the machines dominates the hardware cost over a 2-3 year period. There will be increasing pressure on the IT industry both from within (dollars) and without (carbon footprint, regulation, etc.) to limit energy usage. - You describe a "maximum capacity 27,648 node cluster using 48 port switches". In fact, we based most of our discussion on 48-port switches because they currently offer the most performance per dollar and are a good match to current rack sizes in the data center. However, one could use 64-port switches in a fat tree topology to build out a 65,536 node cluster or 96-port switches to build out a 221,184 node cluster. One nice property of a fat tree is that the number of hosts that it supports grows with the cube of the number of ports in the baseline switching element. More specifically, using k-port switches, a fat tree can support (k^3)/4 hosts. - Finally, I think that one of the compelling arguments for this approach is that it does not rely upon aggregation to higher speed links to achieve its scale (relative to traditional tree-based techniques). Thus, if you wanted to build out a cluster with 10 GigE all the way to the edge servers, you would not have any options today using traditional techniques because there is no standard for higher-speed Ethernet (e.g., 40 GigE) yet available. Even when it does become available, available switches will have low port density and, typically, prohibitive costs. The fat tree approach uses identical switching elements throughout the structure, meaning that it could deliver 10GigE to the edge even today. Thank you very much for your note and for the article. I think that the article is well done. As co-author of the original article, I have a few comments:
- You mention that “The power efficiency argument is not going to be that persuasive. The real win is in the dollars.” I would argue that, if you care to, you can reduce the power efficiency argument to dollars. For PC’s in a data center, the cost to cool the machines dominates the hardware cost over a 2-3 year period. There will be increasing pressure on the IT industry both from within (dollars) and without (carbon footprint, regulation, etc.) to limit energy usage.
- You describe a “maximum capacity 27,648 node cluster using 48 port switches”. In fact, we based most of our discussion on 48-port switches because they currently offer the most performance per dollar and are a good match to current rack sizes in the data center. However, one could use 64-port switches in a fat tree topology to build out a 65,536 node cluster or 96-port switches to build out a 221,184 node cluster. One nice property of a fat tree is that the number of hosts that it supports grows with the cube of the number of ports in the baseline switching element. More specifically, using k-port switches, a fat tree can support (k^3)/4 hosts.
- Finally, I think that one of the compelling arguments for this approach is that it does not rely upon aggregation to higher speed links to achieve its scale (relative to traditional tree-based techniques). Thus, if you wanted to build out a cluster with 10 GigE all the way to the edge servers, you would not have any options today using traditional techniques because there is no standard for higher-speed Ethernet (e.g., 40 GigE) yet available. Even when it does become available, available switches will have low port density and, typically, prohibitive costs. The fat tree approach uses identical switching elements throughout the structure, meaning that it could deliver 10GigE to the edge even today.

]]> By: Robin Harris http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197267 Robin Harris Mon, 25 Aug 2008 07:29:59 +0000 http://storagemojo.com/?p=895#comment-197267 Anon, You've noted a "wordo" - an artifact of learning to write with my mouth. I've started using dictation software. The spelling is good, but it misses entire words and I haven't yet figured out how to catch those. Please bear with me. Robin Anon,

You’ve noted a “wordo” – an artifact of learning to write with my mouth.

I’ve started using dictation software. The spelling is good, but it misses entire words and I haven’t yet figured out how to catch those.

Please bear with me.

Robin

]]> By: Anonymous http://storagemojo.com/2008/08/24/fat-trees-and-skinny-switches/comment-page-1/#comment-197264 Anonymous Mon, 25 Aug 2008 03:00:47 +0000 http://storagemojo.com/?p=895#comment-197264 "fat tree"? "fact tree"? Which is it? “fat tree”? “fact tree”? Which is it?

]]>