Comments on: MaxiScale’s Web-scale file system http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/ Data storage info & analysis Sun, 01 Aug 2010 02:16:15 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Robin Harris http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205714 Robin Harris Sun, 04 Oct 2009 02:08:48 +0000 http://storagemojo.com/?p=1608#comment-205714 Jeff, Good point. I've got a call into a contact at IBRIX/HP. Let's see if he can clear it up. Robin Jeff,

Good point. I’ve got a call into a contact at IBRIX/HP. Let’s see if he can clear it up.

Robin

]]> By: Jeff Darcy http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205686 Jeff Darcy Fri, 02 Oct 2009 12:42:33 +0000 http://storagemojo.com/?p=1608#comment-205686 Are you sure the claim was about 1000+ servers, not clients? The well publicized IBRIX deployment at Pixar involved thousands of *rendering* servers, which are clients as far as the filesystem is concerned. Even the biggest Lustre or Panasas HPC sites (e.g. ORNL or LANL respectively) don't have that many servers, because even with their many thousands of clients they don't need that many. If there's really a thousand-server configuration somewhere, especially one supporting commensurate I/O rates (at least 500GB/s), I'd expect it to be big news reported many places. It sure would be a shame to see other vendors or projects dinged for only supporting 32 or 128 servers because IBRIX had deployed 1000 clients, don't you think? Are you sure the claim was about 1000+ servers, not clients? The well publicized IBRIX deployment at Pixar involved thousands of *rendering* servers, which are clients as far as the filesystem is concerned. Even the biggest Lustre or Panasas HPC sites (e.g. ORNL or LANL respectively) don’t have that many servers, because even with their many thousands of clients they don’t need that many. If there’s really a thousand-server configuration somewhere, especially one supporting commensurate I/O rates (at least 500GB/s), I’d expect it to be big news reported many places. It sure would be a shame to see other vendors or projects dinged for only supporting 32 or 128 servers because IBRIX had deployed 1000 clients, don’t you think?

]]> By: Robin Harris http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205679 Robin Harris Thu, 01 Oct 2009 22:41:45 +0000 http://storagemojo.com/?p=1608#comment-205679 Jeff, IBRIX made the claim to me in person several times. HP reiterated the claim this week at the tech day I attended. IIRC, the site they referred to was supposed to be running the IBRIX client software which maintains a client side index and reduces latency and backend chatter. They've been consistent about this for some time, to me anyway. Robin Jeff,

IBRIX made the claim to me in person several times. HP reiterated the claim this week at the tech day I attended. IIRC, the site they referred to was supposed to be running the IBRIX client software which maintains a client side index and reduces latency and backend chatter. They’ve been consistent about this for some time, to me anyway.

Robin

]]> By: Jeff Darcy http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205676 Jeff Darcy Thu, 01 Oct 2009 21:26:20 +0000 http://storagemojo.com/?p=1608#comment-205676 The only place I can find the claim about Ibrix running on 1000+ nodes is on this site. Can the claim be substantiated? The only place I can find the claim about Ibrix running on 1000+ nodes is on this site. Can the claim be substantiated?

]]> By: Nick Kirsch http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205542 Nick Kirsch Mon, 28 Sep 2009 11:15:25 +0000 http://storagemojo.com/?p=1608#comment-205542 Thanks, Robin - great discussion. I'm glad to see the presence of scalable architectures being discussed more and more often. Isilon has many huge internet-scale customers - including very large photo sites, leading social networks, outsourced IT providers, etc. Many of these are well over a petabyte and some are getting to the 10's of PBs (sometimes in the same filesystem, sometimes not). Today's single file system limits (deployed, not theoretical) are 5 PBs, 144 nodes, and 50 Gbps - but the more compelling reason why customers choose Isilon is not simply because of the large numbers we post, but rather, the integral business challenges we're solving. Ultimately Isilon customers look not only to saving money (in terms of CapEx AND OpEx) but also to accelerating productivity while minimizing business risk. If you consider the total cost of ownership, I challenge anyone to find a system which is more reliable, easier to manage and grow, and higher performing than an Isilon cluster. Nick http://twitter.com/Isilon_Nick Thanks, Robin – great discussion. I’m glad to see the presence of scalable architectures being discussed more and more often.

Isilon has many huge internet-scale customers – including very large photo sites, leading social networks, outsourced IT providers, etc. Many of these are well over a petabyte and some are getting to the 10′s of PBs (sometimes in the same filesystem, sometimes not).

Today’s single file system limits (deployed, not theoretical) are 5 PBs, 144 nodes, and 50 Gbps – but the more compelling reason why customers choose Isilon is not simply because of the large numbers we post, but rather, the integral business challenges we’re solving.

Ultimately Isilon customers look not only to saving money (in terms of CapEx AND OpEx) but also to accelerating productivity while minimizing business risk. If you consider the total cost of ownership, I challenge anyone to find a system which is more reliable, easier to manage and grow, and higher performing than an Isilon cluster.

Nick
http://twitter.com/Isilon_Nick

]]> By: MaxiScale « すでにそこにある雲 http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205414 MaxiScale « すでにそこにある雲 Fri, 25 Sep 2009 09:17:09 +0000 http://storagemojo.com/?p=1608#comment-205414 [...] MaxiScale’s Web-scale file system [...] [...] MaxiScale’s Web-scale file system [...]

]]> By: Robin Harris http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205381 Robin Harris Thu, 24 Sep 2009 00:03:26 +0000 http://storagemojo.com/?p=1608#comment-205381 While I've watched Isilon since 2001, I don't think they'd argue that they are a web-scale or massively-scalable system ready to drive Internet-facing apps. Nor should they want to be since that is not where the market is. For example, IIRC they currently top out at 128 nodes, which, I hasten to add, is not an architectural limit but a testing one. IBRIX claimed at least one client with multiple PB running on about 1,000 nodes. And while Isilon prices are competitive, they aren't white box commodity prices. Isilon might have a marketing play in the "private cloud" meme because that won't be driven by economies of scale as much as the public cloud market. Their ease of management will be a win there, as it is in the media market today. I like the IBRIX architecture a lot, but I'm happy to believe that something better is out there - and maybe MaxiScale is it. While I’ve watched Isilon since 2001, I don’t think they’d argue that they are a web-scale or massively-scalable system ready to drive Internet-facing apps. Nor should they want to be since that is not where the market is.

For example, IIRC they currently top out at 128 nodes, which, I hasten to add, is not an architectural limit but a testing one. IBRIX claimed at least one client with multiple PB running on about 1,000 nodes. And while Isilon prices are competitive, they aren’t white box commodity prices.

Isilon might have a marketing play in the “private cloud” meme because that won’t be driven by economies of scale as much as the public cloud market. Their ease of management will be a win there, as it is in the media market today.

I like the IBRIX architecture a lot, but I’m happy to believe that something better is out there – and maybe MaxiScale is it.

]]> By: Joe Kraska http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205372 Joe Kraska Wed, 23 Sep 2009 19:40:21 +0000 http://storagemojo.com/?p=1608#comment-205372 HP and Ibrix: HP, having acquired not just IBRIX but /also/ Polyserve /and/ LeftHand networks, is looking like it's assembling a powerful arsenal of IP, wouldn't you say. I.e., it certainly looks as if it's TRYING to corner cluster file systems, doesn't it? (even if they patently have not done so, YET). Anyway, IBRIX can do blocks and IO's. Isilon isn't marketed for either. A bit of a shame, that, because I think Isilon could do iSCSI passably well. Isilon's distributed clustered architecture suggests it would adapt to scsi IO-redirect protocol capability naturally. iSCSI LUN distributed across a whole Isilon cluster, anyone? I covet, I covet. But alas, no. Joe Kraska San Diego CA USA HP and Ibrix:

HP, having acquired not just IBRIX but /also/ Polyserve /and/ LeftHand networks, is looking like it’s assembling a powerful arsenal of IP, wouldn’t you say. I.e., it certainly looks as if it’s TRYING to corner cluster file systems, doesn’t it? (even if they patently have not done so, YET). Anyway, IBRIX can do blocks and IO’s. Isilon isn’t marketed for either.

A bit of a shame, that, because I think Isilon could do iSCSI passably well. Isilon’s distributed clustered architecture suggests it would adapt to scsi IO-redirect protocol capability naturally. iSCSI LUN distributed across a whole Isilon cluster, anyone?

I covet, I covet. But alas, no.

Joe Kraska
San Diego CA
USA

]]> By: Kevin Closson http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205369 Kevin Closson Wed, 23 Sep 2009 18:03:52 +0000 http://storagemojo.com/?p=1608#comment-205369 Gary Orenstein... I wondered where he landed after Gear6... Gary, if you see this I'm saying, "Hello." Sounds like interesting stuff... Gary Orenstein… I wondered where he landed after Gear6… Gary, if you see this I’m saying, “Hello.”

Sounds like interesting stuff…

]]> By: Nick Kirsch http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205351 Nick Kirsch Wed, 23 Sep 2009 04:47:11 +0000 http://storagemojo.com/?p=1608#comment-205351 Robin, Just curious why you believe IBRIX is the end-all-be-all with respect to the massively scalable market VS say ... Isilon? How do you segment the "massively scalable market" and what are the key properties a solution needs in order to serve it? Thanks. Nick http://twitter.com/Isilon_Nick Robin,

Just curious why you believe IBRIX is the end-all-be-all with respect to the massively scalable market VS say … Isilon? How do you segment the “massively scalable market” and what are the key properties a solution needs in order to serve it?

Thanks.

Nick
http://twitter.com/Isilon_Nick

]]> By: Shehjar http://storagemojo.com/2009/09/21/maxiscales-web-scale-file-system/comment-page-1/#comment-205308 Shehjar Tue, 22 Sep 2009 08:55:13 +0000 http://storagemojo.com/?p=1608#comment-205308 Re point 1, it is a wrong notion I've read in other places also, that, machine/server-level replication can be a complete replacement for disk-level replication like in RAID1, for all types of setup. Both operate at different levels of the storage stack and have their own pros and cons. Which one or what combination of the two gets used should be a function of the particular deployment. I think, or rather hope, MaxiScale is not limiting users to the peer-level replication only. Re point 2, I am probably not understanding it right, but how is the concept of peer-set a wrinkle. One would think that peer-sets are actually reducing the size of the domain within which contention can happen for meta-data. On the other hand, it is still not clear how smaller peer-sets are aggregated into a cluster without, in turn, causing meta-data lock contention among the various peer-sets. <i>They also run MapReduce to manage the cluster. A massively parallel data management tool running a massively parallel cluster. Cool.</i> Heres hoping it does not require a PhD in CS to manage the cluster. ;) Re point 1, it is a wrong notion I’ve read in other places also, that, machine/server-level replication can be a complete replacement for disk-level replication like in RAID1, for all types of setup. Both operate at different levels of the storage stack and have their own pros and cons. Which one or what combination of the two gets used should be a function of the particular deployment. I think, or rather hope, MaxiScale is not limiting users to the peer-level replication only.

Re point 2, I am probably not understanding it right, but how is the concept of peer-set a wrinkle. One would think that peer-sets are actually reducing the size of the domain within which contention can happen for meta-data.
On the other hand, it is still not clear how smaller peer-sets are aggregated into a cluster without, in turn, causing meta-data lock contention among the various peer-sets.

They also run MapReduce to manage the cluster. A massively parallel data management tool running a massively parallel cluster. Cool.
Heres hoping it does not require a PhD in CS to manage the cluster. ;)

]]>