Comments on: Building a 1.8 exabyte data center http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/ Data storage info & analysis Fri, 12 Mar 2010 00:35:33 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: joe http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198748 joe Thu, 04 Dec 2008 00:55:52 +0000 http://storagemojo.com/?p=973#comment-198748 Is this a joke? When you say "I dont know what the boxes would cost but it would be at least several times the racks.", are you budgeting $1000 for 10 Sunfire 4540 boxes? These numbers are way, way wrong. Is this a joke? When you say “I dont know what the boxes would cost but it would be at least several times the racks.”, are you budgeting $1000 for 10 Sunfire 4540 boxes? These numbers are way, way wrong.

]]> By: Bill Mottram http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198377 Bill Mottram Sat, 08 Nov 2008 18:12:16 +0000 http://storagemojo.com/?p=973#comment-198377 There is a limit to how many Sun fire 4540 can be mounted in one rack. The limit I understand is 4. This will increase the rack count significantly. There is a limit to how many Sun fire 4540 can be mounted in one rack. The limit I understand is 4. This will increase the rack count significantly.

]]> By: Joe Kraska http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198246 Joe Kraska Fri, 31 Oct 2008 23:01:14 +0000 http://storagemojo.com/?p=973#comment-198246 I think it's 1.6 racks per PB, formatted and usable on the DDN storage. They also have some nice power savings capabilities. Their RAID-6 rebuild on the fly and real time capability is, however, what I find most exemplary about DDN. You basically never even really need to know that RAID rebuild is going on. --Joe. I think it’s 1.6 racks per PB, formatted and usable on the DDN storage. They also have some nice power savings capabilities. Their RAID-6 rebuild on the fly and real time capability is, however, what I find most exemplary about DDN. You basically never even really need to know that RAID rebuild is going on.

–Joe.

]]> By: Emmanuel Florac http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198211 Emmanuel Florac Thu, 30 Oct 2008 11:51:53 +0000 http://storagemojo.com/?p=973#comment-198211 The new Datadirect 6620 may be a step in the good direction : 60 drives per 4U enclosure, with sophisticated power management ( automatic drive spindown). They actually seem to sell the setup S2A9900+6620 enclosure with 1TB drives, it's 1.2PB per 2 racks with 2 RAID controllers, something like 1PiB available storage (with parity and spare) with an average power consumption of 36.6kW. You'll have to had some servers to that setup, however. The new Datadirect 6620 may be a step in the good direction : 60 drives per 4U enclosure, with sophisticated power management ( automatic drive spindown). They actually seem to sell the setup S2A9900+6620 enclosure with 1TB drives, it’s 1.2PB per 2 racks with 2 RAID controllers, something like 1PiB available storage (with parity and spare) with an average power consumption of 36.6kW. You’ll have to had some servers to that setup, however.

]]> By: Joe Kraska http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198165 Joe Kraska Sat, 25 Oct 2008 17:37:46 +0000 http://storagemojo.com/?p=973#comment-198165 The exabyte data center is not far off from being real. We have been working a coming challenge problem for a three letter agency involving ingest and store rates of 1.5PB/day. That's about 17GB/s sustained write, 24/7/365. Individual streams can come in at 1+ GB/s. The real humdinger of it all is that the customer prefers disk and not tape for all storage. Neither dedup nor compression is possible with these data types (they are not duplicative, and they are already compressed). --Joe The exabyte data center is not far off from being real.

We have been working a coming challenge problem for a three letter agency involving ingest and store rates of 1.5PB/day. That’s about 17GB/s sustained write, 24/7/365. Individual streams can come in at 1+ GB/s.

The real humdinger of it all is that the customer prefers disk and not tape for all storage.

Neither dedup nor compression is possible with these data types (they are not duplicative, and they are already compressed).

–Joe

]]> By: Joe http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198163 Joe Sat, 25 Oct 2008 17:11:10 +0000 http://storagemojo.com/?p=973#comment-198163 I didn't see this post till today. It is interesting given what occurred at a small regional event we went to. We had a booth at a small event (Ohio Linux Fest: ... hey we sold a JackRabbit to an attendee from that last year, so we were hoping to replicate that success). You can see your humble contributer at some of the pictures <a href="http://scalability.org/?p=835" rel="nofollow">here</a> . The interesting thing was, during our booth time, someone came up to me and started grilling me on JackRabbit and ΔV capability. While Sun and others were there, he didn't quite look like your typical G2-gatherer. He finally caved, indicating he was looking to build an exabyte sized data center. I gave him a rough estimate of the number of units of anyone's storage, power consumption, costs, etc. Not that far off Robin's. I had chalked this up to a somewhat ... eccentric ... person asking if it were possible. I doubt this as a person from a three-letter-agency (though curiously we are getting more hits on the sites from the "Maryland procurement office" these days ... hmmmm). Now Robin, you are giving me pause to reflect upon this conversation. They asked me similar questions. If that person is the same one who asked you about this, and they are lurking, I am curious as to how serious they were. Intrigued, not from a vendor perspective, but from a management perspective. As you scale up the number of parts, your probability of failure over some interval approaches an asymptotic limit of 1. Which means that no matter what you do, you will always have to contend with some aspect of a failed part. This is IMO far more important than other considerations mentioned. Some mentioned de-dup as a technology to use for this, though this begs the question as to why they think that 90 GB data sets would have duplication in them? I would imagine that run length encoded compression may be more beneficial and far faster than de-dup. More to the point, I didn't get a sense from my questioner what their data sets were. I don't know if Robin got that. If this is imagery, and they want do avoid doing lossless compression, RLE and other lossless techniques could help, at the cost of processing power. If this is genomic or similar data, you have other techniques. De-dup doesn't quite factor into these. It seems to me that the phased build out approach would make the most sense. Moreover, someone suggested slotting in and out x4500's (ok, we would prefer <a href="http://jackrabbit.scalableinformatics.com" rel="nofollow"> JackRabbits</a>. This may be feasible, though rather than adapt the robotics to handle that, mount the units vertically, and use the robotic mechanisms to slot in and out drives into the chassis. The large tape storage folks could do this. Then the question of how to have some sort of file system handle this. You would need to envision some sort or large cache file system for handling inbound data, some sort of distributed meta-data mapper (standard meta-data plus a directory of where the data lives). Rather than de-dup, you would want to either dup or code the data to handle drive lossage. Would be interesting to talk about the tech behind this. I didn’t see this post till today. It is interesting given what occurred at a small regional event we went to.

We had a booth at a small event (Ohio Linux Fest: … hey we sold a JackRabbit to an attendee from that last year, so we were hoping to replicate that success). You can see your humble contributer at some of the pictures here .

The interesting thing was, during our booth time, someone came up to me and started grilling me on JackRabbit and ΔV capability. While Sun and others were there, he didn’t quite look like your typical G2-gatherer.

He finally caved, indicating he was looking to build an exabyte sized data center. I gave him a rough estimate of the number of units of anyone’s storage, power consumption, costs, etc. Not that far off Robin’s.

I had chalked this up to a somewhat … eccentric … person asking if it were possible. I doubt this as a person from a three-letter-agency (though curiously we are getting more hits on the sites from the “Maryland procurement office” these days … hmmmm).

Now Robin, you are giving me pause to reflect upon this conversation. They asked me similar questions. If that person is the same one who asked you about this, and they are lurking, I am curious as to how serious they were. Intrigued, not from a vendor perspective, but from a management perspective. As you scale up the number of parts, your probability of failure over some interval approaches an asymptotic limit of 1. Which means that no matter what you do, you will always have to contend with some aspect of a failed part. This is IMO far more important than other considerations mentioned. Some mentioned de-dup as a technology to use for this, though this begs the question as to why they think that 90 GB data sets would have duplication in them? I would imagine that run length encoded compression may be more beneficial and far faster than de-dup.

More to the point, I didn’t get a sense from my questioner what their data sets were. I don’t know if Robin got that. If this is imagery, and they want do avoid doing lossless compression, RLE and other lossless techniques could help, at the cost of processing power. If this is genomic or similar data, you have other techniques. De-dup doesn’t quite factor into these.

It seems to me that the phased build out approach would make the most sense. Moreover, someone suggested slotting in and out x4500’s (ok, we would prefer JackRabbits. This may be feasible, though rather than adapt the robotics to handle that, mount the units vertically, and use the robotic mechanisms to slot in and out drives into the chassis.

The large tape storage folks could do this. Then the question of how to have some sort of file system handle this. You would need to envision some sort or large cache file system for handling inbound data, some sort of distributed meta-data mapper (standard meta-data plus a directory of where the data lives). Rather than de-dup, you would want to either dup or code the data to handle drive lossage.

Would be interesting to talk about the tech behind this.

]]> By: Nicolai Plum http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198045 Nicolai Plum Sat, 18 Oct 2008 20:56:58 +0000 http://storagemojo.com/?p=973#comment-198045 About the "OSHA won't let you put ten X45xx in a rack": the weight of the J4500 is specified at 77kg. Most racks and the datacentre floors they sit on are rated for something like 400kg including the weight of the rack, PDU, etc. I expect that Sun's 4 boxes-in-the-rack figure is based on what's generally achievable in most datacentres today. Sure, you could get racks that will hold 800kg, and floors to hold up the rack-and-storage combo that weighs in at about 1000kg/rack all up, but they're not standard items. Might want to use racks sitting on the subfloor, run the power and network and air overhead, and talk to the architect about the building's design parameters. You'll also still need a baby forklift to get the units into the top of the rack. Then each rack is going to consume and emit 11KW, heavy duty air circulation will be required. About the “OSHA won’t let you put ten X45xx in a rack”: the weight of the J4500 is specified at 77kg. Most racks and the datacentre floors they sit on are rated for something like 400kg including the weight of the rack, PDU, etc. I expect that Sun’s 4 boxes-in-the-rack figure is based on what’s generally achievable in most datacentres today.
Sure, you could get racks that will hold 800kg, and floors to hold up the rack-and-storage combo that weighs in at about 1000kg/rack all up, but they’re not standard items. Might want to use racks sitting on the subfloor, run the power and network and air overhead, and talk to the architect about the building’s design parameters.
You’ll also still need a baby forklift to get the units into the top of the rack.
Then each rack is going to consume and emit 11KW, heavy duty air circulation will be required.

]]> By: Richard B http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198020 Richard B Fri, 17 Oct 2008 15:49:16 +0000 http://storagemojo.com/?p=973#comment-198020 Ewen, and others who’ve mention de-dup ratios, you should take a large pinch of salt with vendors’ claims about the reduction they can effect. Data Domain, EMC Avamar and others talk in the context of *backup* data - that is repeated storage of very similar data, so they are able to point to huge reductions. You will clearly see a lot of block level duplication in this sort of environment. When you’re looking at primary storage at GreenBytes seem to be, and NetApp are now starting to, the type of data is all important. If this is archived data it could be scanned images, in which case the amount of duplication could be close to zero. Ewen, and others who’ve mention de-dup ratios, you should take a large pinch of salt with vendors’ claims about the reduction they can effect. Data Domain, EMC Avamar and others talk in the context of *backup* data – that is repeated storage of very similar data, so they are able to point to huge reductions. You will clearly see a lot of block level duplication in this sort of environment. When you’re looking at primary storage at GreenBytes seem to be, and NetApp are now starting to, the type of data is all important. If this is archived data it could be scanned images, in which case the amount of duplication could be close to zero.

]]> By: Joe Kraska http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198015 Joe Kraska Fri, 17 Oct 2008 02:13:48 +0000 http://storagemojo.com/?p=973#comment-198015 BTW, the 1.8 exabyte archive: how fast does it fill up? If it takes a year, I believe that this is ~60 GB/s ... 24/7/365. So this archive may have some needs for a few fairly hefty OC links as well. Joe. BTW, the 1.8 exabyte archive: how fast does it fill up? If it takes a year, I believe that this is ~60 GB/s … 24/7/365. So this archive may have some needs for a few fairly hefty OC links as well.

Joe.

]]> By: Joe Kraska http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-198014 Joe Kraska Fri, 17 Oct 2008 02:10:16 +0000 http://storagemojo.com/?p=973#comment-198014 Another issue with data storage at this scale will be namespace management. 1.8 exabytes divided by typical maximum volume sizes today will lead to a volume proliferation nightmare, upon which much very nasty OPEX could very easily rest. As far as who is most suited for this activity now, I'd take a look, perhaps, at HP's Extreme Data Store when combined with PolyServe. You have both industry-leading density (1.6 racks per PB) as well as fairly large volumes. This, assuming that you wish to store everything on spinning media. BTW, I did an analysis like this very recently for a future sustained 17GB/s problem (sustained: 24/7/365). That's a 1.5PB/day write rate. Kinda big, eh? Anyway, the problem is much more achievable than it sounds if the problem is in 2013. In 2013, I would expect 8TB SATA drive (or something equivalent) to be readily procured at or less than today's 1TB drives. Others mentioned deduplication and compression, but there is no mention regarding the duplicative nature of the data or its current compression. I will say this: it sounds a bit like a content addressable storage (CAS) problem. The CAS space seems to be kind of formative to me, but some of those technologies might complement a truly gigantic archive. Finally: I would think such an archive would be dying for ILM of some sort. Joe. Another issue with data storage at this scale will be namespace management. 1.8 exabytes divided by typical maximum volume sizes today will lead to a volume proliferation nightmare, upon which much very nasty OPEX could very easily rest. As far as who is most suited for this activity now, I’d take a look, perhaps, at HP’s Extreme Data Store when combined with PolyServe. You have both industry-leading density (1.6 racks per PB) as well as fairly large volumes.

This, assuming that you wish to store everything on spinning media.

BTW, I did an analysis like this very recently for a future sustained 17GB/s problem (sustained: 24/7/365). That’s a 1.5PB/day write rate. Kinda big, eh? Anyway, the problem is much more achievable than it sounds if the problem is in 2013. In 2013, I would expect 8TB SATA drive (or something equivalent) to be readily procured at or less than today’s 1TB drives.

Others mentioned deduplication and compression, but there is no mention regarding the duplicative nature of the data or its current compression. I will say this: it sounds a bit like a content addressable storage (CAS) problem. The CAS space seems to be kind of formative to me, but some of those technologies might complement a truly gigantic archive.

Finally: I would think such an archive would be dying for ILM of some sort.

Joe.

]]> By: Matt L http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-197989 Matt L Wed, 15 Oct 2008 01:32:55 +0000 http://storagemojo.com/?p=973#comment-197989 What about Data DeDup options like Data Domain, EMC, FalconStor and others. Even Compressed Data can be Dedupped at the block level and only Unique Data will be Archived. Now admittedly I am a little green in the storage game but what type of availiblity do you require for the data. How quickly does it need to be accessed, how will it be accessed? What about Data DeDup options like Data Domain, EMC, FalconStor and others. Even Compressed Data can be Dedupped at the block level and only Unique Data will be Archived. Now admittedly I am a little green in the storage game but what type of availiblity do you require for the data. How quickly does it need to be accessed, how will it be accessed?

]]> By: Kevin Closson http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-197985 Kevin Closson Tue, 14 Oct 2008 15:21:01 +0000 http://storagemojo.com/?p=973#comment-197985 I didn't see the word compression anywhere in this post (did I speed read?). Since it isn't mentioned then we are either fantasizing about 1.8 exabyte of compressed data (perhaps 6 to 8 exabytes of uncompressed data). If the "Thumpers" (SunFire 45XX) were to write uncompressed data as fast as the AMD HT 2.0 interconnect could sustain it would take something like 3 years of nonstop maxed-out writing of random bytes to lay 1.8 exabyte onto disk. Did the Soviet's start talking about strapping Yuri a lightning bolt and sending him to another solar system after his first lap around the earth ? :-) I didn’t see the word compression anywhere in this post (did I speed read?). Since it isn’t mentioned then we are either fantasizing about 1.8 exabyte of compressed data (perhaps 6 to 8 exabytes of uncompressed data). If the “Thumpers” (SunFire 45XX) were to write uncompressed data as fast as the AMD HT 2.0 interconnect could sustain it would take something like 3 years of nonstop maxed-out writing of random bytes to lay 1.8 exabyte onto disk.

Did the Soviet’s start talking about strapping Yuri a lightning bolt and sending him to another solar system after his first lap around the earth ? :-)

]]> By: Jake http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-197984 Jake Tue, 14 Oct 2008 14:19:06 +0000 http://storagemojo.com/?p=973#comment-197984 Fully loaded capacity is 512TB each, so yes you would need more than my original petabyte calculation. It still is a far better choice than a DIY solution using Sun boxes to front end. Fully loaded capacity is 512TB each, so yes you would need more than my original petabyte calculation. It still is a far better choice than a DIY solution using Sun boxes to front end.

]]> By: Robin Harris http://storagemojo.com/2008/10/12/building-a-18-exabyte-data-center/comment-page-1/#comment-197982 Robin Harris Tue, 14 Oct 2008 13:47:35 +0000 http://storagemojo.com/?p=973#comment-197982 Jake, did you slip a few decimal points? It looks like a fully expanded DS 8300 is about a thousand drives in a couple of racks. You'd need over a thousand to reach 1.8 EB. Robin Jake, did you slip a few decimal points? It looks like a fully expanded DS 8300 is about a thousand drives in a couple of racks. You’d need over a thousand to reach 1.8 EB.

Robin

]]>