36 Comments

1. 

   Carter George on Tuesday, 29 July, 2008 at 2:42 pm

   Robin,
   This guy is spot on in his criticism of what I’d call “Green 1.0”, which amounts to “throw away your existing storage and buy my newer, more efficient storage.” I agree that that’s not going to be a winning strategy for most customers, not least because storage is “stickier” than servers – people stick with their suppliers, and hate storage migrations. My discussion of the “O Tier” goes into detail, at: http://storageoptimization.wordpress.com/2008/07/23/capacity-optimized-storage-the-emergence-of-the-o-tier/.

   In essence, Denegri is missing two key points:
   First, the overall amount of storage customers have is growing exponentially, so their existing storage footprint is relatively small relative to the footprint they’ll have in a few years. I just talked to a customer who – in their 100 year history – has accumulated 25 Terabytes of data. Their projections show them having 250 Terabytes by the end of next year. So by end of 2009, 90% of their storage footprint will be net new storage.

   If you were to apply Mr. Denegri’s car analogy to that, if you told someone that starting tomorrow they’d have to drive 100 times more miles per week than they do now, do you think they might be more inclined to buy a hybrid?

   Second, Green 1.0, or buying new storage, is not the only answer. Green 2.0 – which is what Ocarina does – amounts to “store ten times more data on disks you already have”. By using a combination of online storage optimization techniques (subfile deduplication, next-generation compression, and others), Ocarina is able to get up to 10:1 data reduction results for online storage (not for repetitive backups, but for online data sets).

   Again, to use the car analogy, do you think in today’s market if there were a product that got you 10x more gas mileage on your existing car that that might sell well? The final thing that’s worth noting about the car analogy is one that EMC and NetApp might take to heart. In the big automotive industry transition from the ‘70s to today, in which the vendor to OEM ratio shrank so dramatically, the other thing that happened was that new leaders emerged to take market leadership away from the long time leading OEM’s.

   In the ‘70s, Ford and GM dominated the auto industry; their failure to respond quickly enough to changing demands by consumers for more energy efficient cars led to the rise of Toyota and Honda as dominant suppliers. Ford and GM were chased in to the niches where gas guzzling was still OK – pickup trucks and mini-vans. Now that energy costs have gone up once again, and those last two bastions of the domestic car business are collapsing, Ford and GM are in big, big trouble. The same thing could well happen in storage – new vendors may emerge who respond better to the data center’s need for storaging more stuff at drastically lower watts per Terabyte. The big vendors will have the first shot to defend their turf, but if they don’t, the door is open for those who will provide both Green 1.0 and Green 2.0 storage technology.

2. 

   Ray Lucchesi on Tuesday, 29 July, 2008 at 4:48 pm

   Power curves notwithstanding, storage power use per GB or per IOPS is on an overall decline.

   The problem is one half utilization issues (poor utilization rates drive more GB than needed which drive more Watts than needed which drives more cooling than needed) and one half technology limitations. Rotating media takes a certain amount of power to drive an HDA. And then there is always tape if power is a real concern

   There are a plethera of ways to address the utilization issue, Thin provisioning, deduplication, and Sun/IMM being just some examples. If power costs continue to skyrocket I predict utilization rates will start to creep up to the wall (70-90% utilized) from where it is today in open systems (20-40% utilized) using all the known tools and then some.

   The technology considerations can be dealt with also. 3.5″ form factor will give way to 2″ form factor and power consumption will go down. SSD flash has some interesting considerations when you compare Watts/IOP on SSDflash to hard drives – there is almost orders of magnitude differences in power. (imagine your car could get not 30 miles to the gallon but 300 or 3000 this would change the economic reality considerably). Basic hard drive capacity doubling every 18 to 24 months takes the Watt/GB and effectively halves it – as the same power is required to power a 500GB drive as it takes to power a 1TB drive. Finally. there is always tape and/or optical if power is a real concern, nothing takes less power than media sitting on a shelf

   I don’t see any of this as show stoppers for the storage market place. More or less business as usual, technology runs into barriers and it takes real thinking but breaking those barriers turn out to be obvious over time.

   It’s kind of an emotional argument to equate the discussion with the car market but if anything the barriers to entry in the storage business are coming down not going up. To start a car company today might take $100M to $10B or more. To start a storage company takes $10-30M. And thats going down not up or at worst is staying flat. How many of those will make the big time is anyone’s guess but when new constraints emerge is a prime time for transition, especially if the big guys are asleep at the wheel. But as far as I can see nobody is sleeping.

   Ray Lucchesi

3. 

   Bill Todd on Wednesday, 30 July, 2008 at 12:25 am

   Wow – this article is wrong on so many levels that the mind boggles at just how little competence it must take to be a ‘star industry financial analyst’ these days.

   1. Let’s take an ‘enterprise’ disk using a full 15 W 24/7 running flat out. At the average 2007 commercial rate of $0.07/KWH specified in one of the author’s citations (though in some parts of the country it’s of course higher) it will require the princely sum of about $46 to power it for the disk’s entire nominal 5 year service life. Adjust upward by a bit to allow for power supply inefficiencies and then double the result to take cooling costs into account if you want to: it’s *still* only a small fraction of the cost of the disk, let alone the far greater cost of managing its data over that period of time.

   In other words, *even if electricity costs rose by a full decimal order of magnitude* when you include the cost of managing the storage there wouldn’t be anything like the kind of energy-cost-driven storage crisis that he envisions – even if you ignore the fact that storage energy use can (and very likely will) *drop* by close to an order of magnitude (per disk, let alone per GB) as the industry shifts to far more energy-efficient 2.5″ drives over the next few years.

   So my first conclusion is that competence in grade-school arithmetic must definitely *not* be one of the qualifications required of a ‘star industry financial analyst’.

   (Incidentally, the above also contains the answer to Robin’s question, “Higher energy costs create massive problems for airlines – why not for IT?”: the reason is that energy costs for storage are an insignificant part of its total cost, whereas they’re not an insignificant part of the cost of operating airplanes.)

   2. As Carter just observed, the consolidation of the U.S. auto industry had nothing to do with lack of available energy: it was, rather, driven by the presence of external competition able to deliver more fuel-efficient vehicles at a time when consumers perceived this as important to them (despite the fact that gas *still* didn’t cost diddly-squat compared to its real value). If the storage industry were faced with competition which could provide storage at half the energy cost there’d be some similar movement (despite the fact that the required energy similarly doesn’t amount to diddly-squat compared with the value we get from it) – but since the only more energy-efficient competitively-performing alternatives to disk storage are vastly more expensive to purchase than the energy they’d save over their lifetimes no situation analogous to that which the U.S. automotive industry faced obtains.

   So my second conclusion is that an understanding of the historical events which he attempts to use for analogies is definitely not one of the qualifications required of a ‘star industry financial analyst’.

   3. The reason that storage power consumption is increasing at a faster rate than networking and server power consumption is a no-brainer: it’s because demand for storage capacity is increasing faster than demand for processing and network bandwidth. So what? The real question is whether storage demands for power are in any danger of overwhelming available supply (the ‘energy ceiling’ that he babbles about), not which horse happens to be ahead by a nose at the second turn in the Power Sweepstakes – and as observed in point 1 above the answer is a definite ‘No’: if we can’t afford the electricity to run our storage, we’ll be in far worse shape because of all the other, more critical things we can’t afford to power (though given the eminently feasible potential for obtaining electricity at reasonable cost from renewable sources the only reason this may happen is due to complete incompetence on our part).

   So my third conclusion is that an ability to provide relevant information (rather than irrelevant statistics and ‘The End Of The World Is Nigh’ admonitions) must not be one of the qualifications required of a ‘star industry financial analyst’.

   4. The storage industry is nothing like the automotive industry in terms of barriers to entry: its relatively small number of fundamental components are increasingly based on commodity products and thus are increasingly available to anyone who wants to put them together into storage systems (adding only their own software to differentiate themselves). Compare the number of storage start-ups with the number of automotive start-ups over the past couple of decades to see just how ludicrous his suggestion that automobile-industry-style consolidation is inevitable really is (though it still can’t quite rival the incompetence of his assertion that “the storage industry is shying away from the harsh reality that a sufficient amount of energy is, unfortunately, not available to keep the industry growing”).

   But this lack of understanding (and resulting poor analogy) is too close to my second conclusion above to be worth giving its own separate status.

   5. Given the garbage that he’s shoveling into his argument, the fact that his conclusion of the existence of a menacing ‘green trend’ is also garbage is hardly surprising. Rather than the result of energy scarcity, whatever ‘green trend’ is actually occurring is the result of a) existing sites coming up against their current power and cooling limits due to increases in power dissipated per square foot (rather than being squeezed by the actual cost of the power itself, at least as far as power specific to storage is concerned) and b) the fact that green is ‘in’ and therefore considered a salable advantage by marketing (the ‘initiatives’ that he describes have nothing to do with any kind of industry ‘dire state’ – they’re just good PR which, as we all know, often has very little to do with whether they’re actually effective).

   So our ‘star industry financial analyst’ completely missed the boat yet again.

   6. Then he suggests that storage vendors, instead of developing more cost-effective storage, should really be lobbying to “expand the capacity of the power grid”. This is abjectly incompetent on multiple levels: a) the capacity of the power grid is not a problem (utilities are able to provide all the electricity that’s requested, with infrequent exceptions), b) increasing generating and distribution *capacity* wouldn’t do anything to decrease per-KWH *costs* (in fact it would increase them unless all the new capacity were used, and perhaps even then), and c) technology companies that fail to keep up with technical advances (like, keeping their products as cost-effective as possible, including their use of energy if indeed that use is significant) and instead get diverted into areas far outside their core competencies tend to fall by the wayside.

   So apparently one can be a complete idiot and still be a ‘star industry financial analyst’ – amazing!

   7. Under the circumstances we should not be surprised that the author plays a bit fast and loose with the content of his citations. For example, where he states that “the authors cite that the three-year cost of powering a server exceeds the purchase cost of the server beginning next year” the cited material in fact indicates that it’s the three-year cost of powering the server *plus all associated power overhead (including cooling)* that would, according to their model (see comment just below) *almost* equal the server cost next year (and also doesn’t seem to have taken into account the significant decrease in server power consumption that the entry of Intel’s new ‘Core’ architecture and AMD’s expanded presence in the server market caused; furthermore, its assumption that in 2006 a 1U server dissipated 500W – *not* including associated overhead – and that this would increase by a factor of 1.2 every year is also a bit difficult to swallow). The authors of the cited article, to their credit, note later that the *actual* rate of increase in data center power consumption has fallen far short of that predicted by the model they previously described, but our ‘star industry financial analyst’ appears to have stopped skimming before reaching that observation.

   8. “Imagine buying a new car faced with the dilemma that the gas required over the first three years of ownership will exceed the cost of the vehicle.” I don’t have to imagine very hard, Steve: given today’s gas prices and the mileage that my wife rolls up, that could easily be true right now (and for a great deal of the rest of the world it’s been true for decades: we’ve just been spoiled by fuel prices far below the actual value of this irreplaceable and limited resource).

   But, yet again, your analogy is worthless, and yet again on multiple counts:

   a) while the combination of the up-front and fuel costs dominates the TCO of the car, the combination of the up-front and energy costs is only a small fraction of the TCO of storage (given that most studies suggest that the cost of managing storage is 5x – 10x the up-front purchase cost),

   b) as I just noted, the projection of three-year energy costs in the study that you cited is flawed, and

   c) in any event, that projection was for *servers*, and (as I quantified in my very first point above) it in no way reflects the cost of energy for *storage* (which is a far smaller fraction of the purchase price: there’s no danger of three-year or even five-year energy costs approaching the purchase price of storage for the foreseeable future).

   Wow – that was sloppy, Steve – or perhaps reminiscent of some slick politician attempting to apply one set of circumstances to an entirely different area, if it was calculated (but I’m willing to give you the benefit of the doubt that it wasn’t, given the quality of the rest of your presentation).

   9. “Moore’s Law will soon be downgraded to theorem status, because you can’t increase compute performance at the necessary pace for very long without an ample supply of electricity.” Tsk, tsk, Steve – you really shouldn’t babble about things that you don’t even begin to understand: a) in the first place, Moore’s law has nothing to do with power, and b) even in your misconceived understanding of it available power has nothing to do with the flattening-out of (single-thread) performance (rather, it’s because there’s no way to *dissipate* the power that it would take to continue to increase single-thread performance at historical rates, plus problems associated with leakage at the smaller feature sizes, plus, eventually, problems continuing to shrink the feature size itself).

   10. Given your near-complete misunderstanding of the entire situation, it’s not surprising that your next conclusion is a wet as all your others. There’s no new crisis to deal with, and storage is in pretty much the same place it’s been for decades: continuing to ride the increasing density curve in hardware but still very slow to innovate in software (i.e., there’s *lots* of room left for improvement there). Since energy use is at most a minor issue, the fact that there likely won’t be too many significant improvements in power efficiency just doesn’t matter (it’s nice that they’re happening, but save for in archiving situations – where MAID’s spun-down disks may be about as important in ensuring longevity as in conserving power – the advent of far more efficient 2.5″ disks may be the dominant event).

   11. Oh, dear: you then veer back to servers again (despite the putative subject of your article being storage: can it be that you just don’t understand the difference?). But since you’re on the subject, do you even have a clue how the move to massively multi-core chips can increase the efficiency of the kind of data center that can make use of them? The non-Moore’s-Law concept that you were bandying about earlier is tailor-made for them, and indeed *can* continue to pump out more and more compute cycles using less and less power (not, as I said, that this has much to do with the non-crisis in storage power consumption that you’ve been imagining).

   12. Just parenthetically, the Indianapolis 500 sometimes *has* been won by the driver who could make the most laps on a single tank of gas: your ignorance of automobile racing (and in particular some of the arcane rules that have been applied to it over the years) appears to approach your ignorance of storage and servers.

   Steve (to return to the third person here – it just seemed appropriate to speak directly to him at a few times above) concludes with some additional drivel not worth commenting upon (it’s repetitive, and I’m tired of picking it apart). It truly (as I said at the start) boggles the mind that people get paid for such tripe (and in his case one might suspect from his bio sketch paid well).

   – bill

4. 

   Jonathan Morgan on Wednesday, 30 July, 2008 at 2:29 am

   I very much agree with Ray: if tomorrow a 2TB HDD comes to market then it’ll use the same energy as a 1TB. This alone separates the storage industry market away from the “car market” analogy used.

   Secondly, the article is a bit convoluted. It argues that in the car industry supplier to oem ratio dropped 5:1, but also says “the storage industry has and always will be supplier dominated… 10 suppliers…” Erm. So unless the argument is that the number of suppliers will change to 2, there’ll be no big change there. On July 9th, Hitachi launched a 1TB disk, the Deskstar 7K1000.B, that gives “43% energy savings when disk is on standby”. I’m sure the other suppliers will follow suite. Good stuff. And, an energy efficient product that my storage company will OEM and pass on to its customers.

   Where I disagree with some commentators is about tape. In scenarios where tape results in an extra employee (or more) being required, or creates less work efficiency, then any “energy savings” are probably not really all they seem.

   By using storage virtualisation older disk technologies can be mixed in with new to create a single disk pool. This means that older technologies can be rolled-out only when they don’t make commercial sense anymore (the virtualisation, if clever enough, can take care of the data migration).

   Jonathan
   —
   object-matrix.com

5. 

   Steve Denegri on Wednesday, 30 July, 2008 at 5:54 am

   Gentlemen:

   All of you mention noteworthy points, but please allow me to defend myself!

   The article I wrote discusses the data center, not merely storage in isolation. Many of you are ignoring the simple truth that data center equipment, servers in particular, simply cannot continue consuming electricity at their current pace. There’s simply not enough capacity on the power grid, especially here in the U.S., to support this growth ad infinitum. Compare the U.S. to China, it’s no contest. China builds a new thousand-watt power plant every few days. We’re lucky to get a new one each year in this country. As a result, “infinitum” is quickly approaching (how’s that for an oxymoron?). Ask the typical data center manager, and they will tell you quite bluntly: energy use is quickly rising toward the top of buying criteria for new hardware.

   Most of you are examining storage in isolation. You can’t do that. In fact, you still see quite a few data centers that prefer to keep their storage within the server chassis. The EPA study shows that storage’s consumption of power is a very small fraction of what networking and server equipment use. I’m not arguing that storage needs to fix its energy problem, the industry decided to do that on its own. If anything, I’m arguing against that decision, for the mere fact that storage companies are using silly green marketing to peddle new products. It’s the servers that have the energy problem, and “green” approaches for computing are utterly ridiculous. You can’t stop the freight train that is exponential growth in electricity consumption by servers through the design of processors that use lower wattage for very long, simply because the pace of servers being deployed far surpasses those energy savings. In fact, not one of you mentioned the overall use of electricity by data centers. Go look it up, it now registers in the single digit percentages of total available power capacity here in the U.S. That’s how quickly data center electricity use is growing. It simply can’t continue at this pace, and no new energy-saving processors from Intel or AMD are going to halt the exponential growth in electricty use by data centers, trust me.

   “Green computing” is a ludicrous marketing concept, that’s the bottom line. No matter what the industry is, when its focus changes from taking advantage of its growth potential to re-engineering to operate within fixed constraints, that industry’s profit potential is on the decline.

   Also, to address a few other points some of you missed:
   1. The storage industry is OEM dominated. 70%+ of revenue generated in the industry is posted by fewer than ten vendors. Suppliers of components to these OEMs, hardware or software, total several dozen companies whose end customers are not end users but instead these fewer than ten OEMs. I wasn’t suggesting that the OEM number is going from fewer than ten to two, I was suggesting that the dozens of suppliers to these OEMs has the potential to fall by a factor of five, just as it did in the automotive industry.
   2. The Indy500 example went right over your head, I guess. Last I checked, there’s a checkered flag and a finish line in that race, meaning that there’s a finite stopping point toward which all the drivers strive to achieve at the fastest pace. If the race measured engine efficiency on a single tank of gas, then the car who wins would be identified as the last one still running after all the other cars had burned their gas. If the Indy500 ever had such rules, then it was long before my time and it was probably quite boring to watch.
   3. Barriers to entry in the storage market are, indeed, dropping from a technological standpoint, but not so from a sales and marketing standpoint. Reaching the end customer and earning their respect are costing more and more each day, in terms of dollars and market barriers, because end customers know that storage is the single most important computer component they own. I’ve watched this market through the Internet bubble as more than 300 startups set their sites on storage, and I’ve watched as most of them disappeared a few years later. Storage is a very, very difficult market for a startup to penetrate. There are those few who have seemingly crossed the chasm, but let’s see if they can stand the test of time. I’ve found that the real barrier is the $250 million revenue mark, very few can surpass that level without the realization that their potential has peaked.
   4. To say that the storage market is nothing like the automotive market is just plain wrong. Go count the number of disk drive vendors 20 years ago, then count how many still exist. Now go do the same for the number of automobile suppliers. If anything, the storage industry is worse than what the automotive industry experienced. Here’s another exercise. Count the number of RAID & JBOD vendors that existed 20 years ago versus today. You’ll see a similar pattern. Mr. Todd, I’m sorry, but you’re just dead wrong on that point.
   5. To suggest that there is ample supply of electricity is also just plain wrong. You should go talk to customers. Those who co-locate data center equipment will gladly tell you that they must pay penalties if their energy use exceeds very specific wattage constraints on a per-rack basis. Also, supply and demand, which is economics 101, will show you that the greater the supply of any item, the less it costs. So more supply of electricity will, in fact, lower its cost to the customer, no matter how surprising that concept is!
   6. I told Robin that I do not deserve any “star” accolades, and I wrote this article as I do all of my articles without requesting a dime. I write them because I enjoy it, and Robin has been a great friend and a good source of information for a number of years. I think all of you will agree that storagemojo is a very impressive site and speaks to Robin’s expertise quite well. If anyone deserves the “star” description, it’s him.

6. 

   Robert Pearson on Wednesday, 30 July, 2008 at 11:21 am

   Kudos to Steve, Robin, Carter and Ray for Vision and Thought Leadership in a time of need.
   Four “Points of Light” in the dark Storage.
   Storage is at a major inflection point.
   Internet Data Centers (IDC) have not solved many of the basic Storage problems except commodity pricing and economies of scale.
   This topic is “Right On!”.
   Try to imagine what the Information Stack (IS) would, or should, look like?

7. 

   Bill Todd on Wednesday, 30 July, 2008 at 12:24 pm

   Steve, your ‘defense’ is as pathetic as your original article:

   1. In direct contradiction to your contention that you were really just describing an overall data center problem with energy consumption, the article that you wrote very specifically singles out storage, and storage vendors, as having a problem with their energy use: “storage is not keeping pace with servers and networking equipment as it relates to the amount of energy each of these hardware products uses in the data center”, “storage is consuming an increasing portion of the data center’s power budget”, “No wonder the EPA study recommends that the storage industry dramatically improve upon its power management semantics for disk and tape systems”, “vendors [the associated immediately-preceding context is “in the storage industry”] would be better off recognizing that this heightened attention to energy efficiency is less indicative of a new growth opportunity and, more likely, portends an uncertain future for the industry, as a whole. Countless industries have reached an energy ceiling over the past half century”, “If this, indeed, is the direction that storage is headed, the coming decade will see a massive shake-out in the enterprise computing industry. The storage industry, in particular, is very vulnerable to this outcome”, “To get a glimpse of the future of storage, the automotive industry saw a major transition to energy-efficient products beginning in the late 1970s”, If your company is one of the many suppliers to OEMs in the storage industry, then you should recognize that this “green” trend, over the long term, bodes poorly for your company’s existence”, “the storage industry is shying away from the harsh reality that a sufficient amount of energy is, unfortunately, not available to keep the industry growing”, “Instead of elevating the rhetoric on the essential need to expand the capacity of the power grid, the storage industry is incomprehensibly embracing the energy efficiency paradigm”, “However, the storage industry is blind to its customers’ true needs. All vendors in the industry should take heed: what the customer really wants is lower utility costs”, “Imagine buying a new car faced with the dilemma that the gas required over the first three years of ownership will exceed the cost of the vehicle. Now consider how the storage industry would respond to the problem”, “the storage vendor might say, the customer is “stuck”: they have no choice but to frequently rip up and replace/upgrade storage equipment in order to contain utility costs”, “the storage industry will quickly hit a wall, because the vast majority of energy improvements are bound to come about in the first few generations of product”, “you would think that the largest consortiums representing the storage industry would have already formed working groups with end customers that speak to the need for more power plants”, “without more power plants, the typical storage industry consortium had best realize that its membership numbers will soon be on the decline”, “it’s mesmerizing to consider the possibility that the peak in the storage industry will come about not by technological limitations with regard to areal densities, not by the commoditization effects of TCP/IP networking, but rather by something as simple as the lack of available electricity to facilitate growth”. For Christ’s sake, save for a very few more general comments about data center energy use and for the general title that you pasted on it, *the entire article is about this mythical power problem that you claim storage, and storage specifically, has* – and your attempt to back-pedal now in the face of the very words that you yourself previously made public is ludicrous.

   So those of us who are criticizing this drivel are not ‘ignoring a simple truth’ at all: we’re simply responding to what you actually said. Of course there’s “not enough capacity on the power grid, especially here in the U.S., to support this growth ad infinitum”: that’s in fact a tautology true of *any* such situation (duh). But there’s plenty of power in and already planned for the grid to fuel *storage* growth for the foreseeable future, and there may in fact be enough to fuel processing and networking growth as well, given the increased efficiencies that we’re seeing (and which the study that you cited did not take into account, because they’re relatively recent developments).

   2. China builds power plants far faster than we do because a) their population is four times what ours is and b) it’s moving rapidly up the power consumption curve rather than already having achieved the relatively mature near-steady-state situation that ours has. Duh again.

   3. As I already explained to you, the reason that “energy use is quickly rising toward the top of buying criteria for new hardware” is not (*especially* in the case of storage) because of the cost or availability of power per se but because of the limitations of existing *power-consuming* facilities and the major step-function cost of building additional ones to house additional (or just hotter) equipment.

   4. “Most of you are examining storage in isolation. You can’t do that.” We sure as hell can when responding to your own very specific singling-out of storage as having a major power-consumption problem, Steve. Read the article that you actually wrote, not the one that you now wish you had written.

   5. “The EPA study shows that storage’s consumption of power is a very small fraction of what networking and server equipment use.” Indeed: it’s a pity that you didn’t consider this fact a bit more carefully before launching into your diatribe about the imminent perils for (very specifically) storage due to its power consumption.

   6. “I’m not arguing that storage needs to fix its energy problem” – but you were very emphatically asserting that it *has* a critical energy problem, and that’s the blatant falsehood that we were addressing.

   7. “It’s the servers that have the energy problem,” – possibly true: in any event, they’re certainly a more important factor in data center power use than storage is.

   8. “and “green” approaches for computing are utterly ridiculous.” Absolutely false: not only does energy efficiency (especially for servers: you *were* talking about computing generally in this instance, unlike most of the content of your article) help stave off the day when those new facilities have to be built (and the number which must be built if that day eventually arrives), but in a more general sense it reduces the overall energy demand which we aren’t (yet) able to satisfy using renewable resources (though perhaps you’re as ignorant about the consequences of continuing to burn fossil fuel as if – apt phrase, this – there were no tomorrow as you are about most other aspects of this discussion).

   9. “You can’t stop the freight train that is exponential growth in electricity consumption by servers through the design of processors that use lower wattage for very long, simply because the pace of servers being deployed far surpasses those energy savings.” The point is not to stop it forever, but to slow it down (both short-term and long-term) for the reasons I just explained. It’s also likely that at some point (in a manner analogous to the difference between the rate of increase in Chinese electricity demand and our own which I touched upon earlier) the situation will mature and the rate of increase in processing demand will level off; meanwhile, the more work we can get done for less energy, the better.

   10. “In fact, not one of you mentioned the overall use of electricity by data centers.” Still trying to change the subject to the focus of the article that you now wish you had written, I see.

   11. “No matter what the industry is, when its focus changes from taking advantage of its growth potential to re-engineering to operate within fixed constraints, that industry’s profit potential is on the decline.” Complete straw-man argument: there are no fixed energy-consumption constraints for the storage (or for that matter the overall computing) industry – there are simply variable costs associated with variable energy use, and minimizing those costs is one way to make one’s product more attractive.

   12. You really should stop digging your ‘Indy 500’ pit-of-ignorance deeper than it already was: you know as little about that subject as you do about storage, and will just similarly embarrass yourself further. The Indy 500, like virtually all racing save for the few really ‘unlimited’ classes, operates within very specific constraints rather than simply allows one to build the fastest car possible and drive it as fast as one can. One of those constraints is fuel cell (tank) capacity: the more efficient your car is, the fewer pit stops it must make to refuel (and that’s *important* to winning) – though recently the rules have tended to make it convenient to refuel each time tires are changed which reduces the importance of tank capacity somewhat. For a while (though perhaps no longer) drivers had a limited *total* amount of fuel that they were allowed to use as well, and if they ran too fast and used it all up before the finish line they just coasted off to the side and sat out the rest of the race.

   13. “Storage is a very, very difficult market for a startup to penetrate” – perhaps, but it’s nowhere nearly as difficult to penetrate as the automobile manufacturing market is (as I already observed by suggesting that you compare the number of start-ups in the two areas during the last couple of decades – and to discount the anomalous Internet ‘bubble’ let’s limit that to still-viable start-ups, but do include those which were so viable that they got bought out).

   14. Your comparison of the automotive and storage markets remains as superficial (and invalid) in other respects as well. The fact that disk drive manufacturers have consolidated simply reflects the commodity status of that component: their products are barely differentiable from each other (unlike automobiles, and unlike the components in them save for commodities like tires), and thus benefit from production economies of scale. And there were almost no RAID vendors 20 years ago, since the term had only just been coined (though vertically-integrated suppliers like IBM and DEC were producing a couple of RAID flavors well before that). Nowadays, the number of RAID variations (and vendors thereof) has far outstripped the nomenclature, though there are still a bunch of traditional vendors around too.

   15. “To suggest that there is ample supply of electricity is also just plain wrong.” Not if you’re talking specifically about powering storage (as you most definitely were): I can run an enterprise disk on less power than it takes to run the energy-efficient compact fluorescent light that I often leave on 24/7 in my cellar.

   16. “Those who co-locate data center equipment will gladly tell you that they must pay penalties if their energy use exceeds very specific wattage constraints on a per-rack basis.” You’re confusing energy-user plant costs with energy availability again, Steve: do try to get the difference straight before the next time you bring it up.

   17. “Also, supply and demand, which is economics 101, will show you that the greater the supply of any item, the less it costs.” Dear me: I would have assumed that an MBA had a better grounding than freshman economics, Steve. There are many reasons why prices may be inelastic: production cost (a biggie here – especially when you’ve got lots of *new* expansion plants to pay off, as you advocate), regulation, captive markets (which in a competitive market requires a degree of collusion to leverage, but regulated utilities do not necessarily resemble a competitive market), etc. Perhaps you’ll be able to understand it better by considering just how much electricity prices would have to drop to stimulate significant increases in demand (e.g., to become competitive with alternate sources of heat and vehicle fuel) and whether that would require selling it at a loss: when *demand* is relatively inelastic there’s no reason to increase supply and then compete to cause the price to drop (as farmers understand all too well).

   That about covers the ground, I think.

   – bill

8. 

   Joe Kraska on Thursday, 31 July, 2008 at 8:14 am

   Having just done an exhaustive storage market analysis with 5 year TCO calculations for a large multi-petabyte storage procurement we are working on, I’ll have to agree with Bill. The power costs of both power and cooling, as well as the capital costs of the power and cooling equipment, associated with storage are all only a small part of the TCO of storage.

   However: the numbers are a bit worse than Bill is letting on. As a reminder, tis the power cost of both the power and the cooling one needs to look at. Further, there is implied capital infrastructure: the stuff you have to buy to power and cool the equipment.

   While the former add to only a very small fraction of your storage cost, the physical infrastructure cost necessary for the power and the cooling of the storage you buy can be more than 10% of the cost of the storage.

   At those percentages, you’ll quickly find that you won’t buy storage based on reducing those numbers (you’ll buy storage based on reducing its pro forma CAPEX, and perhaps labor-related OPEX), it is still worthwhile looking at storage vendors that have good upfront properties and working on their power footprints.

   Joe Kraska
   San Diego CA
   USA

9. 

   Bill Todd on Thursday, 31 July, 2008 at 2:14 pm

   Thanks for the confirmation, Joe. I did note that power supply inefficiencies plus cooling requirements could more than double the raw energy cost of running storage in my original post, so didn’t think it necessary to reiterate that later.

   Ancillary costs (square footage, UPS, and so on) are also relevant, and if they’re as small as 10% of the raw up-front storage cost I’d be pleasantly surprised (unless you’re using EMC boxes as your cost baseline, in which case never mind). More to the point, though, is that those costs aren’t proportional to power cost/availability (the boogeyman that Steve was singling out): the up-front costs of cooling infrastructure facilities and UPS (at least that typically small portion that can be chalked up to storage) are proportional to storage power consumption and hence benefit from storage power efficiency but wouldn’t benefit at all from the additional power plants and resulting hypothetical power price drops that Steve points to as the solution for his mythical problem, and the square footage costs are proportional to storage density (and thus similarly wouldn’t be affected by the ‘solution’ that Steve proposes).

   – bill

10. 

    Robert Pearson on Friday, 1 August, 2008 at 12:07 am

    My interest is in “A useful metric is how much processing gets done per watt.”
    To clarify, this comment was made in the direct context of “There is another factor to be aware of. If the CPU spends 25 million clock cycles waiting for random HDD data, but only part of that waiting for SSD data, the actual increase in notebook power consumption may be in the CPU. A useful metric is how much processing gets done per watt. If you are willing to scale back performance to that of an HDD-based system, an SSD-based system should deliver significantly longer battery life.”

    While it is still true that CAPEX and OPEX are just as important as the toilet in a space craft, the difference in cost between the $25 million USA version and the Russian plastic “baggies’ is due to different desires and objectives. Who or Whom are you trying to please?
    If you do the correlation between CAPEX, OPEX, TCO and ROI versus Management salary increases, bonuses and perks you will not like what you see.

    There are processes to please. Carter George mentioned a few in his comment above…
    “Second, Green 1.0, or buying new storage, is not the only answer. Green 2.0 ? which is what Ocarina does ? amounts to ?store ten times more data on disks you already have?. By using a combination of online storage optimization techniques (subfile deduplication, next-generation compression, and others), Ocarina is able to get up to 10:1 data reduction results for online storage (not for repetitive backups, but for online data sets).”

    Ray Lucchesi added a few more…
    “Power curves notwithstanding, storage power use per GB or per IOPS is on an overall decline.
    There are a plethera of ways to address the utilization issue, Thin provisioning, deduplication, and Sun/IMM being just some examples. If power costs continue to skyrocket I predict utilization rates will start to creep up to the wall (70-90% utilized) from where it is today in open systems (20-40% utilized) using all the known tools and then some.”

    These power hungry processes will sweep through an IT infrastructure like a tsunami.
    The most interesting processes are just starting to form their wave.

11. 

    DaveG on Friday, 1 August, 2008 at 10:26 am

    After years of bearing witness to inefficient technologies (Hybrid cars, curly-que Gore-bulbs, etc) gaining market share due to clever “green” marketing campaigns, I am pleased to be able to recommend “green” storage technologies to my customers that actually make a difference. Exponential, Moore’s-law-esque (not directly applicable, I know, Bill) increases in per-spindle density have clearly had the biggest impact. But MAID, block-level deduplication, and more efficient storage management techniques leveraging CAS, clustered filesystems, and effective tiering are essential to designing scalable data centers in the face of exploding digital data production.

    If promoting these technologies and techniques as “green” is effective PR, it does not detract from their significance as good engineering practices for tomorrow’s data center. To dismiss them as “silly” conveys an ignorance of the real impacts of storage costs for many operations today… and most tomorrow.

    – DG
    Reston, VA

12. 

    Joe Kraska on Friday, 1 August, 2008 at 5:13 pm

    What “green” storage technology would you recommend to your customers that makes a difference?

    Not that the recommendation matters much. 8 out of 10 vendors these days have something “green,” either now or on their roadmaps. To include, disk spin down, dedup, ultra-density, ILM tiering, etc.

    Anyway, it’s not silly. It is, however, holistic. There are no existent vendors that have an offering that distinguishes itself particularly on any one attribute, but when taken as a whole, the attributes can add up.

    Barring all that, the up front price you pay for storage, today, is still without question the most essential factor in establishing a low overall 5 year TCO. Arguably, your storage labor hours will play as well, depending on complexity.

    Joe Kraska
    San Diego
    CA

13. 

    Bill Todd on Friday, 1 August, 2008 at 10:35 pm

    Robert –

    I’m afraid that equating OPEX with the toilet in a spacecraft completely breaks down given that when you include storage management costs the storage OPEX costs (unlike the toilet cost, even for the best non-baggie versions) typically dominate overall storage TCO. And while Carter’s observations (while hardly unbiased) have merit, they aren’t a panacea for anything like *all* environments (many of which just don’t contain all that much duplicate data, either by their nature or by careful attention to avoiding redundancy at higher levels: deduping via strategic use of symbolic or hard links is a decades-old practice, for example). Carter did not, after all, claim across-the-board 10:1 data reduction but only the ever-popular “UP TO 10:1″ metric, which may easily be satisfied by Exchange customers who replicate huge email attachments promiscuously or by virtual servers that replicate large portions of their state but is unlikely to be true for already-compressed (and unduplicated) media files, for example.

    I’m not sure that tape is more attractive than disk from a power standpoint, since you can spin down a disk to zero power consumption and still access the data on it far faster than you can access data on tape (and it wouldn’t surprise me if it took less energy than loading and then spinning up a tape as well).

    I’m not sure that Ray’s prediction about the likely consequences of ‘skyrocketing’ power costs will have any real impact until they actually reach a substantial percentage of overall storage TCO, which shows no signs of happening any time soon.

    Dave –

    While I’m not 100% sold on hybrid cars yet (since my bog-standard Neon with nearly 100,000 miles on it and 11-year-old conventional technology has averaged over 40 mpg since it was new – admittedly it sees little city driving, where a hybrid has real potential to shine, but a lot of highway driving at 70 – 75 mph), the efficiency of compact fluorescents compared with conventional incandescent bulbs is completely real. Even the over-priced versions can easily pay for themselves several times over in energy savings over their lifetime, and since they’re now available in the $1 – $2 price range they’re a real no-brainer (and a very legitimately ‘green’ product rather than a mirage perpetrated by marketeers, the minute amount of mercury which they contain notwithstanding since it’s far less than the mercury that would have been released in a power plant providing the additional energy required by a less efficient but mercury-free bulb).

    In case it wasn’t obvious from my response to Steve’s assertion that ‘“green” approaches for computing are utterly ridiculous’ (see point #8 in my last post), I”m heartily in favor of energy-efficient storage products even though I don’t believe that storage per se is particularly threatened by a shortage of available energy (though I don’t discount the possibility that our infrastructure is so threatened across the board). My earlier comment about ‘green’ initiatives being ‘good PR’ whether they were actually effective or not referred more to Steve’s example (irrelevant though it was to his argument) of generating energy from employee garbage, not to significant consumption savings in products.

    – bill

14. 

    Steve Denegri on Saturday, 2 August, 2008 at 8:40 am

    A few items:

    Mr. Todd, your four posts to this thread are hereby deemed sufficient. If for nothing else, in the efforts to minimize superfluous storage usage in the vein of this energy-centric discussion, let’s discontinue posting comments merely for the sake of redundancy or to self-designate oneself the resident expert, particularly when a question isn’t being asked of you. Thank you for your understanding and cooperation, in advance.

    Mr. Kraska, I’d really be interested in learning from you the degree to which power and cooling costs exceeded the raw electricity cost for the recent TCO analysis you completed, if available. I was made aware of a recent installation whereby the power and cooling costs for a data center were close to 3x the raw electricity cost. I’d be curious if you believe this was an outlying case or if it is becoming more the norm, particularly since many utilities have jacked up the cost of electricity even further in recent months.

    Finally, Mr. DaveG, it sounds as though energy costs are, indeed, near the top of the list of buying criteria in the customer base you serve. Although it’s clear from Mr. Kraska’s experience that energy costs are still manageable, I’d like to hear any examples you might have where customer are beginning to push back on energy issues as it relates to the products you represent.

    –Steve

15. 

    Joe Kraska on Saturday, 2 August, 2008 at 8:04 pm

    Steve:

    Power to the cooling is typically 50% or less of the power to the equipment being cooled (generally, I wag 30%). The ratio is dependent upon when the equipment was bought (efficiency has gone up over the years), and how old it is now (old equipment loses efficiency) and also depends on a great deal else, such as the outside environment. Increasingly, designers are opening their loops to heat exchangers when the outside environment justifies. That can be all night in some places, and half the year day and night in others. In those settings, the power to cool the data center can be remarkably low.

    As far as good use of power goes, I think a lot of the benefits of consolidation are more to do with duplicate computing and storage, along with newer, more efficient systems, and economies of scale than the pure kWhr play on the base stuff. I.e., the power concerns are there, just not entirely after the fashion that many people might be thinking.

    Anyway, how about some specifics? Looking at my analyzer right now, I see that the five year cost of power+cooling for storage runs $0.25/GB for Tier 2 (SATA) storage.

    When you realize that Enterprise class Tier 2 storage often runs you ~$2.0/GB in bulk (petabytes at a time), the cost of power and cooling is not inconsequential; however do note that this is 5 years of power and cooling.

    You should see by looking at the numbers that there is incentive to use less power. There’s just not incentive to pay *more* for storage so that you are using less power. Or at least not more than $0.25/GB worth, if you get my drift.

    You will find, if you dig, that most of the major storage vendors have lower power plays forthcoming soon. Stay tuned.

    If enterprise storage were 4X denser for the same price and power, the 5 Year TCO for powering and cooling it would be 50% of the cost of buying it. A sign of things to come, perhaps?

    Joe Kraska
    San Diego CA
    USA

16. 

    Bill Todd on Sunday, 3 August, 2008 at 7:55 am

    Er, shouldn’t that last statement be “If enterprise storage were 4x denser for the same price and *per GB* power, the 5 year TCO for powering and cooling it would be 50% of the cost of buying it”? And wouldn’t that be pretty unrealistic, given that power per GB and cost per GB tend to track each other at least moderately closely (i.e., their relative contributions to TCO tend to be far more stable as storage density varies than your original projection would suggest)?

    – bill

17. 

    Joe Kraska on Sunday, 3 August, 2008 at 4:12 pm

    Bill, no, I think I said it correctly. To put it another way, if you can suddenly buy 1 PB for 1/2 the price you did before, but it’s the same 35 kW it was before, the relative $/GB of procuring the system will be proportionately less when compared to the power.

    Power per GB tracking $/GB? Do you mean it tracks # platters? (Which it does) Or RPM (which it does?). But you mustn’t think it tracks areal density (I don’t think it does, particularly). Perhaps you can help me out here.

    Joe Kraska
    San Diego CA
    USA

18. 

    Joe Kraska on Sunday, 3 August, 2008 at 5:27 pm

    For fun, I went back in time and grabbed the spec of an old 40GB IDE drive. I see that it operates a 2W under load. Looking at a modern, efficient 1TB drive, I see it operating 7.5W under load.

    It would appear that the truth is in the middle. With this stellar sample size of 1 :-), let’s calculate the relative efficiency in w/GB over the years:

    40GB/2W = 20GB/W
    1000GB/7.5W = 133GB/W
    133/20 = 6.66GB/w improvement

    So, in one fell swoop we have proved two different things:

    1. Storage power calculations are evil (6.66, hint).
    2. And the number of GB/watt under load has improved an evil amount.

    The truth appears to be in the middle. Storage density has improved 25 times, while efficiency only at a rate of 6.66.

    If you think that my sample size of 1 convinces 🙂 then density is improving approximately 3.75 times faster than efficiency.

    Joe.

19. 

    Bill Todd on Sunday, 3 August, 2008 at 11:16 pm

    While I’m a bit surprised that the 40 GB drive which you cite used as little power as it did (unless it was a single-platter drive, in which case comparing it with a contemporary single-platter drive would have been more appropriate; it would also be appropriate to ensure that the drive speeds were similar) in fact the statement that I made said nothing whatsoever about how density increases related to power requirements, only about how per-GB power requirements related to per-GB purchase cost.

    So the real question (assuming that the 40 GB drive was a 7200 rpm drive and used a comparable number of platters to today’s 1 TB drive) is how much the cost/GB fell between the two: if it fell by a lot more than the factor of 6.66 that the power/GB fell then that would tend to give some support to your original suggestion.

    I happen to have a product manual for a two-platter ATA 200 GB Barracuda 7200.7 drive that I bought about 4 years ago. The retail version had a street price of $130 at that time. Seagate’s current four-platter 1 TB SATA Barracuda 7200.11 has a street price of $180 (OEM version, but I won’t quibble about that), indicating that $/GB has dropped from about $0.65 to about $0.18.

    Interestingly, despite having twice as many platters (I couldn’t find detailed power figures for a current 2-platter Barracuda in their product manual) the new drive consumes a tad *less* power than the old one (11.16W vs. 12 W for a fairly heavy mixed workload that appears to have been the same for both). Hence despite the extra two platters power/GB dropped from about 0.06W to under 0.012W – noticeably *more* than the cost/GB dropped and thus tending to refute your original thesis above.

    (Though as I noted it’s not relevant to the specific question we’re discussing, it’s interesting to note that both purchase cost/GB and power/GB dropped *faster* than the apparent areal density increased.)

    – bill

20. 

    Bill Todd on Monday, 4 August, 2008 at 2:07 am

    While I don’t recall what the street prices were, I did turn up a couple more Seagate product manuals (I like their drives, though both the competence and the attitude of their customer service sucked the one time I had the misfortune to encounter them). The 4-platter Barracuda 7200.10 used 16.5% more power than the 7200.11 (13 W vs. 11.16W) in the same test, suggesting that Seagate has been reducing power requirements with each generation (which makes the previous comparison where an older 2-platter drive required more power than a current 4-platter drive more understandable). A single-platter Barracuda 5400.1 consumed only 6.6W in the same test, but if the drag-related portion of power dominates the total and rises approximately with the square of the angular velocity then that’s still considerably more power per platter at normalized speed than my 7200.7 consumed (again suggesting that Seagate has been steadily reducing relative power requirements for quite a few years now).

    All of which makes at least the portion of your thesis where you postulate that the power required by 1 PB of storage may remain constant as density increases pretty difficult to justify, I think: not only is power per platter within the same product family decreasing but capacity per platter is increasing, both of which operate to drive down the power per GB. Since purchase cost/GB has also been decreasing consistently this was the reason for my initial comment that they tended to track each other – even if not precisely then at least in a manner that made your assumption (4x density increase at the same power/GB) quite unrealistic.

    – bill

21. 

    Joe Kraska on Monday, 4 August, 2008 at 7:15 am

    Bill:

    At a certain point you will be able to count on spinning media using more or less constant power at idle (discounting power-stepped drives, but let’s leave those out of the discussion for now), merely because the cost of keeping the drive spinning at 7200RPM reduces to simple power physics. I really can’t say that we’re there yet, because I don’t know how much margin the manufacturers have between practical engineering and physics. But it’s there…

    Anyway, yah. This stuff can be a bit complicated. This is why I have a TCO analyzer. First instincts aren’t always correct, and proper valuation of the cost aspects of your storage enterprise require some significant work. Having done it, I recommend that anyone working with plural-PB procurements do so. Tis important.

    Joe

22. 

    Steve Denegri on Monday, 4 August, 2008 at 7:10 pm

    Mr. Kraska, thanks for your feedback! By chance, do you have the power & cooling data in terms of cents/kwh? The example I mentioned is paying roughly 20.5 cents/kwh for power + cooling vs. 7 cents/kwh for raw electricity. If you don’t have the data, no biggie.

    By the way, I’m well aware of the forthcoming energy-centric storage products you hinted at (in fact, this is really what made for the impetus for my article)! For example: prepare to be “flashed” at your neighborhood data center.

    –Steve

23. 

    Joe Kraska on Tuesday, 5 August, 2008 at 7:55 pm

    Steve:

    You’re really confusing me. I’ve never heard of any power company charging different rates for the “type” of kWhr used, as they don’t really have a means of telling how it’s used. Why would the electric company care?

    Out here in Southern California, that’s just not done. What region of the world are you in?

    Joe.

24. 

    Bill Todd on Wednesday, 6 August, 2008 at 12:52 am

    While that’s what Steve said, I don’t think it’s what he meant (which was probably just a – ‘redundant’ – restatement of his earlier claim that cooling power when added in close to tripled the raw system power requirements; parenthetically, though, at least for residences electric utilities often *have* charged different rates based on use, e.g., for space heating, by metering those circuits separately – they also have charged different rates for water heating based on time-of-day constraints, again using a separate meter, but such timed use isn’t as applicable to the current discussion).

    This response is also something of a test, since several of my recent posts have been ‘awaiting moderation’ for quite a while now, and Robin hasn’t been responding to emails asking why.

    – bill

25. 

    Steve Denegri on Wednesday, 6 August, 2008 at 6:42 am

    Mr. Kraska, they definitely do care, based upon usage. For example, 7 cents/kwh would be a “base” rate, but the instance I mentioned, 20 cents/kwh, would imply that the usage was far above this base threshold established by the power company. The added resources necessary for power + cooling, many times, throw the usage figure way north of the maximum usage that would correlate with the base rate, meaning that the charges are higher. That’s why I was curious the usage, in your case, in terms of cents/kwh versus a base rate.

26. 

    Joe Kraska on Wednesday, 6 August, 2008 at 7:45 am

    Steve:

    Usage is one thing, categorizing it is another. I’m sure this will matter entirely based on region. A major data center we support pays $.08/kWhr, no matter how much they use, and since the entire data center is for the powering and cooling of IT equipment, that’s that.

    You understand, though, that unless your power is metered separately, there’s no way to know what”s what, except to say you exceeded your budget and are paying penalty fees.

    Joe.

    p.s., as an aside, $0.20/kWh would start to make the five year TCO for powering storage, at least in nearline, a fairly notable line item (particularly if you are like us, where you’re nearline storage is costing more towards $1 than not).

27. 

    Steve Denegri on Wednesday, 6 August, 2008 at 8:20 am

    Mr. Kraska, that 20 cent figure is the real deal, although I can’t say for sure if it’s for a nearline versus other application site (I’d assume it’s not nearline). I’m glad to hear you, who is clearly an expert in this area, agree that this cost makes utilities a meaningful consideration in the storage TCO equation.

    Usage certainly depends upon region, you’re right. I guess it all depends upon the overall supply of the energy in the region being considered. Those utilities who increase the charges for higher degrees of use, you have to assume, are in regions of tigher energy supply. Which goes back to my original thesis: we need more energy supply, over the long run, to keep usage costs down. These higher usage pricing thresholds are a killer. Redesign of compute infrastructure to accommodate lower supplies of energy is a short-term fix. At the rate of growth of data center computing, these “green” fixes will be quickly offset. This is the crux of my argument.

28. 

    Joe Kraska on Wednesday, 6 August, 2008 at 10:37 am

    Steve:

    If you’re argument is that the “green” fixes are mere low hanging fruit patches, that are nothing more than stopgaps, I generally agree. We’ll see a big boost from MAID-technology variants (for those who have relatively inactive nearline use cases), another boost from SSD’s (assuming that they can get past the unrecoverable read incident rate problem), and then for the most part we’ll be back to sucking the local municipal grids dry.

    The real problem is supply and base energy costs.

    Joe.

29. 

    Bill Todd on Wednesday, 6 August, 2008 at 6:36 pm

    Hmmm, Joe. Given Steve’s less-than-impressive record of accuracy so far I wouldn’t be so quick to accept his claim that the site he cites is actually paying over $0.20/KWH for electricity (as compared with the $0.07/KWH that he says they’d be paying if they only needed to purchase electricity to run their equipment rather than also to cool it): I still suspect that he’s just confused between this and the possibility that they’re paying over $0.20 for the *combination* of 1 KWH of storage (and/or server) use *plus the associated likely 1+ (p9ssibly even close to 2) KWH of required cooling* for that hour’s use of the equipment (though, while contrary to your own experience, it’s possible that they might also be paying a premium in addition to what would otherwise have been the base-rate cooling power cost for the increased total use: it would be nice to see the primary material that Steve is paraphrasing so we could analyze it ourselves).

    I also suspect that he’s talking about a data center that comes nowhere near your own experience that “Power to the cooling is typically 50% or less of the power to the equipment being cooled (generally, I wag 30%)” – and hence that the facility he’s describing could be doing a *lot* to decrease their electric load required for cooling. If so, the fact that they are not taking such steps reflects the relative unimportance of even such inflated energy costs to them (contrary, as usual, to Steve’s basic premise that electricity is too expensive and scarce for storage growth to continue).

    So when we toss in your own additional experience that a) “A major data center we support pays $.08/kWhr, no matter how much they use” and b) “power costs of both power and cooling, as well as the capital costs of the power and cooling equipment, associated with storage are all only a small part of the TCO of storage”, I find it somewhat difficult to reconcile your stated position with Steve’s assessment of it (“I’m glad to hear you, who is clearly an expert in this area, agree that this cost makes utilities a meaningful consideration in the storage TCO equation”): once again he appears to be confusing your *conditional* agreement that *if* electric rates actually *were* $0.20/KWH *then* the cost of powering both the storage and the required cooling equipment would become a “fairly notable line item” with agreement that it already *is* a fairly notable line item.

    Then again, your last statement (“The real problem is supply and base energy costs”) makes me wonder: are you agreeing that some ‘real problem’ (as contrasted with a relatively minor line item at worst, as you seem to have been saying up until now) with storage energy use actually exists, or just the more mundane observation that conservation measures by definition have limits after which the only way to further contain costs (ignoring the question of whether they are of such a magnitude that such containment is important) is to reduce prices?

    (Given the moderation constipation which my own posts encountered there a couple more things I’d like to discuss, but I’ll make them separate posts to try to keep the points separated.)

    – bill

30. 

    Bill Todd on Wednesday, 6 August, 2008 at 7:32 pm

    Joe –

    It looks as if you wrote your last response to me before Robin got around to posting the addendum which I added to the post to which you were responding, describing the fairly consistent decrease in *per-platter* power requirements over time within a single Seagate drive family – a progression which helps drive down power/GB even faster than the simple increases in areal density do (and thus helps these reductions equal or even exceed the $/GB price reductions – the point of my original response to you, though I’d still like to know more details about the isolated example to the contrary which you came up with).

    So while I agree that first instincts can be misleading (in this or almost any area), I’m not sure I agree that the situation is particularly complex: while it’s certainly necessary to go through very specific calculations to generate a funding proposal, merely characterizing the general trends (and how relatively important they are to costs) is pretty straight-forward (though still a matter of elementary-school arithmetic rather than mere ‘instinct’).

    In that vein, I’m interested in the specifics of how your TCO analysis arrived at the conclusion that the 5-year power & cooling costs of Tier-2 enterprise SATA storage were $0.25/GB (or $250/TB), since at the $0.08/KWH pricing that you referred to earlier the 5-year power cost (not including cooling, which you earlier WAGged at an additional 30%) of a contemporary 1 TB Seagate Tier-2 enterprise SATA drive is only $40 (or $0.04/GB) at the highest rate of power consumption that they tested for. Even their previous 750 GB model would have required less than $47 to power it for 5 years (or about $0.062/GB – still less than 1/3 of the number that your analysis came up with even after adding another 30% for cooling load).

    Are you by any chance quoting costs per *net* GB (e.g., that would require 2 GB of raw storage in a mirrored system)? Do controllers and other SAN hardware add significantly to the power use that you’re including? Are any costs that are not directly proportional to power costs included (e.g., costs of power or cooling equipment)?

    Please understand that I’m not asking for any information that might be considered proprietary: since you felt free to volunteer a number, I’d just like some general idea of what it actually reflects.

    Thanks,

    – bill

31. 

    Joe Kraska on Thursday, 7 August, 2008 at 7:43 am

    Bill:

    The costs are per net usable GB, yes (although we do not mirror). The power TCO costs are for all power, including the controllers, etc… and less we forget: JBOD systems and their power supplies do not necessarily pull at the stated idle/average times of the drives within them. And of course are 5 years of all those things.

    To be clear, I do not “see” drives or their power requirements. I see multipetabyte proposed storage solutions, with min/avg/peak power ratings for full trays of storage and the control infrastructure and the like, at the system level.

    Those are where those numbers come from.

    You will find yourself not dismissing power costs in a full TCO calculation, but also finding that the most important things to concentrate on are $/GB CAPEX and labor. Things like power, space utilization, and disk replacement charges will fall into a more holistic assessment.

    BTW, I’m seeing 25-30kW/PB typically from a survey of 14 nearline offerings (net usable, not mirrored: RAID-6 or equiv, and after file system subtractions). It would be worse in Tier-1 of course.

    BTW, looking over my figures, $0.24/GB is actually the hottest vendor. Some fare better; $0.20 is pretty typical. Some of the cooler vendors, $0.14. COPAN $0.04.

    Joe

32. 

    Steve Denegri on Thursday, 7 August, 2008 at 12:34 pm

    Mr. Todd, Mr. Kraska had the same question about the rate I mentioned, but 20+ cents is the per-kWh rate for this specific instance.

    There are multiple components to a utility charge, but generally speaking the costs of the energy that the utility supplier must overcome in order to provide electricity service profitably come in two flavors: a fixed capital cost and the variable costs associated with supplying the energy. The former refers to the capital cost of constructing the power facilities, which are amortized by the utility supplier over a number of years (the exact # depends upon the terms of the loans used to construct power facilities). In order to offset the amortized debt cost, the utility company assesses its largest customers a “demand charge” that is established based upon peak usage that’s often determined over a rolling multiple-month average. Penalties are paid if a customer exceeds this peak usage, often times at 2x the rate that is charged at usage below peak. Bottom line is that it is crucial that electricity usage stays below the peak amount that a company has indicated its need would be.

    The second component to the utility charge, of course, is variable based upon usage. The costs to fuel, maintain, and operate the power facility are often passed-through to customers who use a small amount of electricity. In many cases, however, higher usage corresponds to higher per-kWh energy charges. In fact, the per-kWh rate charged to commercial busineses often stairsteps, depending upon the amount of electricity consumed. Therefore, a data center that has to pay for the added costs of cooling can easily find itself in a higher usage bracket, as was the case with the example I mentioned. Also, I should mention that many utilities charge higher rates in the summer vs. the winter, indicative of the higher costs associated with cooling infrastructure at the data center and the added burden placed on the power supply at the local utility.

    Anyway, all this to say that it’s not always a “one charge fits all” in the case of electricity. If anything, power plant economics center around the ideas that a) unexpected spikes in demand are costly and must be adequately offset through necessary charges, and b) the more power that’s consumed, often times, the more that must be charged on a per-unit basis, particularly when electricity supply is constrained.

    By the way, as part of the feedback from this article, I was made aware that Microsoft charges its customers that subscribe to its data center services not on a square-footage basis as many co-lo’s do but rather on a per-kWh basis. Mr. Todd, I thought you’d find this interesting. Go do a search and you’ll find it, I believe it was mentioned by Microsoft about a month ago. I would guess more will change their pricing metrics to Microsoft’s methodology in the months ahead.

    If you would, gentlemen, please don’t parse each and every word, I am not able to visit this site all that often so let’s cover this in more general terms. We can argue this for weeks, which is undesirable. I just want to get the crux of these main points across. Thanks.

    –Steve

33. 

    Bill Todd on Thursday, 7 August, 2008 at 4:27 pm

    Thanks for the additional information, Joe. For RAID group sizes of 6 – 10 RAID-6 increases raw required storage by 50% – 25% (and of course proportional cooling costs as well), which would bring the power required for raw storage plus cooling up to as much as $0.12/GB if you were using previous-generation Seagate drives even without adding in whatever your controllers may be using (or adjusting for somewhat less efficient cooling than your 30% WAG would suggest). File system overhead should typically be pretty negligible unless your environment consists of zillions of small files. The power figures from Seagate that I used weren’t for idle/average power but for continual-use power (in fact, the highest of the figures that they reported in most cases), and I calculated them over the full 5-year nominal service life, so there should be little difference there unless the JBOD supply is for some reason extremely inefficient.

    My main concern was whether somehow the raw disks were consuming far more power than Seagate claims they do, because $0.25/GB for just the raw-disk power over a 5-year life *would* be significant to people who elect to use software solutions for managing their redundancy (hence don’t have the added expense associated with high-end hardware controllers to drive aggregate power costs back down into the relatively unimportant category – though management costs could still eclipse up-front and power costs in many cases).

    The COPAN number is impressive, though I suspect assumes something much closer to backup/archival than to primary-storage use (allowing them to keep most disks spun down most of the time). With primary disk storage, there’ll always be a trade-off between spreading out the hot data across as many spindles as possible for performance and being able to spin down idle disks effectively, I’m afraid.

    Incidentally, I just checked our local utility’s power rates, including transmission and distribution charges of various kinds: they are (as you found with yours) flat for large businesses (over 1,000 KW demand) regardless of amount of use (save for cases in which the business’s demands are peaky rather than stable, where they wind up paying a bit extra to account for the additional *potential* capacity that typically goes unused in whole or in part). Small businesses (under 100 KW demand) pay less per KWH the more they use, again with a penalty for peakiness. Intermediate sized businesses pay rates between those for large and small busineses, and less per KWH used the more they use (so basically all businesses pay less per KWH the more they use up to the large business category where the rate flattens out at the lowest figure).

    Interestingly, the base energy price is the same for all users and regardless of demand: it’s the distribution and transmission prices that vary. Residential and large business users also can get price breaks for (metered) off-peak use.

    Finally, there’s that matter of moderation constipation again. While Robin managed to unstick his process for some of my posts, one remains hung (and another recent one that asks why as well: since he was unresponsive to email and suggests on his “Contact & About” page that comments are another means of communicating with him, I tried that). The posts in question are well within the ‘moderation’ bounds that he describes on the “Contact & About” page, and I’m being forced to the conclusion that he’s just censoring me in a surprisingly (to me, not knowing him save for my contact here) arbitrary and petty manner.

    So this constitutes another probe to see whether whatever criteria he’s using may relate to a post’s content in its entirety. If you’re curious what the fuss is about, I’d be happy to email the posts to you so that you can form your own conclusions (as I’ll do if this post also gets stuck in the queue).

    – bill

34. 

    Robin Harris on Thursday, 7 August, 2008 at 5:10 pm

    Bill, I’m tired of your gratuitous harshness. That’s why I’m rethinking my standards for comments.

    I want StorageMojo comments to be a place were people who know less than you feel comfortable asking questions and venturing opinions. Right now, I’m pretty sure that isn’t the case.

    You know how to express yourself politely. Please do so from now on.

    Sincerely,

    Robin

35. 

    Joe Kraska on Thursday, 7 August, 2008 at 8:40 pm

    Bill,

    As a neutral third party, I can see the basis of Robin’s opinion in his remark to you. I’ve been on the net since 87 or so in one form or another. Robin is well-advised to keep even hints of unnecessary rancor from his communities. It’s never fun when things turn personal, I think.

    Anyway, this is a bit of an industry joke for you:

    “For file system overhead being generally low, I see that you unaquainted with NetApp”.

    *ba-da-bing*

    On to the next subject, and for COPAN, you are correct. You need a VTL like use case. “Store a great deal, access seldom”. The interface need not be a VTL interface, but the use case ought to be.

    We use “per usable GB” measures of our merits in our TCO analyses for a very good reason. Various vendors with storage appliances (NAS based ones, notably) have variable and often high file system overhead. There are also limits in aggregate sizes that make RAID overhead high with some vendors and not others.

    Joe.

36. 

    Tom on Thursday, 30 October, 2008 at 1:21 pm

    A continuation on Steve’s theme , focused on cloud computing can be found at:

    http://www.virtual-strategy.com/Features/Abstract-Microsoft-and-Google-Cloud-Computing-Dominance-Through-Renewable-Energy.html

This site uses Akismet to reduce spam. Learn how your comment data is processed.