Hospital ship Haven in Nagasaki, Japan, 1945

by Robin Harris on Monday, 25 May, 2015

StorageMojo is republishing this post to mark this Memorial Day, 2015. In a few months we will be marking the 70th anniversary of the end of World War two as well. My father was a career Navy officer and this is a small part of his legacy. See the original post for the comments, many from the children of my father’s shipmates.

While most of what modern storage systems protect are business records there is also the use of storage for saving our cultural heritage – of which this is a small part.

My father, Tom, was an officer in the US Navy Medical Corps during World War II. As a newly commissioned 2nd lieutenant he was aboard a submarine tender anchored at Pearl Harbor on December 7, 1941. As a doctor he spent the next 36 hours in an operating room working on the wounded.

Less than 4 years later he was aboard one of the first US ships to enter Nagasaki’s harbor after the Japanese surrender. In a brief memoir he describes a visit to Okinawa on the way to Tokyo – where he was aboard the USS Missouri when the formal surrender was signed – and then on to Nagasaki, the 2nd city to suffer an atomic bomb attack.

USS Haven, a USN hospital ship, in 1954

USS Haven, a USN hospital ship, in 1954

The primary mission of the Haven was the collection of Allied POW’s in need of medical care from the many camps in the area.

The trains began arriving every three or four hours each one with several hundred men. Each new arrival was a thrill with the band playing “Hail, Hail the Gang’s All Here” and the sailors and marines on the platform cheering. It was an experience to see the somewhat bewildered expressions of the men on the trains change to tears, smiles and laughter as they realized that they had reached the end of the road – that the day, the longing for which had sustained them through months and years of torture and mistreatment, was at hand.

While in Nagasaki he visited a Japanese hospital:

What we saw in that hospital was something I wouldn’t have missed seeing for anything but something I never want to see again.

Everywhere you looked there were desperately sick people, mostly women and children. Many were horribly burned and over and around all of them were flies by the millions. There were no beds – all patients were lying on straw mats on the floor. In the corridors of the hospital, the patient’s kin had set up their charcoal burners and were preparing a meal thus filling the hospital with smoke. One sensed that death was hovering over many of these people – while we were examining one recent admission, two died close by.

My father soon had his hands full with some very sick POWs.

Within a few days after the released prisoners of war had started arriving at our processing station, my two wards were filled with sick men, many of them living skeletons. Many people thought that it would be only a matter of “resting them up for a couple of days” and giving them plenty to eat. Those of us working with them, however, soon realized that a great many of them were desperately ill and urgent measures were necessary to save them.

But he also got the chance to meet with some of the scientists from the Manhattan project that developed the bomb.

It was our good fortune that the committee sent out by president Truman to study the atomic bomb explosion arrived in Nagasaki soon after we did. They asked to be quartered on board the Haven and inasmuch as I was in charge of the officer’s mess, it was my duty to look after them. As a result I had many interesting discussions regarding the atomic bomb and its possibilities with the members of the committee several of whom were scientists who had worked with the bomb from the beginning. Of course, they gave out no information except what had been released for publication, still it was a thrill to talk with the men who had done much to work it out.

After a lecture by one of the scientists my father concluded:

It may have ended the war for us, but it may some day be turned against us and we would lose the things for which we fought this long bloody war. Our country could be the greatest force for peace and security in the world if it would but accept the responsibility. Even out here few think of anything but getting back and forgetting what they have seen out here. “Let’s get home and look after our own affairs – what these people do out here is none of our business”, they say. And these are intelligent men – it depresses me. We are still selfish and materialistic, we have learned nothing apparently.

Courteous comments welcome, of course. The complete 12 page document – scanned and OCR’d into a PDF – is available here. Scanning from an old typescript is imperfect so there may be errors.

StorageMojo will be back to its regularly unscheduled programming tomorrow.


No-budget marketing for small companies

by Robin Harris on Wednesday, 13 May, 2015

You are a small tech company. You have a marketing guy but it’s largely engineers solving problems that most people don’t even know exist.

How do you get attention and respect at a low cost? Content marketing.

When most people think about marketing, they think t-shirts, tradeshows, advertising, telephone calls, white papers and brochures. Those are tools of traditional marketing.

It’s assembly-line marketing. You get a lead from handing them a t-shirt – a phone number or email – and then you contact him to see if they’re interested in your product and what the application might be.

You gather leads at the one end of a large funnel and over time, through repeated contacts, you keep narrowing the funnel from leads to suspects to prospects and, hopefully, customers. This works, but it is expensive.

If you are large enough and have the resources, go for it. But many small companies do not.

Content marketing is an alternative. You share what you know to help other people. Some of those people will discover they want what you have to sell and you’ve won a pre-sold prospect.

Backblaze, the consumer online backup service, is an excellent example of how content marketing can drive visibility and business. In pursuit of low cost storage they built their own high-density storage array. And open-sourced it.

The total price for a fully loaded array was a fraction of legacy vendors and their work went viral – (see Build a 135TB array for $7,384 for one example) getting them way more exposure than they could ever have afforded.

As they noted in their blog

Om Malik wrote about it at GigaOm, as did Robin Harris at StorageMojo, and Cory Doctorow on Boing Boing. Soon after, CrunchGear, VentureBeat, ZDNet, Mashable, TUAW, Electronista, MacWorld,, NetworkComputing, On-Storage, PSFK, Enterprise Storage Forum, eWeek and dozens of others picked it up. . . .

In the UK, The Guardian and the Inquirer wrote, but so did news sites in Africa, Australia, Belgium, Bulgaria, China, France, Germany, Italy, Poland, Russia, and Sweden. . . .

Bottom line: a single act of sharing put their name in front of millions of people. For free.

But Backblaze didn’t stop there. They were also keeping track of their disk drive experience. Since they bought on the open market, they had no confidentiality agreements with drive vendors and could share their results.

Backblaze dripped out drive life info every few months to create more buzz. Google “drive reliability” and guess who’s on top?

Did it work? Backblaze is profitable, growing and they bootstrapped themselves until they took their first funding in 2012. Yeah, it worked.

Bad example
Then there’s Company C (CC). Stellar engineering team, deep expertise, some very impressive tech. A publicly-held company that is struggling.

Why? Because no one knows who CC is and what CC does. Including me.

I buttonholed their CTO at a conference for the elevator pitch. I realized that millions of people would love a simple utility – using their core expertise – that you would get them lots of free publicity and a healthy search ranking.

I knew this was true because I’d written about the topic. It was one of my most popular posts for years! I explained that to the CTO.

I offered my help for free because getting this utility out would help millions of people with a common storage problem. We exchanged cards, he promised to call.

Still waiting. And CC’s business is still sucking wind.

Clearly, CC – and their marketing guy – doesn’t get content marketing.

The StorageMojo take
Many engineers have a long-time disdain for marketing and marketers. At the same time they also have products and services that have earned their strong loyalty, not realizing the role that successful marketing has played in making them happy customers.

Connecting technical abstractions to people’s lives is an art, not a science. Hundreds of millions of computer users know almost nothing about computer tech, so if you want them to care about your company you have to reach them where they live.

Backblaze has earned millions in free publicity by using what they knew to reach millions of techies. The kind of people other people ask for advice.

What do you know – and probably take for granted – that other people might care about?

Courteous comments welcome, of course. Maybe if I had told CC my help would cost $25k. . . .


Hike blogging: Sunday May 10 on Brins Mesa

by Robin Harris on Monday, 11 May, 2015

The Soldiers Pass, Brins Mesa, Mormon Canyon loop is my favorite hike. It has about 1500 feet of vertical up to over 5000 ft and the combination of two canyons and the mesa means the scenery is ever changing.

This shot is taken looking north from the mesa to Wilson Mt. It was a beautiful day and the fluffy clouds always add interest to a picture.

Click to enlarge.

Click to enlarge.

It was so nice I’m going to do it again, right now!


FAST ’15: StorageMojo’s Best Paper

by Robin Harris on Monday, 11 May, 2015

The crack StorageMojo analyst team has finally named a StorageMojo FAST 15 Best Paper. It was tough to get agreement this year because of the many excellent contenders. Here’s a rundown of the most interesting before a more detailed explication of the winner.

CalvinFS: Consistent WAN Replication and Scalable Metadata Management for Distributed File Systems impressed with its ambition.

Analysis of the ECMWF Storage Landscape, an analysis of a 100PB active archive, impressed with its scale.

FlashGraph: Processing Billion-Node Graphs on an Array of Commodity SSDs answered important questions about big data and flash.

Reliable, Consistent, and Efficient Data Sync for Mobile Apps holds out hope of a fix for a major failure of most sync services.

The mostly EMC paper RAIDShield: Characterizing, Monitoring, and Proactively Protecting Against Disk Failures offered useful insight on drive failure modes based on EMC’s internal support records – something StorageMojo has been agitating for these many years past.

Towards SLO Complying SSDs Through OPS Isolation offered the long needed observation that

. . . performance SLOs [Service Level Objectives] cannot be satisfied with current commercial SSDs. We show that garbage collection is the source of this problem and that this cannot be easily controlled because of the interaction between VMs.

And Skylight — A Window on Shingled Disk Operation – the FAST Best Paper winner – definitely deserves a post. But there can only be one StorageMojo Best Paper.

Best Paper
The winner is Having Your Cake and Eating It Too: Jointly Optimal Erasure Codes for I/O, Storage and Network-bandwidth, by K. V. Rashmi, Preetum Nakkiran, Jingyan Wang, Nihar B. Shah, and Kannan Ramchandran, all of the University of California, Berkeley.

The paper explores a holistic view of high-scale storage, simultaneously optimizing I/O, capacity and network bandwidth.

Our design builds on top of a class of powerful practical codes, called the product-matrix-MSR codes. Evaluations show that our proposed design results in a significant reduction the number of I/Os consumed during reconstructions (a 5× reduction for typical parameters), while retaining optimality with respect to storage, reliability, and network bandwidth.

In a Reed-Solomon 7+1 RAID 5 (7 data blocks and 1 parity block) the loss of a single data block causes 7 data reads and 7 times the size of single block in bandwidth consumption. When the loss is a terabyte+ disk, the glacial pace of reconstruction is mute testament to this feature of Reed-Solomon codes.

Dimakis et al. introduced minimum-storage-regeneration (MSR) codes, that can reduce the data transfer for reconstruction by 2/3rds or more.

However, the I/O overhead of MSR codes can be much higher than the Reed-Solomon codes used in current RAID arrays and some scale-out storage. For disk-based systems, that’s a problem.

The paper proposes product-matrix reconstruct-by-transfer (RBT)codes that achieve optimal system resource utilization. They also offer an algorithm that converts any product-matrix vanilla code into an RBT code.

The paper offers some graphs showing the results of experiments with Reed-Solomon (RS), product-matrix (PM) and RBT codes carried out on Amazon EC2 instances:

RBT Performance

The StorageMojo take
Disks are going to be with us for decades to come for cost and streaming performance. Networks – typically ethernet – are a limited and costly resource as well. Learning how to optimize both in scale-out systems is necessary.

The shift to high-IOPS media, like flash drives, means cheap I/Os on expensive media. But that doesn’t change anything for disk-based scale-out storage where massive capacity guarantees that data reconstruction will be common.

For future research I’d like to see more on the latency impact of advanced erasure codes. As object storage continues to displace file servers latency will become a critical issue. Update: K.V. Rashmi was nice enough to let me know that they are indeed working on the latency issue. Good to know! End update.

Courteous comments welcome, of course.


EMC II’s ragged last quarter

by Robin Harris on Monday, 27 April, 2015

As reported in a Seeking Alpha quarterly call transcript, EMC’s storage unit had a $75 million shortfall in Q1.

CEO Joe Tucci said

. . . we were disappointed that we fell a bit short of our Q1 revenue plan, approximately $75 million short. This $75 million revenue shortfall occurred in our storage business. That said, our storage product backlog did increase by $100 million year-on-year, right on plan.

Tucci again:

About two-thirds of this miss was due to the fact that we didn’t execute as crisply as we normally do. The other third was due to negative geopolitical effects in Russia and China that slowed down bookings.

He also cited CIO’s changing priorities: a focus on enterprise-wide “digitization” and cyber security. As a result:

. . . CIOs are searching for ways to reduce cost and increase efficiencies in their current infrastructures and legacy applications, while maintaining and even enhancing overall quality and performance.

Reducing costs for current infrastructure is a big problem for legacy vendors, most of whom are locked into high-gross margin business models that guarantee customers high costs and frequent forklift refreshes. EMC’s Information Infrastructure business knows that very well, as they laid off 1500 employees this quarter.

Evidently that layoff hit muscle and bone as well as fat, according to David Goulden, EMC II CEO. About a third of the $75 million storage shortfall was attributed to the layoffs:

In the course of the first quarter inside EMC information infrastructure, we reduced about 1,500 positions. Obviously when you do something of that size, it did impact some of our field-facing parts of the business and that caused a bit of a slowdown in terms of how quickly we got out of the gate in the first quarter on the go-to-market side.

But VMAX customers may not welcome Goulden’s (old) news that EMC is milking VMAX to fund new business:

. . . the remaining factor, which was again one-third of the miss, relates to the cost optimization cost management we’ve been doing inside of the company. As you know, we’re aggressively driving cost in our traditional business to fund investments in our newer businesses.

Good news – for EMC – on XtremeIO
Goulden expects that XtremIO will reach $1 billion in revenue this year, at excellent gross margins.

Actually the XtremeIO gross margin profile is actually quite attractive for a couple of reasons. First of all, it sold as an appliance and hardware/software combined. Secondarily, because it’s an appliance model, we start charging maintenance on basically day one for that product. So we don’t have to accrue for a multiyear warranty. So over the life cycle of an XtremIO system, the gross margin profile is actually quite good and very comparable to our traditional VMAX, VNX blended margins.

Goulden on DSSD

First of all, DSSD coming to market later this year, we couldn’t be more excited about that. I did mention though I expect to see some exciting news around XtremIO at EMC World as well. So the flash agenda is alive, well and strong, and they’re positioned quite differently. Basically XtremIO is a SAN attached Block Storage Device designed to make existing applications run much, much faster. DSSD is aimed more at the NextGen in-memory third platform applications. It’s well-handled. Protocols like Hadoop as well as the native protocols designed to be basically an extension of memory and the applications that we’re using the assistance for are actually quite different. So they’re both based upon flash technology, but the applications they are aimed for are different and quite complementary. So couldn’t be more excited about the flash agenda. You’ll hear a lot more about XtremIO in two weeks time, a little bit more about DSSD as well.

The StorageMojo take
The analysts weren’t happy about EMC’s failure to meet or beat the Street EPS expectation of 36¢, but EMC toughed it out, saying that their 31¢ met “internal expectations.”

I wonder about that $75m shortfall, of which some 2/3rds evidently migrated to the “expected” $100m backlog. If they were expecting $100m, why didn’t the orders shifted to this quarter increase it to $150m?

On the DSSD front EMC seems to be confident that DSSD is coming out this year, and since it is almost May, they should have a good idea by now. But, of course, EMC’s definition of “coming to market” is fluid: it could mean beta; or announce; or, least likely, shipping v1.0 product for revenue. (For more StorageMojo analysis of DSSD see What is DSSD building?, EMC goes all in with DSSD and EMC’s DSSD FUD.)

As the largest and most diversified storage vendor, EMC is admirably positioned to survive the current tsunami of new technologies and companies, unlike some of their less well-positioned competitors (see How doomed is NetApp?). But they are counting on being able to “brand” commodity hardware with a combination of proprietary and open-source software while maintaining 60-70% gross margins.

This may work for a while, but ultimately scale-out object storage will win file-based workflows. The first vendor to offer a scale-out software solution that doesn’t require costly hand-holding to install and manage will hit the Pivotal business model hard.

This game isn’t over by a long shot.

Courteous comments welcome, of course. I’m probably one of the only – if not the only – analyst who hasn’t taken a freebie from EMC in years. Boo-hoo!


How doomed is NetApp?

by Robin Harris on Monday, 13 April, 2015

The current turmoil caused by plummeting cloud storage costs, new entrants sporting modern architectures and the forced re-architecting due to flash and upcoming NV memories is a perfect storm for legacy vendors. Some are handling it better than others, but some, like IBM and NetApp, appear to be sinking.

NetApp is signalling that their 2015 sales may not reach expected levels – not a surprise – and that more layoffs – on top of the ≈1500 in the last couple of years – could come soon.

The latest troubling sign from NetApp is the failure of their FlashRay project to ship a competitive product. The VP in charge, Brian Pawlowski, left NetApp for Pure Storagelast month and the company folded the development effort into the ONTAP team.

According to press reports, the FlashRay project started almost 3 years ago, but has yet to ship a competitive product, despite the efforts of 100+ engineers. Given that there’s nothing FlashRay was supposed to do that was terribly novel, the problems are likely political than technical, a conclusion reinforced by placing FlashRay under the ONTAP team.

NetApp has well-known problems integrating acquisitions into their products. Now it seems they have problems developing new products as well. Not promising, given the threatening secular trends pinching their core business.

The secular trends include:

File server obsolescence. NetApp’s original raison d’etre is no longer state of the art – or all that interesting. File servers were a great idea 25 years – like RAID – but newer technology – object storage – is looking to replace them.

Cloud encroachment. When arrays were the only game in town, customers had to buy another when they needed capacity. But now old and rarely used files are moving to the cloud – and the repeat business with them.

Margin pressure. The combination of object and cloud storage – both much less costly than traditional arrays – is waking up customers and producing buying resistance.

A recent note from a disgruntled NetApp customer highlights this issue [edited for length]:

I’m currently engaged in replacing a FAS3240 going off maintenance with a newer version because Netapp offered a deal I couldn’t refuse. . . . [W]hile the drives and shelves are at least not inflated to ridiculous multiples (Why the heck am I being charged even a penny over retail?) they’ve now instituted a $34/TB surcharge to “license” me to use the very storage hardware I just bought from them.

. . . I am incensed at their audacity to charge me a FAT 60-70% margin on commodity hardware. . . .

The reason that customer feels he is being gouged is that the direct sales model is expensive. You pay for your salesman whether you like him or not – and he’s paid a lot more than your local Best Buy clerk.

The StorageMojo take
Bulk file storage hardware is commoditized. The upstart vendors are virtually all software-only, offering tin-wrapped software through resellers rather than direct sales, which saves a lot of margin dollars.

NetApp is in the position that DEC was in the ’90s, where commodity servers eviscerated the VAX business, aided by supply chain innovations by Dell and others.

Servers became a box to be ordered over the phone, not painstakingly configured with a sales engineer and delivered in a few months. Storage is finally following suit. And it doesn’t look good for NetApp.

Courteous comments welcome, of course.


EMC’s DSSD hiring is exploding

February 18, 2015

DSSD, the Valley startup acquired by EMC last year (see EMC goes all in with DSSD) is continuing to hire at an accelerating rate. Informed sources put the current DSSD team at 160 heads with plans to grow it to 800 over the next year. This is a program in a hurry. Hiring such numbers […]

7 comments Read the full article →

Latency in all-flash arrays

February 17, 2015

StorageMojo has been writing about latency and flash arrays for years (see The SSD write cliff in real life), with a focus on data from the TPC-C and SPC-1 benchmarks. The folks at Violin Memory asked me to create a Video White Paper to discuss the problem in a bite-size chunk. Latency is the long […]

3 comments Read the full article →

EMC’s missing petabytes: the cost of short stroking

February 10, 2015

A couple of weeks ago StorageMojo learned that a VMAX 20k could support up to 2400 3TB drives, it can only address ≈2PB. Where did the remaining 5 petabytes go? Some theories were advanced in the comments, and I spoke to other people about the mystery. No one would speak on the record, but here’s […]

6 comments Read the full article →

Help StorageMojo find the VMAX 20k’s lost petabytes!

January 21, 2015

While working on a client configuration for a VMAX 20k – and this may apply to the 40k as well, as I haven’t checked – I encountered something odd: The 20k supports up to 2400 3TB drives, according to the EMC 20k spec sheet. That should be a raw capacity of 7.2PB However, the same […]

4 comments Read the full article →

Facebook on disaggregation vs. hyperconvergence

January 6, 2015

Just when everyone agreed that scale-out infrastructure with commodity nodes of tightly-coupled CPU, memory and storage is the way to go, Facebook’s Jeff Qin, a capacity management engineer – in a talk at Storage Visions 2015 – offers an opposing vision: disaggregated racks. One rack for computes, another for memory and a third – and […]

13 comments Read the full article →

Friday hike blogging: Highline trail

December 19, 2014

Hike blogging got interrupted by some schedule conflicts and bad weather. Federal district court jury duty was one interruption. Some needed rain and cold temperatures was another. To make up for the lapse, here are two pictures from this week’s hike around Cathedral Rock. Winter is one of my favorite seasons because of the clouds […]

0 comments Read the full article →

WD acquires Skyera: whoa!

December 18, 2014

The Skyera acquisition could signal a sea change in the relationship between storage device and system makers. It is overdue. Traditionally, device makers avoided competing with their customers. This is what it made Seagate’s acquisition of Xiotech (now X-IO years ago so surprising. StorageMojo was critical of Seagate’s Xiotech acquisition because there were large and […]

9 comments Read the full article →

The 30% solution

December 2, 2014

The existential battle facing legacy storage vendors is about business models, not technology or features. Market forces – cloud providers and SMBs – are trying to turn a high margin business into a much lower margin business. We have already seen this happen with the servers. Many large minicomputer companies enjoyed 60 to 70% gross […]

6 comments Read the full article →

Friday hike blogging: Cockscomb Butte

November 21, 2014

Thanks to re:Invent and some other issues I haven’t done much hiking in the last 2 weeks. But back on the 8th I took a hike with my friend Gudrun up Cockscomb. There’s no official Forest Service trail so I was glad to have a guide for my first ascent. We spent about 90 minutes […]

0 comments Read the full article →