StorageMojo

The parallel computing/manycore initiatives may be missing the point. The challenge of manycore computing is burn up as many CPU cycles as possible doing things that we don’t do today because the computational cost is too great. Making existing apps go faster is secondary.

Today’s focus on creating manycore development platforms like OS X.vi server’s Grand Central may be a subset of where the real action will be. Maybe current levels of parallelization are good enough for most apps. So what does that leave?

How else can we use manycore computing?
Some thoughts:

Application speed up That won’t be the big win for current apps - most feel current processors are fast enough - look at the popularity of the Eee. But I’d love Handbrake to rip my DVDs faster.

Advanced UI capabilities such as voice recognition that are loosely coupled independent processes. Your application won’t run any faster, but it will be easier to use. This is an area Microsoft is looking at. Historically, the UI has been a major consumer of improved CPU and display capability.

New forms of communication and entertainment, such as 3D virtual worlds. This is an extension of the video editing market. And just think of the storage requirements!

Communities of cellular automata One core, one or a few automata. For example, Brian Tung’s and Leonard Kleinrock’s 1996 paper Using Finite State Automata to Produce Self-Optimization and Self-Control discusses using automata to guide a group of agents to cooperate on a task in a distributed systems environment.

Optimistic computing defined by David Jefferson in a 1990 ACM paper titled Virtual Time II: Storage Management in Distributed Simulation as

An optimistic simulation mechanism is one that takes risks by performaning speculative computation, which, if subsequently determined to be correct, saves time, but which is incorrect, must be rolled back.

Update: Rethinking virtualization because once a core costs $3 and you’ve got 32 or 64 of them in a $2k server, why would you spend hundreds of dollars on software to create virtual machines when you’ve got dozens of real ones?

There’s value in easy migration of virtual machines from one physical server to another. A “thin” virtualization layer atop a manycore OS - Windows 7? - could enable Microsoft to take back VMware’s market cap and reassert control of the entire OS stack.
End update.

High desert optimist
Many performance enhancements already use optimistic concepts. But the ability to throw massive computes from networks on a chip - oh, and how about reconfiguring those on-chip networks on the fly - could take us in directions we, or at least I, can’t imagine.

The StorageMojo take
The first effort with any new technology is to recreate what you could do with the old technology. It is only with the 2nd generation that the truly innovative stuff enabled by the new technology gets built.

Consider this an effort to short-circuit that historical process.

Comments welcome, of course. Thanks to Prof. West for pointing out the Jefferson paper to me.

Over on O’Reilly radar, Nat Torkington, does a neat riff on the enterprise SOA movement. He likens enterprise IT to a stern father:

. . . with strict rules, transgressors to be punished;. . .

while the Web is:

. . . the nurturing parent (the API provider) who encourages experimentation, self-development, and happiness.

It is an amusing read, but like lots of developers and engineers, Nat misunderstands enterprise IT’s motivation. They aren’t into control for the sake of control. (Well, some of them are, because some people are like that. But that isn’t the key reason.)

Control is a means to an end. The goal is production. Enterprise IT is a factory. The Web is a playground.

Expecting the two to be similar is a fundamental confusion. If you were put in charge of Goldman’s IT, you’d turn into a control freak too.

Statistical process control
Factories produce more and higher quality goods by reducing variability. Variability creates problems that cost money, either warranty costs or greater downtime/setup costs.

Enterprise IT is a factory
I first learned this truth when I was selling to engineers for development and to manufacturing for MRP. The engineers were all about the money and the freedom to tinker.

The manufacturing guys just wanted it to work. Save a few bucks on a 3rd party expansion rack? Why? Any glitch would wipe out the savings. So they wouldn’t go there.

The Web is a playground
Sure, there are people, like me, for whom the Web is instrumental in their work. I have backups for everything. The big destination sites do the same.

But for most of us the Web is something more casual: entertainment; shopping; news; communication. As long as it usually works we’re fine. The local cable loop goes down for a couple of hours and we’ll survive.

The StorageMojo take
The engineering and manufacturing cultures are very different, even though both groups are technical. This is why the gap between Silicon Valley and enterprise IT is so wide: the SV engineers think they get IT. And they don’t.

If you can show IT how your product reduces variability in their environment, giving them more certainty about production, you will have their attention. NUMA architectures, for example, add variability, despite higher average performance on tuned workloads.

So you could predict they wouldn’t be successful in the enterprise.

Words like “flexibility,” “experimentation” and “mashup” just don’t compute in the enterprise infrastructure. I’ve been as frustrated by the IT mindset as anyone, but complaining won’t change it. They are doing the best they can with the tools they have.

Want to do something great? Give IT better tools for managing variability.

Comments welcome, of course.

I’ve put together a couple of ~3 minute video excerpts from the Seattle Scalability Conference last Saturday. I’ve edited them to be useful standalone intros. Maybe they’ll entice you to learn more.

Chapel: productive parallel programming at scale
Bradford Chamberlain of Cray talks about a new language that he and his colleagues are developing. It isn’t released to the public yet, but he is looking for collaborators interested in moving it beyond a pure HPC focus.

Chapel appears to dramatically simplify parallel programming, if the code samples are any indication.

This is only 3 minutes out of 30, so if this whets your appetite be sure to look for the full video - shot on better equipment - on YouTube. As of this writing it isn’t up yet.

Carmen: a scalable science cloud
This is 3 minutes from early in a talk that Paul Watson of Newcastle University gave on cloud computing for neuroscience research. Neuroscience has a number of issues - including 100,000 researchers worldwide - that lend themselves to a cloud approach.

The full talk is up on Google Video.

Commenters on my ZDnet blog
inform me that Microsoft has solved all these multicore programming problems. Maybe the next scalability conference should be held in Redmond.

Can single-user OS X be far behind?
Here’s the official Apple announcement:

For business-critical server deployments, Snow Leopard Server adds read and write support for the high-performance, 128-bit ZFS file system, which includes advanced features such as storage pooling, data redundancy, automatic error correction, dynamic volume expansion, and snapshots.

The StorageMojo take
Cool! And only 2 years later than I’d predicted. I’m an optimist.

As I noted almost 2 years ago:

StorageMojo.com has devoted time to this issue because today’s computer business is largely driven by consumer computing, not enterprise computing. Putting a really modern integrated file and storage management system on a consumer OS would raise the bar for everyone else.

I stand by that.

Comments welcome, of course.
For more on ZFS see:
Want to know more about ZFS? I’ve been hot on it for over a year. See:

• ZFS: Threat or Menace? Pt I
• ZFS: Threat or Menace? Pt II
• Bring Me The Head Of WinFS
• ZFS On Leopard: How Cool Is That?
• Means, Motive & Opportunity: Apple Kills the Media Center PC

Gary Orenstein has published a podcast of a discussion we had a couple of weeks ago about cloud computing.

Cloudy days on the hype cycle
Cloud computing and storage is still climbing the hype cycle. Remember client-server computing? It was going to change the world. It did, but not as we expected. Now it is an invisible part of the infosphere.

Likewise cloud computing. It is another arrow in the quiver, not a howitzer. The critical issue is how creatively and transparently we utilize it. No doubt many of us will be surprised.

In 15 years cloud computing will be as obvious to users as client-server is today.

The StorageMojo take
The podcast discusses other issues in cloud computing and storage. Kudos to Gary for putting on the cloud computing series.

Comments welcome, of course. I’ve done work Gary’s employer, Gear6, in the past. This discussion was conducted gratis.

I’m relaxing in beautiful Port Townsend, Washington today, under the gray skies of the coldest June in almost 100 years. The fire in the wood-burning stove and Frank’s strong coffee provide the good cheer.

Temporal compare
My comments are more impressionistic than considered. No “best of” selections now.

Comparing this year’s conference to last year’s is tricky. The Googlers who selected the papers didn’t profess a theme, choosing what they found interesting. So it may be a Rorschach inkblot test to see a pattern in the 2 conferences, but I do.

Last year’s conference focused on cluster scalability - building really big clusters that go beyond the 8,000 or so node clusters Google uses. Jeffrey Dean last year was open about Google’s desire to knit their data centers into a single global name space.

This year the focus moved up the stack to file systems and programming languages. The problem of multi-core chips seemed especially pertinent.

Bradford Chamberlain’s Chapel language attacks the issue of programming multicore/processor systems and sounded promising [download a technical pdf on Chapel here].

Vijay Menon’s “Scalable multiprocessor programming via transactional memory” seeks to replace clustering’s traditional reliance on threads and locks with an atomic transactional model of file access. He noted that Azul Systems uses hardware transactional memory in their 800+ core Java servers.

And there was more.

The StorageMojo take
Scalability is a key problem. The Googler’s desire to involve industry as well as academe gives this conference a dual personality that I like. At its best we see ideas beginning to morph into platforms.

The slow take will be coming as I look further into the papers that were presented. In the meantime Garth Gibson, CMU prof and RAID paper co-author, made some interesting comments on the earlier Scalability Conference post.

Comments welcome, of course. Looking forward to returning to NoAZ tomorrow.

That’s right: the second Seattle Conference on Scalability - sponsored by Google - is this Saturday [see a couple of posts back for more info]. I’m also attending the bonus meeting in Fremont Friday evening.

I’m bringing the video production backpack and I’ll try to get some video clips up if I capture something short & interesting. Sunday I’m going to get some Father’s Day love and then up to charming Port Townsend for a couple of days R&R with Frank.

If you’ve spent time in PT, you know Frank. So no guarantees on the video.

The StorageMojo take
The StorageMojo team has been celebrating the 500 post mark - by not posting. But now its back to work.

If you’re at the Conference look me up. Always pleased to meet StorageMojo readers - even occasional ones - or people who could be StorageMojo readers.

I wrote a short piece on ZDnet about Los Alamos National Labs new Cell Broadband Engine based supercomputer, Roadrunner. With ~14k v.3 Cell processors - an earlier version powers the PS3 game console - and another ~7k dual core Opterons, the Roadrunner’s ~3,250 compute nodes pack a lot of compute cycles.

The key compute element is the new version of the PS3 chip - called a PowerXCell 8i Processor - features 8x faster double-precision floating point and over 25 GB/sec of memory bandwidth. And it can address 64 GB RAM. There are 4 8i’s per compute node.

Nothing I read mentioned the disk storage - until the friendly Panasas PR person suggested I talk to Larry Jones, VP Product Marketing. Panasas is providing the back end storage for Roadrunner.

I did, and here’s what I learned.

LANL storage infrastructure
LANL’s 6 supercomputers + Roadrunner share the Panasas storage through LANL-developed IO nodes. While Roadrunner itself uses dual-data-rate 4x Infiniband for internode communication, the I/O nodes attach to Panasas through trunked GigE.

The advantage of the I/O nodes is that the entire Panasas storage pool is available to each supercomputer. Lots of bandwidth.

Roadrunner currently has about 80TB of RAM, roughly 24 GB per compute node. That works out to about 4 GB RAM per processor.

The jobs these machines run are huge. A simulation can run 6 months or more. Depending on criticality a job gets checkpointed every hour or maybe once a day.

The Panasas installation at LANL, begun in 2003, is currently 2 PB. Assuming an average of 500 GB drives, that means 4,000 disk drives.

Panasas uses 5 trunked GigE links to each of the 8 controllers in a single rack. They are now in beta for 10 GigE, which reduce link count from 40 to 8 per rack while doubling bandwidth.

The hot rodders at LANL should like that.

The StorageMojo take
Roadrunner’s 80 TB RAM is a sizable storage infrastructure in its own right. Keeping it fed and backed up is a major job.

Consumerization of IT is a common concept - but what we see here is the consumerization of HPC: Playstation CPUs; SATA drives; Linux OS; air cooling. The old model of highly customized kit for HPC is dead.

Which is a good thing for the rest of us. We get some of the smartest people in computing working on platforms that we might also use, developing applications that otherwise would never be available to the consumer market.

I’ll never run molecular dynamics codes, but maybe my kids will. After all, I can now edit feature length movies on my desktop. Who would have believed that just 20 years ago?

Comments welcome, of course. Disclosure: I did some work for Panasas last year and - who knows? - might do some more in the future. I like the team and the way they are pushing pNFS.

Consumerization is the ultimate scale-out application
I spoke to EMC’s CTO, Jeff Nick, at EMC world and video’d his comments. I didn’t know what to expect, as some past EMC CTO’s have been lightweights whose insight wasn’t up to Silicon Valley standards.

But Nick is different: a former IBM distinguished engineer Fellow - their highest technical level; holder of many patents; leader of IBM’s grid initiatives. He swims in the deep end of the pool. Once he realized I’d done some homework he proved voluble and insightful.

We discussed several topics, including why Maui is late (short answer: productizing advanced technology is hard). The best part was describing what Paul Maritz’ Pi Corp brings to the table.

Here’s the video:

The StorageMojo take
EMC is taking Dell’s purchase of EqualLogic seriously. They are intent on building EMC into a trusted consumer brand for personal information storage in the cloud.

That is easier said than done. Yet Google - the obvious 1st choice for this market - has hurt their brand by dithering on privacy issues. Why trust your most private data to a company that makes its money selling your information? 

EMC is unleashing a triple whammy on its traditional competitors

• Leading edge technology in Maui
• Consumer-focused services with Mozy and Iomega
• A next-gen software infrastructure in Pi that - if it delivers - will change how consumers manage their data forever

These are all game-changers. Together they bring on the consumerization of IT - storage industry division - at a fast pace. While cloud storage must still overcome the Internet’s 3 9s availability, EMC’s added-value approach is promising.

Comments welcome, of course.

On Saturday June 14th Google will host the 2nd Seattle Conference on Scalability. I’m planning to be there.

Interesting topics
Given the dual-track format, I may not be able to see everything I’d like. Here’s what looks cool to me:

• CARMEN: a Scalable Science Cloud. “CARMEN is a $9M project building a scalable science cloud. Its focus is on supporting neuroscientists who will use it to store, share and analyse 100s of TBs of data.”
• Chapel: Productive Parallel Programming at Scale. “Chapel is a new programming language being developed by Cray Inc. as part of the DARPA-led High Productivity Computing Systems Program (HPCS). . . . Language concepts that support this goal include abstractions for globally distributed data aggregates and anonymized task-based parallelism.”
• Conquering Scalability Challenges with Transactional Billing. “A huge problem faced in the world of billing and charging is the ability to process a large number of transactions per day.”
• GIGA+: Scalable Directories for Shared File Systems. “This talk is about how to build file system directories that contain billions to trillions of entries and grow the number of entries instantly with all cores contributing concurrently. The central tenet of our work is to push the limits of scalability by minimizing serialization, eliminating system-wide synchronization, and using weaker consistency semantics.”
• maidsafe: A new networking paradigm. “We describe a significant new way of networking and data handling globally. This data centric network is likely to revolutionize the IT industry in a very positive fashion.”

Here’s a link to registration and all the presentations.

The StorageMojo take
They all look good, don’t they? I’m very interested in the transactional billing talk, since the major reason for the disconnect between IT and the lines of business is that IT is essentially tax-supported rather than priced.

IT costs are peanut-buttered across LOB overhead - so expensive resources get over-consumed - leading to over-configuration and business value blindness. These guys aren’t talking about that problem - but it is the key to making utility computing and SOA business tools instead of marketectures.

Comments welcome, of course. I’m looking forward to seeing Seattle with blue skies.

What’s in a name?
I’ve started doing some work for IBRIX. Despite looking at their web site several times I’d never understood what, exactly, they did. Turns out they make a cluster file system. One good enough for Dell, EMC and HP to resell.

As part of our get acquainted process I spent some time with their CTO, Sudhir Srinivasan. I taped him discussing their segmented file system. 

The Cliff Notes version
The segmented file system is similar to the Google File System in that it uses commodity servers with a local file system and puts a software layer above that to cluster them. It differs in that there is no dedicated metadata server node that needs to be hardened or prepared for failover. 

The secret sauce is that when there is a file request, the receiving node can swiftly refer the request to the node(s) with the data. And it does it without a lot of back channel chatter eating up cycles and bandwidth. 

The StorageMojo take
I haven’t fully grokked the technology. But the appeal is undeniable: commodity servers and storage; scalable; adept at both large and small files - the latter usually a problem with dedicated metadata servers.

EMC is using IBRIX as the file system for the local storage pools under their upcoming Maui product - IMHO their most important product introduction since the original DMX. I got the feeling Joe Tucci wasn’t entirely happy about that at his press conference this week, but hey! they’ve got bigger fish to fry.

Comments welcome, of course. If you’ve used IBRIX how has it worked out for you?

Greetings from Las Vegas
And EMC World 2008.

Dave Donatelli, president of EMC’s storage business, presented to the press room this morning. His most interesting statement was that flash drives will have cost-parity with, and therefore replace, high-end rotating magnetic disks, by the end of 2010.

Let’s run some numbers
Dave said that EMC has measured STEC’s flash drives at 30x the IOPS of a high-end disk with sub-millisecond access times. That alone would justify a premium over existing drives. He also said that the performance of the flash drive was better under load. A double win.

A 15k 74 GB Seagate SAS drive is about $175 or roughly $2/GB. A 2 GB Single Level Cell (SLC) flash chip is currently about $8/GB on the flash spot market. If flash keeps dropping at 50% a year they’ll be where the current disk price is in mid-2010.

But that’s raw chip vs finished disk
The remaining question is how much does the chip controller and other infrastructure cost? STEC isn’t selling its 74 GB flash drives for $8/GB - $80/GB is closer to the mark. Volume should amortize their engineering costs. PC boards are cheap.

That leaves the flash translation layer. That should fit nicely on an FPGA and, once the bugs are out, on an ASIC. The 1st ASIC is expensive; the 100,000th is cheap.

The StorageMojo take
Flash drives don’t need absolute price parity to win against high-end FC drives. Getting within 30% should do it for most people. Their performance advantages are worth at least that.

Of course the drive vendors aren’t going to sit still. They can pull several levers before breaking the glass for the big red one labeled “margin.” Many have claimed disks are dead and they’re all gone.

But this looks serious. High-end drives are a small piece by units, but their high margins would be sorely missed.

Comments welcome. This isn’t about notebook disks which are currently less than $0.40/GB and headed down much faster than FC and SAS drives.

This is StorageMojo’s 500th post! Thank you, thank you.

Just sat through EMC CEO Joe Tucci’s keynote for EMC World 2008.

Conspicuous by its absence: Maui; the software piece of the Hulk/Maui global data repository [see EMC's Maui and everybody else].

Joe promised Hulk/Maui in 7 months last November. EMC showed the hardware at NAB last month. But the software has always been the hard part.

Some inside skinny
Spoke to an EMC customer - and loyal StorageMojo reader - after the Goo Goo Dolls concert last night. He’d seen a Maui demo and confirmed that it looked like a productized OceanStore - a layer of software above local pools of storage.

The StorageMojo take
This is a very tough problem, so I’m not surprised that they didn’t make Joe’s deadline. I expect they’ll announce something - there’s an item about Global Data Recovery yet to come - but there isn’t the kind of hoopla that says “we nailed it!”

Which is good news for EMC competitors. Well marketed, Maui is a very compelling reason to turn even more of your data center over to EMC. Competitors are catching a break. Will they take advantage of it?

Comments welcome, as always. And here’s a picture from the Goo Goo Dolls concert.

Getting rid of the hacked files and spam links wasn’t the end of it
Dreamhost notified me that the load on my server was excessive and they’d disabled StorageMojo.

Yikes! Had I been hacked again? DDOS attack? What?

Building the correct mental model
In short order I brought up my SFTP client, my tracking site, the Dreamhost webpanel and my son on chat. He had me toss a new index.html file into the site folder to let people know that the problem was getting addressed.

On to problem solving
It took a while to figure it out because I’d never seen it before.

The load was coming from Google referrals for charming search terms that I’m going to misspell on purpose in hopes of not attracting similar traffic:

• download sh*mail
• downlode free 1ndian s3x movies
• pharmasuitical affiliate prom0
• 0rgish/behe*ding
• h1nd1 p0rn m0v1es

*Lots* of pee-oh-rn requests for many different ethnic types. Some things are universal - at least among guys.

There were no hacked files still on StorageMojo - I’d gotten them all last week and they were still gone. But the tracking site was referring to them, so for a while I thought they were there but that for some reason I couldn’t see them.

But then my son checked what happened when someone tried to go to the spam links. The site was delivering a “system error” message - not the static 404 page I’d expect - so the site wasn’t delivering the spam content and it really was gone. Presumably processing for the “system error” page created much of the extra overhead Dreamhost was seeing.

For a while StorageMojo was getting thousands of hits an hour from these Google referrals. At some point Google must have crawled the site again, saw the content was no longer there, and stopped referring people.

Not a moment too soon!

So what was this all about?
My son hypothesized:

This looks like a two-step scheme…step one is that they hacked your site and got all those bad SEO files uploaded. Step two is to send lots of fake Google traffic through your site to increase PageRank.

Then I went one step further and checked out one of the spam pages that Google had cached. In big bright colors it told me that my XP system was infected with viruses and I should download their *free* virus scanner.

Whoa, scary. Except I’m on a Mac.

Botnet recruitment? I don’t know.

The StorageMojo take
I’ve made a number of changes to tighten up StorageMojo. As I was researching this I found that there are many security “folk remedies” out there, but very little on what the high priority issues are.

Keeping software up to date seems to be the critical success factor - and sad to say, I’d been lax. In addition to keeping current I’m now checking my site files more often among other changes.

Hopefully these requests will tail off as Google stops referring people. And StorageMojo can go back to being a quiet little site.

Thank you for your patience.

Comments welcome, of course.

First it was the bogus “national security” argument against a Chinese buy-out of Seagate. Now it’s suing STEC over solid state drives (SSD). Has William Watkins, Seagate’s CEO, jumped the shark?

STEC said, per legal SOP, that the suit was “without merit.” After looking at the patents I agree.

Look at the patents
There are 4 patents named in the Seagate suit.

• 6336174 issued 2002
• 6404647 issued 2002
• 6849480 issued 2005
• 7042664 issued 2006

The links will download pdf versions if your insomnia is acting up.

6336174
The first patent covers an invention called a hardware-assisted memory module (HAMM) that can, when there is a system glitch, isolate

. . . itself from the host computer system before copying digital information from volatile memory to nonvolatile memory.

This reminds me of a common RAM-based SSD strategy - used 20 years ago in DEC SSDs - to copy data held in RAM to a disk drive when power failed.

6404647
The 2nd patent is for a solid-state mass memory storage device. This device

. . . comprises a printed circuit assembly and a plurality of non-volatile, high density storage devices mounted to the printed circuit assembly and electrically connected thereto.

A picture is worth 1,000 words:

More than a passing resemblance to a compact flash card - a product I bought in 1993.

6849480
“Surface mount IC stacking method and device.” This patent covers a technique that solves 3 problems:

• Routing signals through stacks of similar chips
• Stacked devices with identical dies that are made into distinct chips - flashed? - later
• Long, high-capacitance interconnects between stacked devices

Seems like folks have been stacking chips for a while. Is there anything new here?

7042664
“Method and system for host programmable data storage device self-testing.”

. . . providing programmable self-testing of a data storage device comprises selecting one or more host programmable tests stored in memory in the data storage device by setting data in a first log in memory of the data storage device.

This invention’s goal is to enable disk drive testing without removing the drive from the host. It embodies the concept of the host providing test parameters for an attached device - which the patent imagines to be a disk.

Size matters
Part of Seagate CEO William Watkins’ pique with STEC is fueled by a suit from 3.5″ drive inventor Rodime that Seagate paid $45 million to settle. Rodime patented the 3.5″ form factor for disk drives - and got the courts to enforce it.

Watkins knows that patenting disk drive form factors is silly - they have to be standard sizes - but if the USPTO grants them and the courts enforce them, why not?

The StorageMojo take
The IC stacking patent may have some merit - I’m no judge of chip packaging. But the other patents, especially for compact flash, seem dubious at best.

The good news is that the Supreme Court has forced the patent courts to change course. In KSR v Teleflex the Supremes ruled that the non-obviousness is a legal question, not a factual one. That bit of hair-splitting means that lower-court rulings can be appealed. Under the old rule once the trial court made a “finding of fact” it could not be re-examined in the appeal.

Rodime would have lost under that rule. While it will take time for KSR to play out, in the short term it almost certainly reduces the value of existing patents - like Seagate’s ludicrous flash drive patent.

While some have portrayed this as Seagate trying to stymie a competitor, it’s more likely that Seagate believes STEC has some useful technology. The promise of a costly legal battle might persuade a smaller company than STEC to settle with a quick cross-licensing deal.

That would help Seagate catch up in the high-end flash SSD market. I hope STEC resists that temptation and continues to focus on the knotty issue of fast random write flash performance.

Comments welcome, of course.