StorageMojo

I had a con call with Chris Gladwin and Russ Kennedy of Cleversafe a couple of weeks ago. They’ve come to market with a product line that seeks to deliver:

• Massive scalability to meet growing digital content requirements
• Unprecedented Security and Privacy for critical digital assets
• Survivability against disasters, dishonesty and time
• Extremely cost-effective infrastructure compared to traditional methods

That’s a quote from their pitch.

Cleversafe’s product line
Cleversafe, IIRC, started as a software company, but their announced products come in nice rack-mountable boxes. There are 3 of them:

• CS Slicestor - Dispersed Storage server - $11.3k
• CS Accesser - Dispersed Storage router - $12.3k
• CS Manager - Dispersed Storage network manager - $12.3k

The Slicestor is a 1U storage server containing 4 disks. The Accessor slices up the data and distributes it - think slice router. The Manager works out of band to monitor and manage the storage network components.

I assume the pricing includes some room for volume discounts. There is an open-source version (c. 2006) of the software. The company intends to offer a software-only version as well.

Why hardware?
The Conventional Wisdom in VC circles is that tin-wrapped software ramps revenues faster - hey, you’re selling tin + bits - at the cost of lower margins and loss of focus.

Qualifying hardware is non-trivial; so you tend to stay on one platform longer than you should. At liquidity event time, software companies fetch higher multiples, so it may be a net loss. VCs live by the Golden Rule: he who has the gold makes the rules.

What it does
Cleversafe has an iSCSI or block storage interface. It takes the data, slices it into small pieces using Information Dispersal Algorithms and then ships the slices off to storage either locally or around the world.

In the latest version you can specify how many slices the system makes and how many slices are required to rebuild the data. If you have 11 data centers around the world, you can specify that, say, 6 are required to recreate the data.

You could lose access to 5 data centers and still recover. If the local controlling authority busts into 3 or 4 data centers, they get nothing. Pretty cool if you worry about corrupt government officials getting hold of your company secrets.

The company is planning on adding FTP, CIFS and NFS in the fullness of time.

How well it works
Cleversafe claims that given sufficient low-latency bandwidth the dispersed storage is as fast as a local disk. That’s a tall order, but for now I’ll take their word for it.

Who should buy it?
The company is aiming the Dispersed Storage Network at ISPs to offer as a service and multinationals with round the clock operations and critical data.

How it works
Cleversafe uses Cauchy Reed Solomon erasure codes to slice and dice the data. These codes have several advantages:

• More capacity efficient and failure tolerant than parity codes
• Doesn’t require a license
• Code and decode are faster than other stack operations

If you’d like to play with Cauchy Reed Solomon, check out Dr. Jim Plank’s software page which includes

. . . Reed-Solomon coding, Cauchy Reed-Solomon coding, general bit-matrix coding, Reed-Solomon coding optimized for RAID-6, and Liberation coding. The documentation provides some tutorial material on matrix and bit-matrix based erasure coding.

I met the good doctor at FAST, where he was delighted to find that Clevesafe - also a FAST presenter - was using techniques he’d worked on a decade ago.

The StorageMojo take
I’m impressed with what Cleversafe has done. They will look even smarter after EMC’s Hulk/Maui announcement this spring. I suspect they’ll be bought by year’s end.

Kudos to the Cleversafe team.

Comments welcome, of course.

What’s wrong with White House backup?
I published a review of David Gewirtz’s book Where Have All the Emails Gone? over on ZDnet.

A quick overview:

• The White House may or may not have lost 5 million emails. They aren’t sure.
• Gewirtz, an email expert, started investigating the White House email infrastructure and found:
• The mail archiving process is unprofessional and unworkable.
• The claimed loss of email in a Notes to Exchange migration is highly unlikely.
• Over 100 million emails from the White House were sent through an insecure ISP in Chattanooga TN.
• Existing law - the Hatch Act - mandates an external email system for partisan political activity, a ludicrous requirement in a 7×24 Washington.

The Hatch Act prescribes what partisan political activities are acceptable for federal employees. One of the prohibitions is the partisan use of government property. While a good idea in general, in the case of telecom the prohibition is senseless.

White House communications need to be secure. When we force White House employees to use multiple email, IM and computer systems it is inevitable that material received on the internal system will go out over the external system. A single secure system is easier to achieve.

This isn’t about George Bush
This is about maintaining records so the next administration can know how policy got developed and what committments were made. I’ll let others worry about if the loss of the emails was part of a deliberate attempt to cover up criminal activity.

Ironic, isn’t it?
American companies are spending billions for backup and archive software and hardware. But the White House, head of an executive branch with a $3 trillion budget, can’t manage its email backups despite a clear legal requirement to do so under the Presidential Records Act?

The StorageMojo take
Gewirtz recommends that a professional, non-partisan IT organization be detailed with the job of protecting and archiving all White House email communications. There are many groups with the ability and the motive to snoop White House email going out over the public Internet. That has to stop.

Making a single entity responsible, as the Secret Service is for Presidential safety, is the best way to ensure that vital public records are protected. It will also help remind White House officials that they are accountable to the people of the United States.

Comments welcome, as always. BTW, Congress also needs to clean up its data protection act. It is less urgent thant the White House, but just as important.

Update: As luck would have it the New York Times reports another Bush attack on America’s right to know. After passing Congress unanimously he’s gutting the latest freedom-of-information law in the budget. A new high in bipartisanship! Less than a year to go!

What’s the difference?
I came across a thoughtful essay on the “Top Ten Differences between Disk-based Archive & Disk-based Storage” in the MatrixStore blog. MatrixStore is a Mac cluster-based disk archive for Apple’s to-be-announced-RSN Final Cut Server.

MatrixStore is focused on one market segment - video content archiving - but their comments seem to be generally applicable. With 2008’s likely focus on the disk-based backup and archive market, it is worth starting the conversation now.

Key points
SANs aren’t designed for archiving.

Reason 1.

If you are archiving your data, it’s probably because you don’t want to lose it.

Raison d’etre for a disk based archive? To keep data - safe. For a SAN? Speed of delivery, QoS… You wouldn’t put 256 bit delivery checksums into a SAN; SANs cut corners on flushing to disk; SANs don’t build in search or audit-trails, or security; SANs can down completely because of single-points-of-failure in the hardware; a bad software update in a SAN and…. Don’t do it. With nursing care and attention they can run fine for years, but they are inherently tightly coupled, software version sensitive, high maintenance, error prone and hardware technology dependent… even if they are brilliant at fast storage and delivery of information…

A disk-based archive must be: loosely coupled and free from dependencies between hardware components on independent nodes (surely the greatest example of a loosely coupled solution is the world-wide-web; you have no fear on the www that a server going down, say, hosting an IBM site, is going to bring down another in Cupertino!); free from requiring constant latest updates to software/firmware; able to guarantee safe delivery and storage of data; and basically, able to safely, securely store and protect data for year upon year, without complications, manual intervention, spanners…

Archives must be engineered for easy adoption of new technology
In storage everything is cheaper next quarter. So why buy now?

Reason 2.

There’ll be bigger, better, cheaper, more efficient disks in 2009, and in 2010, and in 2011…

Will there be bigger, better, cheaper, more energy efficient storage devices coming out this year, and every year that follows? Yes, of course there will be.

In your SAN do you have to mirror between like-sized devices? What happens when one of those devices goes down in 2 years time? Do you end up throwing away the good device? In your SAN can you bolt on new technologies as they arrive; holographic disks that store 10TB a shot, or new fibre connectors?

In ZFS can you decommission a part of a storage pool, replacing it with new storage devices without significant bleeding edge techniques and without disrupting the rest? Ideally, it be great to bolt new technology into an archive, as and when they arrive, rolling out old technologies if they reach the point of diminishing returns; to be able to do that whilst always seeing a single archive storage cluster; and without a maintenance or data migration headache; or should I say; without risk. A disk based archive can achieve that, if selected carefully.

Vendor handcuffs
Long-term storage and proprietary products don’t mix. Along with upgradeability-in-place, this should be high on customer checklists.

Reason 3.

Vendor tie-in is more like Vendor hand-cuffs.

OK - this isn’t strictly about SAN vs Disk based archiving; but fact of the matter is that most SAN/any other disk-based storage solutions tie you in to a particular vendor, which is great when they are supplying the ‘best-in-class’ solution of the moment at time of purchase, but not quite so clever when you come to upgrade that solution a year down the line and they aren’t offering the best in class anymore.

The archive should be vendor independent otherwise, for many reasons, you’re just creating tomorrow’s headache with a solution from yesteryear.

Stability and security

Reason 5.

Viruses. Hackers.

Choice one:

“out of the box” configured with encryption, firewalled, data locked down, all access to data routed through PPK, all maintenance functionality requiring 256 bit passwords.

Choice two:

bolt on each of the above to your favourite SAN/filesystem. Wait five years as your conglomerate of software solutions evolve (along with the workforce) and cross fingers. A disk-based archive must be secure out-of-the-box.

There’s more, of course, and if you are interested please read the whole essay and respond here with your thoughts so every one can see and respond.

The StorageMojo take
EMC’s upcoming backup and archive cluster, code-named Hulk/Maui (HW/SW), will drive a lot of customers to think about this topic. Of course, EMC’s famously disciplined sales force will scrupulously limit Hulk/Maui sales to B&A applications for the first several months weeks days hours after its release. Once the customer utters the magic word “Isilon” Hulk/Maui will suddenly be ready for enterprise use.

[I hope someone has mentioned this to the Maui engineers: forget about summer vacation.]

Disk-based backup and archive is a fast growing application with very different requirements from SANs, arrays and fast NAS boxes. Data migrations will be increasingly infeasible. Management has to be stoner-on-the-night-shift-proof. And the data can’t be held hostage by proprietary standards.

Companies do discontinue products or go bankrupt, after all.

Comments welcome, of course. Anything else?

Mighty Google is worried about getting the shaft from telcos. Shouldn’t you be too?

Larry Downes imagines the worst
Larry Downes’ arguments against net neutrality are button-pushing propaganda designed to inflame, not illuminate. I expect better from a University of Chicago trained lawyer.

In response I’m going to look at the text of a net neutrality proposal and then at Mr. Downes’ mostly irrelevant points.

What is being proposed?
Let’s start with Congressman Markey’s proposed Network Neutrality Act and decide for yourself? The PDF is only 11 pages, while the dread regulations are barely 4 pages.

Here are the core “regulations” Mr. Downs is so afraid of. From the bill, Internet providers may

not block, impair, degrade, discriminate against, or interfere with the ability of any person to utilize their broadband service

for lawful content, applications and services. I expect no less.

Furthermore, service providers are required to

clearly and conspicuously disclose to users, in plain language, accurate information about the speed, nature, and limitations of their broadband service

Truth-in-advertising? Telco marketing will never adapt!

How about this requirement?

offer, upon reasonable request to any person, a broadband service for use by such person to offer or access unaffiliated content, applications, and services

Requiring telcos to take new customers? Tricksy Mr. Markey.

Here’s what gets the telcos mad
The bill requires that a telco

not discriminate in favor of itself in the allocation, use, or quality of broadband services or interconnection with other broadband networks

Isn’t that a Communist common-carrier requirement? Gee, why own a big network if you can’t screw your competitors? No wonder the telcos are miffed.

This gets them madder
Broadband service providers will be required to:

offer a service such that content, applications, or service providers can offer unaffiliated content, applications, or services in a manner that is at least equal to the speed and quality of service that the operator’s content, applications, or service is accessed and offered, and without interference or surcharges on the basis of such content, applications, or services

Hm-m? Requiring equal treatment of unaffiliated content? Just like telegraph companies had to 160 years ago? Medieval.

Now telcos see red
Here’s the heart of the matter. The law would require that

if the broadband network provider prioritizes or offers enhanced quality of service to data of a particular type, prioritize or offer enhanced quality of service to all data of that type (regardless of the origin of such data) without imposing a surcharge or other consideration for such prioritization or quality of service

[emphasis added]

The heart of the matter
The telco can charge for more, time, speed or bandwidth, but they can’t charge more for preferential treatment of packets. This is what being a common carrier means.

The Downes critique, fearlessly knocking down straw men
Larry’s article is mostly a big cloud of smoke, irrelevant to the question of net neutrality:

• Railroad asset accounting has nothing to do with treating packets equally
• Airlines wanted the CAB’s regulation and fought to preserve it to avoid competition
• SOX addresses another financial accounting problem

There are many examples of regulation that works: the drugs we take; the airlines we fly; the building codes that make our homes, offices, schools and factories safer.

Network designers demand non-neutrality?
Mr. Downes then concludes that net neutrality would stymie web engineers efforts to optimize Web traffic.

He might be referring to Bob Briscoe’s IETF problem statement We Don’t Have To Do Fairness Ourselves which discusses the unfair use of TCP, a protocol designed to be fair. Briscoe says the IETF needs to:

. . . focus on giving principled and enforceable control to users and operators, so they can agree between themselves which fair use policy they want locally.

This is very different than giving the telcos a blank check to impose anything on a captive audience of Internet users. All our history with monopolies and duopolies tells us that without basic ground rules the telcos will ream the users.

The deep end
Then Mr. Downes goes off the deep end, positing that a complaint would force the FCC to open every affected packet on the network to determine if a telco were violating the law. This is silly.

It would be far easier to monitor a sample of disputed traffic as it is injected and measure its performance across the network. But how likely is a complaint if the telcos are prohibited from discriminatory treatment? Why would they develop the ability?

What is much more likely is that a telco whose unpopular policies have alienated the public would want government protection. Politicians would provide protection - for a price - such as ready access to the databases that store your surfing habits.

The StorageMojo take
Ultimately, net neutrality is a choice between private exploitation of network users by opaque, profit-driven companies or publicly debated ground rules that set minimum standards. The telcos and their claques whine about how hard all this is, but I’m confident the engineers can solve the problems.

Mr. Downes - like George Ou - doesn’t address the issue of fairness between users and providers. If Google is worried about getting reamed by telcos, why aren’t you?

The SF Federal Attorney is 2 and 0 on these backdating prosecutions. Stephanie Jensen, former VP of HR, said she didn’t know what she was doing was wrong, but the prosecution noted that she cautioned staffers not to email about it. The jury took a day to return a verdict. She faces up to 20 years in prison for one count of fraud and one count of falsifying records.

Former CEO Greg Reyes is still awaiting sentencing on his conviction for 10 felony charges. Jensen’s conviction isn’t good news for him.

Brocade’s marketing nailed by the Wall Street Journal’s Law Blog
On a lighter note, the WSJ Law Blog had this comment on Brocade’s marketing:

Law Blog Corporate Self-Description of the Day: So we’ve done a little bit of reporting on backdating at technology companies, and we have to admit: for the most part, we have no understanding of what it is these companies do. We took a look at the Brocade Web site to gain some understanding. Here’s what we learned:

“Brocade provides key building-blocks for architecting and simplifying IT infrastructures to increase resource utilization, improve productivity, and maximize ROI.”

You can bet we’ll be dropping that into our holiday cocktail-party chat ASAP.

Much classier than shouting “party’s over!”

Seriously
What does Brocade do? Yeah, they build FC switches. But with 10 gigE coming in, and Fibre Channel over Ethernet (FCoE) coming in, that won’t be a growth business. Do you really want to compete with Cisco and Juniper in the Ethernet switch business?

How about storage connectivity? Maybe. Like QLogic, who took an enormous hit when Wall Street realized FC was over, Brocade is grappling with the post-FC world. More on that later.

The problem with Brocade’s self-description is that it could apply to dozens of technology firms - and does. Faster, better, cheaper is what everyone in Silicon Valley does. Working some of the “how” into it makes for a better and more memorable story.

And differentiates you from everyone else who claims the same thing.

Comments welcome. If you start seeing law firm ads here, that is Google’s doing, not mine.

Not “knowledge is power” or “information is power.” If you can’t store it, search it and retrieve it, you’ve got bupkis, friend.

Massive storage is a double-edged sword
And we’ll be forever in sorting it out. Cases in point from the Volokh Conspiracy a legal blog:

• A gang of bank robbers used text messaging to plan their crimes. The prosecution subpoenaed the content of their text messages from the service provider, who evidently keeps them all. The defense says that’s wrong: text messages are speech and therefore need a warrant, not a subpoena, to access. Are text messages records or speech?
• A North Carolina judge and candidate for re-election has evidently had a YouTube video depicting him in ethically questionable behavior pulled. Should politicians be able to hide such information from the public?
• Should the government be able to subpoena Amazon for the customer records of a merchant believed to be evading taxes? The prosecutor, judge and the poster seem to be out of line: surely there are other ways of tracking Internet-derived income - like credit card or PayPal payments to the merchants. Why involve the buyers at all?

The StorageMojo take
It is tempting to think of massive storage as culturally neutral, since it is only storing what people produce. But just as the printing press helped broaden literacy and fueled the scientific revolution of the 17th century massive storage broadens access to information in several ways.

• As Gordon Bell is showing, we will soon be able to record every waking moment of every person’s life. How should that data be used, and who should use it?
• Massive storage enables scientific advances that use statistics to tease out the truth. Like Partial Response, Maximum Likelihood (PRML) and those CERN shots, is reality merely probable?
• The courts will soon twig to the fact that it is cheaper for companies to keep all their electronic data than it is to keep all the paper that has been required for many decades. Highly intelligent search will be required to make sense of it all. “Corporate responsibility” will take on a whole new meaning.

Comments welcome.

The China syndrome pt. II
According to Engadget some Maxtor-branded Seagate drives shipped with a handy little virus:

. . . drives produced by a company sub-contract manufacturer located in China were reportedly sent out with the Virus.Win32.AutoRun.ah program already loaded. Apparently, the molar virus is one that get its kicks by searching for passwords to online games (World of Warcraft included) and sending them back to a “server located in China,” and as if that wasn’t enough, it can also disable virus detection software and delete other molar viruses without breaking a sweat.

So many questions
So what would be different if Seagate was Chinese-owned (see The China syndrome)? I suppose it would be easier to build viruses into the firmware. Array vendors would be likely to see them, but would commodity-based cluster storage have any way to catch them?

What if the virus waited to engage until the drive had 7,000 hours of use? Even array vendors wouldn’t see that during integration.

The StorageMojo take
We can scare ourselves silly thinking about how the Chinese government could use disk drives to ferret out secrets. Ultimately though, any such data has to go through servers and networks to reach the outside world. Scanning outgoing data is the only way to protect against such espionage, be it human or virus based.

Where would that scanning take place? In a router? And where is code developed for routers? Some, at least, in China.

If the Chinese made a $30 billion investment in Seagate they’d have to weigh the short term advantage of surreptitious data gathering against the virtually 100% chance they’d get caught. The impact on their investment and their world image would be huge, especially in all the 3rd world countries that would have no idea how badly they’d been compromised.

Disk-based espionage seems highly unlikely. Router-based espionage seems much more likely.

Comments welcome, of course.

What is Cantonese for “sucks”?
A piece in the UK web site The Register, says of some recent Mac drives:

According to Retrodata, its customers have sent in a much higher number of failed Seagate 2.5in SATA drives made in China and loaded with firmware version 7.01 than of any other current hard drive model.

“We’re getting 20-30 times more failed drives of this kind than others,” Retrodata chief Duncan Clarke told Register Hardware. . . .

Clarke blamed the problem what he described as “poor quality control in Chinese hard drive factories” - an issue he maintained affects other hard drive makers in addition to Seagate. He also warned all hard drive buyers to avoid HDDs manufactured in China.

The fact is that the Chinese already make hard drives and their quality, as as far as I can tell, is fine. Mr Clarke just got lucky with some PR on a slow news day.

Two data points don’t make a trend
And yet just a few weeks ago Seagate CEO Bill Watkins was raising security concerns about Chinese ownership of Seagate. No bid surfaced, leaving me to wonder what game Mr. Watkins was really playing.

Are the security concerns real?
The NYTimes reported that

“Seagate would be extremely sensitive,” said an industry executive who participates in classified government advisory groups. “I do not think anyone in the U.S. wants the Chinese to have access to the controller chips for a disk drive. One never knows what the Chinese could do to instrument the drive.”

Given that the Chinese already build disk drives - Excelstor was the only drive company with no online complaints the last time I looked - and that Seagate already has two large plants on China’s southeast coast in the cities of Wuxi and Suzhou, near Shanghai, I’m not getting the concern. What, exactly, do the Chinese not have access today that buying Seagate would give them?

Of course, we know all about planting logic bombs
The US used software against the Russians in the 1980s.

But disk drives?

Disks aren’t free-standing network connected devices. They communicate to the world through drivers and HBAs and controller chips. Even if you wanted to it isn’t clear to me how disks could become a serious security threat. If they could, why would you start with disks? There are better targets.

Now let’s take a look at routers
A router would seem like a much more likely device to use for spying. Network connected, all the data passing through it, millions of lines of code in high-end routers.

Cisco already does R&D in China. Should we worry about that? Where are the unnamed “security experts” opining about that?

The StorageMojo take
The military-industrial complex that Eisenhower warned us about is bigger, badder and dumber than ever. If the clown in the White House actually cared about US security he’d have avoided selling hundreds of billions in T-bills to the Chinese in the first place. But that would have required discipline and accountability instead of strutting around in a flight suit. Where’s the photo op in a balanced budget?

The China bashing should stop. Like it or not, the world’s largest country and the world’s largest economy are now locked in a long term dance. We need them to finance deficits as far as the eye can see and they need us to keep their economy growing at a breakneck pace.

By wrapping Seagate in a “national security” banner, Watkins has done the US and his shareholders a grave disservice. We need closer economic ties to China and turning $30 billion or so of T-bills into cash for Seagate stockholders and a Chinese stake in the health of the US economy would help both countries better enjoy the long dance we’ll have in the 21st century.

Comments welcome, of course. How would you instrument a disk drive for nefarious purposes?

Our broken patent system
RIM recently settled a patent case for $600 million with NTP while the validity of the patents was still under review. The threat of Blackberry shutdown forced their hand.

More recently NetApp filed suit against Sun in another Federal District Court over ZFS. They went to a court in Texas, even though both companies are headquartered a few miles from each other in Silicon Valley.

This court has seen a 10-fold increase in patent suits since 1999. What’s with the United States District Court for the Eastern District of Texas?

First, where the heck is the fabled Eastern District?

That’s it in the middle of the map.

Wake up call
The RIM/NTP case was a wake up call for the high tech industry. While RIM’s lawyers and RIM itself deserve major blame for the debacle, the idea that a business could be shut down due to possibly invalid patents galvanized the industry.

Patents are a good thing
If an inventor comes up with something smart he should be able to profit by it even though large and wealthy companies bring it to market. But should a patent holder be able to clear $600 million because prior art wasn’t considered and a review wasn’t timely?

Several problems

• Federal courts and the Board of Patent Appeals and Interferences (BPAI) aren’t on the same page. They each have their own calendars and processes and laws. So even though the BPAI fast-tracked the RIM appeals, they weren’t fast enough to rule before RIM was forced to settle. If RIM hadn’t angered the judge by their earlier conduct things might have different, but so it goes.
• Patent examiners need help. Patents should be novel and non-obvious, but how do you know? RIM introduced evidence that the NTP patents were invalid due to prior art, i.e. weren’t novel or non-obvious, but why didn’t the examiner catch that before granting the patent?
• The Eastern District of the 5th Circuit Court. Federal judges have a lot of power over what happens in their courtrooms and judges like T. John Ward have turned their courts into preferred venues by forcing fast discovery and quick trials. Justice delayed is justice denied, so speedy civil suits are a public good. Yet reports that the good citizens of the Eastern District sometimes discount technical testimony about prior art in favor of a “they got the patent, so give them the money” attitude suggests another problem.

And another strategy.

How about marketing the benefits of innovation?
I got to thinking about this when some guy went ballistic on the Sun CEO’s blog over the suggestion that the Eastern District is a preferred venue for patent trolls. Jonathan didn’t say anything derogatory about the district or its people, but that didn’t stop the rant.

A little defensive perhaps?

I have no opinion about the people of eastern Texas. It is an area of small towns, adjacent to Louisiana and east of Dallas and Houston. I’ve seen reports that it has an older-than-average population which suggests it could have lower-than-average education levels. While the jury system isn’t perfect, it is an important check on government power.

The 3rd prong
Industry is already after Congress to change the laws to reduce some of the draconian provisions of current law, a version of which just passed the House. Farm-state senators need support for yet another huge agri-biz subsidy program, so the high-tech states have some leverage.

The GAO just issued a new report on problems in the patent office, and the USPTO is responding. The Peer Reviewed Prior Art Pilot is a pilot project to make it easier for interested and competent parties to contribute to the discussion on prior art during patent exams.

The legal challenge is going well, too. The Supremes just ruled in KSR v Teleflex that the non-obviousness is a legal question, not a factual one, which has implications for the appeals process. It isn’t clear how KSR will play out, but in the short term it almost certainly reduces the value of existing patents.

But what about Texas?
Given the amount of money at risk in the Eastern District, I propose that an industry association form to educate the people about the patent system. There are lots of small towns and small town papers and radio stations that could be briefed. Advertising could be bought and opinion leaders consulted. Unlike major metro areas, the Eastern District would be fairly cheap to reach.

This is just another marketing campaign: focus groups; issue analysis; target demographics; messaging and the like. A few million dollars could go a long way towards making the Eastern District jurors more discerning and critical on patent-law issues.

Comments welcome. Tech companies don’t seem to believe in marketing except when all else fails. Why?

Also, I didn’t say a word about software patents. The US is the only industrialized nation that patents software. Smart or dumb?

I’m sure Tucci would have preferred another topic
I’m not going to talk about the strip clubs, showgirls painted with EMC’s initials - “Mr. Fredrickson [EMC spokesman] says the initials EMC were on the showgirls’ thighs, not their derrières” - or the whipped cream. If you want to revel in the seamy underbelly of EMC culture you’ll have to buy a Rupert Murdoch newspaper. StorageMojo has standards.

I’d give you a link but it is subscription only
EMC is being sued by several former sales reps who claim gender-based discrimination. The article and the suit give some interesting insights into how EMC works.

First and foremost it is about the money.

The Liberace effect
From the WSJ:

Its top salespeople are well paid. In 2004, the 370 people in the top third of its sales force earned about $330,000 a year, on average, according to documents in the Chicago case. Several of the best-paid ones are women, the company spokesman says.

But they work for their money:

Salesmen called their best customers daily, gave them small gifts and sent them expensive bottles of wine when they dined out with their wives. Sales reps were expected to spend evenings dining with clients and weekends golfing with them.

As I noted in an earlier post:

“One of the holy books of the Mammon-worshipping Church of the MBA is Adam Smith’s Wealth of Nations. One of the things I like about Smith is his intellectual ju-jitsu on the issue of the essential amorality of markets.

“Sure, Smith says, the people buying and selling may be greedy shysters with no more social conscience than a fruit fly, but in aggregate, they provide a valuable service to society. Which explains why we have to put up with Donald Trump and Paris Hilton. Smith’s concept: the “invisible hand” of the marketplace, through more-efficient resource allocation and the benefits of comparative advantage, makes society wealthier, which is a Good Thing.

“Smith is, in the main, correct. Better investment decisions, more efficient markets and free trade are beneficial, even though some people get whacked pretty hard in the process. Yet “better than the alternatives” is not to be confused with “perfect”. The direct sales model for enterprise storage is a case in point.

So why won’t they sell me cheap storage?
“When I was a newly hatched sales rep way back when, I quickly figured out that a customer had to have a $50k budget to make a sales call worth my while. My annual sales quota worked out to $17,000 per day. 10% of my calls turned into sales, so each call had to offer at least $50k to be worth a trip.

Liberace, didn’t he work with Bernoulli?
Liberace was a popular, and very gay, American entertainer. He’d walk out for his Las Vegas shows in some outrageously over-the-top rhinestone studded costume and, as described by David Hickey in his book Air Guitar

. . . do a runway turn, and invite the audience to “Hey, look me over!” Then flinging his arms upward in a fountain gesture, like a demented Polish-Italian diva, he would shoot his hip, wink, and squeal, “I hope ya’ like! You paid for it!”

Which gets us to the cheap storage problem
“I hope you like all the salesmen who visit you because, like Liberace’s fans, you’re paying for them. Storage that is too cheap can’t be sold the way most enterprise equipment is sold because the margin dollars won’t support it.”

The StorageMojo take
Techies tend to both denigrate sales people and underestimate the power of a good sales force. When I joined Sun in 1995 they had a great sales force. If you gave them the tools they’d break down walls to sell the product.

EMC rarely has market leading products. Their reliance on checkbook innovation means they often have complex and barely integrated products. Which is why I find it odd that EMC’s sales force is only 13% female: you’d think that cutting out almost half the talent pool would hurt.

But the goal of a storage company is not data protection. Storage companies sell customer protection and the EMC sales force has traditionally made customers feel safe. Customers buy feelings, not products.

Update: a friend’s take
A friend of mine who once worked at EMC wrote in with her opinion. She’s a smart, tall, slim and athletic blonde with 15 years in IT sales and marketing.

The EMC news reminded me of the sales meeting where I was told to wear pants and lock myself in my hotel room after 10:00pm. I knew the sales guys well, and did as I was told. . . . [M]y best training for life was growing up with two brothers. If the woman complains a guy took away some of her accounts, did she try to steal some back, steal his best one away, or steal from another person? Did she try to get revenue booked under her name at the end of the quarter . . . ? In other words, did she fight back? It was never a secret that most of the sales guys in the era mentioned were BC thugs – if a person did not pick up on that during due diligence or interviews, then they were blind. If a guy pinches my butt, I pinch him back or give him a swift elbow in the ribs depending on the message I want to send.

. . . [I]f you want to swim with the sharks, you have to act like one.

[BC = Boston College, a local institution of higher learning]

No, she’s not going to pinch you on the butt
But you can dream.

EMC made the most of the 1990’s, but their culture wasn’t the critical success factor. The swashbuckling EMC narrative leaves out IBM’s role: EMC got lucky. They made the most of that luck, but all of their Israeli-tank-driver/BC jock aggression would have meant little if IBM had fielded competitive products (see “Daddy, tell me again how little EMC beat giant IBM . . ..

[Gee, did I write that 2.5 years ago? Whoa!] The still-unanswered question: can little EMC grow up?

Update II: See Joe Tucci’s response to the WSJ article here.

Comments welcome, of course.

The Wall Street Journal (subscription required) reports on a federal whistle-blower suit that shines light on a high-tech industry practice that I’ll bet most customers don’t know about.

Here’s the deal.

Tech companies sign agreements to cooperate with major integrators such as Accenture or IBM Global Services. These deals include lots of things that you’d hope they would, like expedited access to core engineering resources, volume discounts, national coordination and local support. All good.

Can “objectivity” be bought?
The manufacturers also offer incentive payments for things like replacing a competitor’s equipment with their own. Vendors also offer rebates directly to the integrators based on total purchases.

In one instance cited in the WSJ article, Sun offers members of its “Government Alliance Program” a 10% rebate on purchases for a “competitive knockout” and a 2% rebate on total purchases. Other vendors have similar programs.

Two different issues
The Justice Department is suing several integrators because they believe the practices constitute illegal kickbacks. A judge and jury, and perhaps several years of appeals, will decide that.

As a taxpayer, I’d like to know that the world’s largest consumer of IT equipment, the US Government, is getting volume pricing on the $150 billion it spends each year. The integrators and vendors say the practices are legal and they may be correct.

The issue for private companies is a little different. You hire an integrator because you don’t have the in-house expertise to manage a big project. You rely on the integrator to provide a solid solution at a good price.

So the big question for you is: do the vendor payments directly to the integrator affect the integrator’s objectivity?

The lesser question is: am I overpaying for this solution so the integrator can earn more from the vendor?

The StorageMojo take
These disclosures should and probably will lead to some changes. Integrators typically are on the hook for performance and maintenance of the systems they install, so they have no incentive to knowingly buy something stupid. But in a world where Windows servers and low-end arrays are commodities- a 10% rebate might affect vendor choice at the margin. There must be some reason for offering the rebates besides charity.

But shouldn’t that rebate get passed on to the customer?

Whether the rebates are legal or ethical, the optics are iffy. Disclosure of these payments, or their elimination, is the right path for vendors, integrators and customers.

Comments welcome, as always.

Pardon me while I comment on blogging
I don’t find most commentary on blogging to be worth spit, especially the “end of civilization as we know it” nonsense of Andrew Keen.

Blogging is a new media with low barriers to entry. It satisfies a basic human desire to have one’s say. It isn’t so different than the advent of cheap pulp paper and web presses in the 1800’s or the growth of radio in the 1920’s: suddenly many voices that were not heard before had an opportunity to communicate.

This is discomfiting to the folks who already have their say and disturbing to those who don’t want to hear anything new. But we’ve had a regular stream of new media for the last 250 years and are sure to have more, so get used to it.

A lot of what gets said by the newbies is stupid. So what. Have you looked at the Wall Street Journal’s editorial page lately?

So what about storage blogging
Last month the Storage Anarchist, a blog written by EMC employee Barry Burke, made some interesting observations about storage blogging by employees of storage companies. This is his comment on tutorial blogs.

Written in the style of a print-media advertorial (a contraction of “editorial advertising,” where ad space is purchased to run an editorial-like assessment of a product or service), the blogutorial is emerging as a new, and so-far unchecked marketing tool. Importantly, truth-in-advertising laws require published advertorials to be plainly marked as “paid advertisements” (usually in tiny font in the header where you might not notice it). Apparently this isn’t so for blogketing (”blog marketing”)- even when done by official corporate spokespersons in the public forum of the Internet. Apparently the blog-world is unfettered by truth-in-advertising rules.

This concerns me, because it allows bloggers to act like the snake-oil vendors of the past, getting away unchecked with unsubstantiated claims and misleading inferences of undeliverable capabilities.

Strong stuff - and wrong in so many ways
My, my, you innocently go to a company website only to be bombarded with hype, fluff and spin. Who knew? The reason you don’t see any “advertorial” tags on company web pages is because the entire site is an advertorial. Same goes for company blogs. How hard it that?

The truth: everyone “accentuates the positive.”
I’ve worked for several storage companies and observed many others. I’ve yet to see a product announcement that did not shade the truth.

Features planned for completion by first customer ship, but don’t make it. Customer NDA’s are a field of dreams. Even the power use asserted on the box is wrong.

We measure disk capacity in powers of 10’s while RAM is measured in powers of 2, leaving legions of consumers to wonder where the GB’s went. Disk MTBFs and AFRs that are never seen in the field - and that array vendors won’t own up to. Array performance specs that bear no relationship to any known application.

Corporate bloggers are a Good Thing!
Any buyer whose brain is not in a jar knows this. So where do corporate bloggers fit into this fulsome stew?

• Corporate bloggers provide valuable information even when they are trying not to. The attitude of the people inside the company towards their customers comes through in the blogs. It doesn’t matter if the blogger is an officer or a hack detailed to blog duty. Bloggers who talk down in their blogs, or who are rude or dismissive of valid customer concerns, tell us how the top management of a company views customers and competitors.
• Even unofficial company bloggers, like Barry, reveal more than they know. Their comments reflect their corporate culture, be it combative or comatose.
• Blogging is a conversation. Maybe not a direct one via comments, but everyone gets to comment on everyone else’s blogs, just as I am today. Rather than get hung up on political correctness I’d rather people called ‘em as they see ‘em.
• Bloggers are people. Most are nice. Some are scum. Some are goofy. Some are smart. The great thing about the Internet is that no one knows you are a dog. That is, as I noted in an early email to another blogger, don’t worry about who I am or where I come from. Just look at my ideas. Do they make sense?

Ironically, Storage Anarchist is arguing for a world where “official” bloggers are controlled and “unofficial” but highly partisan bloggers aren’t. Why we’d want this eludes me. Let a thousand flowers bloom!

Storage is too important for anything less than full and frank communication
Do we really want an industry where every communication by insiders passes through the corporate-speak homogenizer? Hell, NO!

Storage is an important industry. We are responsible for the retention and protection of an ever growing fraction of the civilization’s information. We exist at the juncture of several trends: in disks; in laws; in networks; in architectures; in software. There are a lot of choices to be made, a lot of ideas to be examined.

Smart people are entitled to their biases, too
I like hearing what Hu Yoshida and Dave Hitz have to say about storage. Not because I agree, but because they know the business, they’re smart, they’re informed and they are readable. They try to tackle some of the deeper issues. They do it from their perspective. The clash of perspectives is what sharpens our thinking.

Am I disappointed that they don’t say negative things about their employer? Not at all. That’s my job. If their posts incidentally buff the image of their employers as savvy storage players, that is all to the good. Hey, it could even be true!

The StorageMojo take
We’ve seen what happens on a national level when debate is stifled by manipulative ideologues: bad decisions. It isn’t good for America and, in our lesser sphere, it isn’t good for storage. If Hu wants to present information and commentary that makes HDS look good, and other people want to poke holes in it, fine.

What I reject is a viewpoint that makes people wrong for voicing opinions or presenting their view of why what their company is doing is a Good Thing. That is where Mr. Anarchist loses me. I think we need more commentary from storage people, not less. Waving phony legalisms like “advertorials” on company websites is designed to shut down discussion, not encourage it.

I’d like to see more storage users commenting about the industry as well. Storage is ultimately the hardest problem in computing because it is persistently anti-entropic. Let’s not make it harder by shutting down communication from those who are closest to the problems.

Comments welcome, as always. Yes, even you EMC guys. Just don’t expect me to roll over.

Found guilty on all 10 counts
Greg Reyes, former CEO of Brocade, was convicted today of all 10 counts he was charged with of criminal securities fraud for backdating stock options and lying about it in a San Francisco courtroom. He faces 20 years in prison.

Mr. Reyes made some $380 million dollars off Brocade during the dot com boom. What investors didn’t know is that if he had followed the proper accounting rules Brocade’s $67 million FY2000 profit would have been a $950 million loss, at least on paper. Options are a non-cash expense, but so are a lot of other things that show up on income statements.

None of the backdated options went to Mr. Reyes
His defense claimed that he was a sales guy and didn’t understand accounting enough to know that backdating options and falsifying board minutes was a no-no. But the “everyone was doing what I didn’t understand” defense failed to persuade the jury.

As I noted last July

I’m no lawyer, but given that Mr. Reyes sold $380 million of Brocade stock while investors believed the company was profitable, maybe the hope of “enrichment” clouded the man’s judgment.

Evidently a similar thought occurred to the jury.

The StorageMojo take
This has no impact on the Brocade of today, other than their culture is a direct descendent of the company that Mr. Reyes built. Like EMC, Brocade was a sales-focused culture with a “whatever it takes” mentality. They achieved fast growth for a time but are floundering because they handed their future over to storage OEMs who could care less if Brocade lives or dies. Their strategy is in worse disarray than EMC’s while their core fibre channel business is starting to decline.

I hope they can turn it around, but I’m more than dubious. Most of the world doesn’t need fibre channel and there are better places to buy Ethernet and Infiniband.

Comments welcome, as always. If Brocade is stronger than I know, please elucidate.

Don’t read the “EMC has Ph.Ds?” series!
Or at the very least avoid reading the patents I reference. I may have already said too much.

Microsoft’s FUD-slinging on Linux shines some light on patents
Good article with some great quotes on Cnet (disclosure: I’m a paid blogger for Cnet sub ZDnet and this is probably the last time I’ll mention it). So you don’t have to read the article here are my favorite quotes:

“The fear of willfulness is so great that often firms instruct their engineers not to look at patents.”
–Matthew Schruers, senior counsel
Computing and Communications Industry Association

You reduce your legal liability by not reading patents, because if you can show you invented it independently it increases the strength of the “obviousness” defense.

Linus speaks

“There are several reasons why engineers should not read other people’s patents, only their own. And it’s not a ‘hide your head in the sand’ issue, it’s a very practical issue of it being a waste of time,” Torvalds said.

. . . And engineers aren’t likely to comprehend patents in the first place: “Unless you have a patent attorney at your side, patent language usually makes no sense.”

I thought it was just me
I’m not out of the woods yet, though. Linus’ last quote in the article is

“The bulk of all patents are crap,” he said. “Spending time reading them is stupid. It’s up to the patent owner to do so, and to enforce them.”

The whole EMC series, a waste. Boo-hoo.

But my many fans in Hopkinton would be devastated if I stopped now, so the show will go on.

Comments welcome, as always. Although I think Linus said it all.

A SWAT team busts open your door, grabs all your computer gear, and arrests you for child pornography
That would be a swell day, eh? Oh, and then you lose your job, your friends shun you and strangers feel free to beat you to death. Or maybe you just decide to end it all. Thirty-nine people did.

All because of identity theft. And self-righteous, computer-illiterate police.

On the internet, no one knows you’re a dog
Actually, according to a recent article in the Guardian

. . . thousands of cases under Operation Ore have been built on the shakiest of foundations - the use of credit card details to sign up for pornography websites. In many cases, the card details were stolen; the sites contained nothing or legal material only; and the people who allegedly signed up to visit the sites never went there.

Among the accused: “. . . musicians Pete Townshend, the Who guitarist, and Robert del Naja of Massive Attack, both falsely accused of accessing child pornography.”

Landslide
You may dimly recall a credit card site that both processed payments and authenticated credit card numbers whose owner is now serving a 180 year term for carrying listings to several overseas child-porn sites as well as thousands of legal adult porn sites. About 250,000 names and card numbers were seized from the site’s computers.

Over 7,000 of those names were British.

Here in the US, the FBI sent emails to people on the list offering child porn. About 100 responded. In the UK, police assumed that the people on the list were all active consumers. People lost jobs, lives, homes, friends and had to spend thousands defending themselves against the allegations.

And I thought medical identity theft was bad
You haven’t heard of medical identity theft? From a report by the World Privacy Forum:

Medical identity theft occurs when someone uses a person’s name and sometimes other parts of their identity — such as insurance information — without the person’s knowledge or consent to obtain medical services or goods, or uses the person’s identity information to make false claims for medical services or goods.

So you can be tagged as someone with HIV, cancer or TB and suddenly denied insurance. You may be dunned for $40,000 worth of surgery you never had. You may be tagged as a drug addict thanks to prescriptions in your name. Or you may be denied employment due to fabricated mental health problems.

Have a nice weekend.

The StorageMojo take
Massive storage is a wonderful thing. Yet we are still in the early stages of understanding how to ensure that information about people is valid, especially financial and medical information. Electronic medical records, which are a valid tool for improving medical care, only make this risk worse. I know IBM and Sun have been doing some work in this area, and I’d be happy to publish information about what other companies are doing as well.

In the meantime, WPF has a number of recommendations for improving patient access to medical records. As for shoddy police work, I’m afraid that will always be with us and innocent people will suffer the consequences.

Comments welcome, as always. I’ve been cranking away on the toughest competitive analysis project of my career for a client and its been eating up my time something fierce. I’ll be getting to the second part of “EMC has Ph.Ds?” next week.