It gushes money
Gartner’s business model is genius. They gather information from vendors and users – for large fees from both – and then sell that information back to them for even more money. Bliss.

They own a toll booth on the user/vendor information highway. And collect $1.3 billion a year from the traffic – over $300,000 per employee. Drool.

But the best is the Magic Quadrant, Gartner’s money-spinning qualitative graphic. You’ve seen it, but here’s a blank version.

mq_graphic

The “magic” is how it gets tech execs leaping, like spawning salmon, for the upper right corner.

The little engine that couldn’t
Everyone toes Gartner’s line. Until now.

Beth Pariseau posted on ZL Technologies’ suit against Gartner . ZLT hopes to force Gartner to provide:

  • Fair Disclosure on Conflicts of Interest – Gartner generates its revenues from payments made by the same vendors whose products it evaluates. Similar to the new rules now being imposed on financial ratings agencies on Wall Street, Gartner should be required to disclose the revenues received from the vendors it ranks.
  • Fair Disclosure on Evaluation Scores – The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores so vendors know exactly where they are lacking and by how much and take corrective action. Currently, there is zero disclosure, which can lead to arbitrary placement, with no recourse and no basis for appeal.
  • Better Oversight – Gartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. There is a crying need to establish an impartial ombudsman similar to those found in public media, in order to ensure purchasers that they are receiving impartial analysis.

Sour grapes?
In its court filings ZLT talks about the harm it suffers caused by customer reliance on the MQ. I’m not surprised.

As Geoffrey Moore noted in Inside the Tornado IT managers are herd animals. They know there is safety in numbers – in keeping their jobs and getting problems solved – so they like to stick together.

The power of the MQ is that it captures this mentality and gives it a graphic form that comforts even the most technophobic CFO/CEO in a few seconds. IT may not have the foggiest notion about the firm’s 5 year requirements or what implementation will entail, but by going with the big guy they have an alibi and an escape plan all in one.

MQ criteria
Gartner’s lumps all the MQ verbiage under 2 headings:

  • Ability to execute
  • Completeness of vision

Ability to execute
Ability to execute favors the large. If you’ve got 5,000 engineers and free cash flow you can, eventually, execute anything. Never mind that a team with 10 smart engineers and a clear vision will move faster and smarter to solve a particular problem: tiny CDC built the first supercomputer, not IBM.

Many of the criteria explicitly favor size: global presence in large markets; “viability;” market share; marketing and sales to drive acceptance; and more. You’re a smaller company? Tough. It’s Gartner’s quadrant and you may not live in it.

Completeness of vision
The large company bias here is the way markets are defined. For example, the mid to high-end NAS MQ excludes vendors whose focus, today, is not “primary file systems storage, instead of storage that narrowly targets backup or archive data.”

But the available research finds that most enterprise files are created and accessed only a few times. What then is the difference between “primary” and “archive” storage? Shouldn’t Gartner raise their enterprise customer’s awareness of this and other file storage issues?

Gartner analysts read the research. Why doesn’t the MQ reflect the best information in major use cases, instead of reinforcing popular prejudice? Doesn’t Moore’s Law steadily move the bar upward for what “narrowly” focused systems can do?

Mid to high-end NAS that includes both scale-up and scale-out – as Gartner’s market definition does – is lumping 2 very different markets together and blurring the distinctions. Rather than accepting vendor market definitions, Gartner analysts should be in the forefront of defining new market segments.

For $100,000+ per year a CIO should expect no less.

The StorageMojo take
The Magic Quadrant has the analytical rigor of a beauty contest. Implicit and explicit assumptions about customers, markets, technologies, use cases and suppliers obscures more than it reveals. The MQ seeks to rank vendors not only by what their products do, but by what Gartner presumes an enterprise customer should want. They presume too much.

Marketers often disagree about what constitutes a “product:” is it the widget itself; the widget + services; or the widget + services + ?? In truth, customer perception of what constitutes a product changes with time and experience. But Gartner is stuck: if someone has a better widget – as ZLT says it does – there is no way that the MQ will tell you that.

Enterprise IT staffs abhor change, so Gartner could argue they are meeting customer needs by rigging the MQ to favor incumbents. But the current crisis and the need for greater efficiency and cost-effectiveness means decision makers need better data and informed opinion.

If ZLT, for example, can search archived emails 1,000x faster than Symantec – as they claim – then Gartner should disclose that for the customers for whom performance is important. Get feedback from actual customers – what Gartner does today – about ZLT and pass it on. Don’t just ding them because they’re small.

Customers aren’t idiots; they can see that a company isn’t very big. What they don’t know is how well their products work.

Gartner needs to start earning that $1.3 billion, not just collecting it. If the FTC can require lowly bloggers to report vendor freebies and payments, perhaps the day isn’t far off when mighty IT consulting shops will have to do likewise. Kudos to ZLT for noting the emperor’s scanty attire.

Courteous comments welcome, of course.