I’m occasionally accused of holding up Google as the storage of the future. That’s wrong: I’ve noted from the first that the Google File System and BigTable aren’t suitable for the enterprise, or even for most other Internet Data Centers. My real point is that the fast growing workloads: large files; cool data; long-term storage – favor the RAID & array free storage style of Google and Amazon. Yet I consider this a hypothesis, not a fact, and I welcome evidence that sheds new light on the issues.

A dissenting opinion from someone who’s doing it, not selling it
So it was with some interest that I read this post from the blog of Don MacAskill, CEO and Chief Geek of SmugMug, a sharp looking photo sharing site.

SmugMug focuses on delivering a quality product and they expect people to pay for it. Don says the company is debt free and profitable, and with 200,000 users paying a minimum of $40 a year, I believe it. No free accounts. If you want SmugMug you pay for it.

Storage is front and center in the value prop
“Unlimited photos” is the bold claim on the well-designed home page, and at the Learn more page the first headline is “Like Fort Knox for your photos.” Followed by:

Unlimited storage.

Safe & secure.

SmugMug keeps 4 backup copies of each photo in 3 states. Retrieve them anytime.

Lock sensitive galleries with passwords.

Those are strong, storage-centered promises. Naturally, inquiring storage minds want to know: EMC; or Google?

Or door #3: neither
Here’s what Don says, formatted to fit your screen:

• External DAS for the database servers. Don likes dual-controller arrays because it simplifies recovery in case of server death and the dual-controllers give reliable access. SAS attach.
• Spindle love. Typical array has 14.
• No parity RAID. RAID 1+0.
• 15k drive love. Speed is good.
• Love drive enclosures with odd numbers of drives. Makes keeping one hot spare easy.
• Love big battery-backed up write caches in write-back mode. Why? Because super-fast writes are “. . . easily the hardest thing in a DB to scale.”
• Disable array read caching: array caches are small compared to the 32 GB of RAM in the servers. Don wants all array cache for writes.
• Disable array prefetching: the database knows better than the array.
• Love configurable stripe and chunk sizes. 1 MB+ is good.

So what does all this storage get put on? Don says

. . . our typical DB class machine is a Sun X2200 M2 with two dual-core CPUs and 32GB of RAM. The RAM and the disk stuff I talk about above are far far more important for our workload than the # or speed of CPUs . . . .

Is it easy to get all those features, Don?
Turns out, no, it isn’t. Don reports he usually has to compromise on one or two of the items. He thinks they’re getting closer to the perfect array, but they aren’t there yet. He doesn’t say who he buys, except that it isn’t Sun. Which is typical of Sun customers.

The StorageMojo take
Don should be the ideal array customer: fanatical about protection; lots of data; heavy workload, not afraid to spend money. Yet he isn’t completely satisfied, let alone delighted, by what’s out there. A lot of the engineering that goes into arrays is wasted on him, so he’s paying for a lot of stuff he’ll never use, like parity RAID, pre-fetch and read caching.

SmugMug is a point on the curve of evolution away from what has long been considered required in a storage array, towards what delivers performance and availability in today’s world of cheap RAM, cheap disk and large files. Don spends money on what matters to him and SmugMug. Shouldn’t array vendors do so as well?

Update: Don wrote in the comments SmugMug also uses over 200 TB of Amazon’s S3, which he talks about here. I’ve got to spend some more time reading his blog!

Comments welcome, of course. Moderation turned on to protect the innocent from the guilty.