As promised, Martin Green has provided more info on Cnet’s Haystack storage system. Hardly a word about the software, which is where the magic is, but this quote should give the big iron vendors a little shiver.
BSU Haystack consists of many Basic Storage Units (BSU), which are just servers with a lot of disks. Content is scattered more or less randomly across all BSU and spindles to maximize the IO throughput of the system. Multiple copies of the content are maintained on disparate equipment so that no single failure can loose [sic] all copies of an object.
Bottom line: a profit making enterprise, serving over 19,000,000 million users, storage dependent, not using RAID arrays. Granted, photo sharing apps aren’t Oracle databases, but as Martin notes, data cools as it ages, so I/O needs lessen. If the SEC has its way, companies will end up saving everything forever anyway.
Martin, can you squeeze a few words about the software out of the team? Thanks!
I’ll chat with them tomorrow and see what we can come up with. Was what Paul and I wrote informative at all, interesting or relatively predictable? love your thoughts.
Haystack is not MogileFS but it is similar. We feel Haystack can scale better than MogileFS for our purposes. It is proprietary software but it is CNET proprietary software, and it is purposebuilt for the kind of content we’ll store and serve from it.