This came in over the transom from a semiconductor engineer. He’s wants home archive storage and is wondering why no one seems to sell it. I’ve been grappling with the same issue.

Here’s an edited-for-length excerpt from his letter:

I use RAID server products from Netgear and QNAP and have been searching for my ideal server product. I don’t understand why it doesn’t exist. My hunt is for a server with integrated error checking to ensure that bit errors can be caught and rectified. My goal is a system that secures the integrity of the files stored upon it. As far as I am aware this kind of functionality does not exist and isn’t discussed anywhere.

For example, once I have a video file (which can be the result of many hours of editing) it isn’t subsequently modified, just read on occasion as required. I don’t want any bit errors on this file – every change is just a corruption. I backup my files, I just want to make sure that what I am backing up is the same as when it was first written.

I don’t want to run a piece of software over the network, I want the server to run a check and fix any errors that it finds. It is something that I would gladly pay a premium for. I think there is a market for this as there must be a lot of users who have files that never change.

Frustrated in California

Dear Frustrated
You’ve identified the key problem: RAID systems aren’t for archives. RAID keeps your data available after a disk failure – sometimes 2 disk failures – but they do not ensure long-term data integrity. Or even short-term integrity. Not their thing.

This is what archives – traditionally tape – are for.

But tape is a tough sell to the home & SOHO market. Low-end drives – DAT, SVR, VXA – cost several hundred dollars plus the tapes. DLT/LTO drives start around $1200 with $40 tapes.

You can buy an external Blu-ray burner for those prices and 50 GB media for a few bucks each. On sale BR media is starting to reach the 5¢/GB level of 2 TB drives, and the longevity should be better.

But I have over 500 GB of video alone. Shuffling 10 or more BR media – 20 if I’m paranoid – reminds me of floppy backups. Yuck.

The current plan:

  1. Create zip archives of files and folders I want to preserve.
  2. Back them up to 2 local hard drives.
  3. And ship them off to my online backup provider.

What I don’t know is how robust zip archives are. There is a 32-bit CRC, but what does that do for a 10 GB folder of PDFs?

Also, I wonder about the advisability of zipping compressed formats such video and audio files. It might be worth the computational overhead and the possible larger files if the zip file is robust.

The StorageMojo take
Frustrated isn’t the first home user to want an archive and he won’t be the last. Hundreds of millions of home users will see the need over the next decade.

The question is whether or not someone can design a commercially viable system for home and SOHO use. It is obvious that drive vendors have the cost advantage, especially with the advent of easy and cheap USB drive docks, if they build a disk drive designed for that purpose.

An archive drive can be slower – 4200 or even 3600 RPM – and less dense. Optimized for large transfers. Slower, cheaper actuators and drive electronics.

Single platter 2.5″ 7mm drives could be the sweet spot: minimal head cost; slim cartridge-like form factor; and much faster than optical. Then it is just a matter of getting the volumes up and the costs down.

But that’s just one idea. Please comment on how you would solve the home and SOHO archive problem.

Courteous comments welcome, of course.