This came in over the transom from a semiconductor engineer. He’s wants home archive storage and is wondering why no one seems to sell it. I’ve been grappling with the same issue.
Here’s an edited-for-length excerpt from his letter:
I use RAID server products from Netgear and QNAP and have been searching for my ideal server product. I don’t understand why it doesn’t exist. My hunt is for a server with integrated error checking to ensure that bit errors can be caught and rectified. My goal is a system that secures the integrity of the files stored upon it. As far as I am aware this kind of functionality does not exist and isn’t discussed anywhere.
For example, once I have a video file (which can be the result of many hours of editing) it isn’t subsequently modified, just read on occasion as required. I don’t want any bit errors on this file – every change is just a corruption. I backup my files, I just want to make sure that what I am backing up is the same as when it was first written.
I don’t want to run a piece of software over the network, I want the server to run a check and fix any errors that it finds. It is something that I would gladly pay a premium for. I think there is a market for this as there must be a lot of users who have files that never change.
Frustrated in California
Dear Frustrated
You’ve identified the key problem: RAID systems aren’t for archives. RAID keeps your data available after a disk failure – sometimes 2 disk failures – but they do not ensure long-term data integrity. Or even short-term integrity. Not their thing.
This is what archives – traditionally tape – are for.
But tape is a tough sell to the home & SOHO market. Low-end drives – DAT, SVR, VXA – cost several hundred dollars plus the tapes. DLT/LTO drives start around $1200 with $40 tapes.
You can buy an external Blu-ray burner for those prices and 50 GB media for a few bucks each. On sale BR media is starting to reach the 5¢/GB level of 2 TB drives, and the longevity should be better.
But I have over 500 GB of video alone. Shuffling 10 or more BR media – 20 if I’m paranoid – reminds me of floppy backups. Yuck.
The current plan:
- Create zip archives of files and folders I want to preserve.
- Back them up to 2 local hard drives.
- And ship them off to my online backup provider.
What I don’t know is how robust zip archives are. There is a 32-bit CRC, but what does that do for a 10 GB folder of PDFs?
Also, I wonder about the advisability of zipping compressed formats such video and audio files. It might be worth the computational overhead and the possible larger files if the zip file is robust.
The StorageMojo take
Frustrated isn’t the first home user to want an archive and he won’t be the last. Hundreds of millions of home users will see the need over the next decade.
The question is whether or not someone can design a commercially viable system for home and SOHO use. It is obvious that drive vendors have the cost advantage, especially with the advent of easy and cheap USB drive docks, if they build a disk drive designed for that purpose.
An archive drive can be slower – 4200 or even 3600 RPM – and less dense. Optimized for large transfers. Slower, cheaper actuators and drive electronics.
Single platter 2.5″ 7mm drives could be the sweet spot: minimal head cost; slim cartridge-like form factor; and much faster than optical. Then it is just a matter of getting the volumes up and the costs down.
But that’s just one idea. Please comment on how you would solve the home and SOHO archive problem.
Courteous comments welcome, of course.
I think you’re right about the media. SATA disk drives are really overkill in a lot of ways but they’re the cost/capacity sweet spot. Probably do 2N mirroring, maybe a rotating pair of such. Some sort of archiving software front end–paired with offsite cloud backup. Ideally a file system that was really robust, e.g. ZFS.
That’s a big concern for me too…and I can see only one solution: A server with ZFS.
However, I think that a basic version (motherboard with one CPU and ECC memory, a bunch of 3.5″ disks) is too noisy, takes too much power and costs a bit too much.
Noise can be reduced to arbitrarily low level, but the price goes up.
Power? 2.5″ disks use little, but again, they cost (and error rate=required scrubbing frequency. I heard a recommendation (from DDN?) to scrub 10^-15 BER drives weekly. Scrubbing daily or getting enterpri$e drives? Hmm…).
So making it good would cost certainly too much for my liking, while a cheaper config is not really satisfactory.
I ended up with 2 disks, checksums and scripts that verify them scheduled to run weekly and wait for some better option.
Sounds like multiple ZFS pools would fit nicely.
Wouldn’t something based on ZFS solve the data integrity issues?
I don’t know of a pre-packaged home-market device with ZFS. Leaves you with rolling your own small PC and putting FreeNas or equivalent on it.
Download sites and newsgroups have been solving this problem in software for years. Alongside your data, you provide a method for checking the data’s integrity (md5/sha1 hashes or parity files).
Your backup process would hash the file on disk. If the hash fails, restore from backup (the hash file is backed up as well…thus, this process works even in the unlikely event that the corruption is in the hash file). If you are backing up to a location (such as a local raid array) which is conducive to hashing, you can also hash the backed-up copy to make sure it hasn’t deteriorated (in this case the backed-up hash file can prevent accidentally overwriting a good backup with a deteriorated local copy).
Is this process patented? If not it probably belongs to my employer since I thought it up while sitting at work, but I won’t tell if you won’t 🙂
Hi Robin:
I guess I read the edited letter differently. To me, it sounded like the engineer was looking for a filesystem with Checksumming capabilities. According to the table in wikipedia (http://en.wikipedia.org/wiki/File_system_comparison#Metadata), there are a few filesystems which offer this. Right now, I think the best bet is to use ZFS, with either mirror or raidz disk arrangement and regular scrubs to find/correct corrupted data.. Once BTRFS is ready, that would be another solution.
regards,
Paul
It sounds like ZFS to me. Perhaps Nexenta?
http://www.nexenta.com/corp/
If the OP can’t go with ZFS, then archive files with parity as Nathan suggested could be another option:
http://en.wikipedia.org/wiki/Parchive
http://en.wikipedia.org/wiki/Parity_file
Or you could create both PAR files and have them live on ZFS. If/when you wanted to move them else where you’d get the parity checking for “free” to make sure it arrived at the destination without problems.
Keep a copy of the software on the same file system as the archive files, so as long as you can access the file system, you’ll be able to compile the necessary software (especially if you use a Unix-y/POSIX OS).
To me RAID storage not only protects data from one or two drive failures but must also support intrinsic data protection methods such as CRC, ECC, etc. Any of these can be configured to detect single, dual and more bit corruptions on disk, in cache, memory, or across various interfaces.
Perhaps I work around enterprise storage to much, but if there is a RAID storage system out there that let’s a single or double bit data corruption go undetected – it doesn’t deserve the name.
Regards,
Ray
Frustrated in CA describes a common problem that we recently founded Exablox to address. We’re building storage appliances for SOHO and SMB users that provide archival quality primary storage.
The status-quo trickle down of high end technology into SOHO and SMB leads to the kind of pain that Frustrated is experiencing. Are there solutions for enterprise? Yes. Any chance they’ll hit SMB? Sure, but it’s like waiting for Formula-1 tech to make it into compact cars…
What’s wrong with the status quo?
* RAID helps availability (continuing in the face of disk failure), it doesn’t help with long term data integrity. See http://www.zdnet.com/blog/storage/why-backup-isnt-enough/938
* Tape Is Dead. See http://storagemojo.com/2010/06/16/how-tape-dies/
* Burning CDs/DVDs/Blue-ray is a losing proposition. Not only are they a bad long-term archival medium, but long term management of hundreds or thousands of discs is problematic (which is, after all, one reason we’re ripping all of our movies and music!). See http://en.wikipedia.org/wiki/CD-R#Expected_lifespan
* Software solutions are great … If you’re a storage expert or an IT guy interested in storage, or have a pit crew. Frustrated is a semiconductor engineer, with what sounds like a media creation hobby. Build a server, install storage software, configure, manage, and maintain it? That’s a full time job.
Robin has discussed these problems from many different perspectives. To name a few:
* http://storagemojo.com/2009/09/01/cloud-storage-for-100-a-terabyte/
* http://storagemojo.com/2009/10/11/optical-nearing-the-end-of-the-line/
* http://www.zdnet.com/blog/storage/raid-5-theory-reality/983
His current solution? Zip up the media files, make sure they’re on a few local disks and backed up into the cloud. Conceptually, it’s a good model, but the implementation details leave something to be desired*.
The Exablox solution? We’re writing a new filesystem that is far more robust than even ZFS, with end-to-end integrity checking and disk scrubbing designed to catch and recover from corruption. It will be packaged into beautifully engineered, quiet and low power appliances. Just plug it in, and we take care of the rest.
For the user, we add the ability to set policies to indicate the level of protection desired. The level is tunable anywhere from “immutable, scrubbed often, and replicated to multiple spindles, multiple boxes, multiple off-site locations, and into cloud storage providersâ€, to “I don’t care at all about this data, don’t bother with any special protection.â€
We believe that you shouldn’t have to be a storage expert to protect and share your data.
-Tad Hunt
Co-Founder | CEO | Tech Guy | http://exablox.com
[drop us your email to get announcements]
* Bit rot inside a media file may render a portion of un-viewable or un-listenable, while the rest of the file remains intact. Corrupted zip files may become completely unusable. Plus, most media files are compressed to begin with, so additional compression often causes the files to get bigger.
ZFS technically might be a good solution, but NetApp is creating legal FUD with patent lawsuits. Oracle’s apparent abandonment of OpenSolaris, and multiple ex-Sun engineers jumping ship, are creating technical FUD for ZFS.
2N mirroring does nothing to solve the corrupted block problem, just replicates the corrupted blocks quickly and automatically — the opposite of what you need.
Nathan’s MD5/SHA1 hash suggestions are good, but throughput problems hashing multiple terabytes over USB 2 or SATA/eSATA. Might be doable low-and-slow, better than nothing. However, hashing files just tells you which file is corrupted. How do you fix the corrupted file?
Detecting file or block corruption is relatively easy. Correcting the problem is much harder. If you don’t find the corruption right away, you need very robust tools for self correction, which gobble disk space and performance; or you need to keep a long history of “known good” blocks or files – like ZFS.
Crashplan detects corrupted backup blocks and replaces them from the primary copy. Not quite what the engineer asked for, but getting closer.
Drobo detects failed disks, and rebuilds them semi-automatically. Desirable behavior for the SOHO/SMB market, but needs to get down to the file/block level.
Between the legal and technical issues, this won’t be an easy problem to solve.
I’m using ZFS at home for this – built a home storage server and am running zraid equivalent to raid5. If the chassis I had supported more drives, I might have gone with zraid2.
http://blog.flowbuzz.com/search/label/NAS
I was syncing my local zpool with Amazon’s S3… haven’t done so in so long it’s probably worthless.
Actually, you want a *software* solution for archiving. Any hardware will ultimately fail or become so dated as to be unreadable. Do you have anything to read those old QIC tapes from the 80s?
Creating parity files and then signing and/or encrypting them is the way to go. Then, lots of copies in lots of places. Cloud storage is perfect for this.
I use an Exar BitWackr-equipped NAS box at home with Windows. The BitWackr uses an ASIC to hash and compress data before storing. When blocks are written, they are checksummed. The immediate-read-after-write capability of the ASIC (like that used in tape drives with compression) ensures that the block was correctly compressed (great for archive storage). The hash is stored with the compressed block and is checked when the block is retrieved to protect against on-disk bit-rot. This is not only a response to data integrity issues, but it also dramatically reduces the disk hardware I need to store my data. Compression gives me up to 4:1 data reduction (best case) and deduplication makes up for my lack of attention to inadvertently saving the same file in more than one directory or keeping differently-edited versions of video files (portions of which deduplicate nicely). The product is coming on line in Japan now, but could be built and sold by any domestic storage company. And the best part is that its cheap.
Robin,
I enjoy your blog very much! There’s an initiative sponsored by the Library of Congress and the ASMP for photographers that offers best practices for backing up and archiving digital images (and, to less extent, digital video). The site can be found at http://www.dpbestflow.org/.
One of the people involved with that project, Peter Krogh, a working photographer, has written an excellent, detailed book on how to backup and archive images. I highly recommend “The DAM Book” (O’Reilly 2009).
These materials are of great interest to anyone who does photography. For everyone else – well, there’s some good concepts that are explained well.
Cloud storage is still way too expensive when you’ve got 100 GB of images – or 400 GB. Perhaps one of these days…
Hi
I have a esata drive docking station, I put my hard disks into, since the intel ich8 chipset in my server is not hot-pluggable, I use a litte program called scsiadd. (I use linux Opensuse).
Scsiadd removes and adds the driver device and properly rereads it so you can use harddisks of different partition layouts.
My only concern is to start the server with a drive running in the esata bay, otherwhise my hardware wont detect the /dev/sdX port used.
I’m extremely pleased with this solution, a new disk cost $100 (1.5TB) and due to sata the transfer rate is max of what the disk manages (I get around 50-75MByte/s write). Latency is also superior compared to usb. (an example 1.5mill files 650GB) backs up in 3.5hours on esata, while it takes more than 16hours if I run in USB.
Cheaper and faster home-backup solution does not exist.
Next time I will buy a motherboard with Intel IchXr where X is 9,10,11 because they are hot-pluggable.
You could use Parchives:
http://en.wikipedia.org/wiki/Parchive
It would be a manual process, but say you have a folder of videos you want to protect. You run your PAR software, tell it you want, say 10% redundancy, and it makes parity files to go along with your original files. Later on, you can use those parity files to check for and repair any corruption.
Checksums and hashes can only detect corruption. What use are corrupted files–even if you know they are corrupted? Parchives let you repair that corruption and get your original uncorrupt files back.
Ah this age old problem. I have tried all sorts of solutions over the years. I really want the online cloud solution to take off, but as of right now, internet upload is too slow to make it feasible. I mainly store my files locally on sata drives with important things in two copies while using hashing programs to make sfv files to find corrupt files.
Long term I think I am going to use a solution like unraid http://www.lime-technology.com/ with crc files in the top level directories and use cheap big sata drives as tapes for long term storage.
I really wanted zfs to be the savior of our file problems, but…
1. The potential lawsuits pending
2. The lacking hardware support
3. The fact you have to use bsd or a form or solaris
4. Opensolaris virtually disappearing these last few months
5. The final nail in the coffin: Oracle buying Sun, which may have caused several of 1-4.
And the sad thing is that if zfs would have been released as gpl, it could be incorporated into linux and many of the problems would not have existed, except the potential lawsuit issue.
Why not deploy (the free version of) Castor, a system build by the storage guru’s of Caringo. This system forms a cluster build out of simple sata based machines. No OS, no filesystem, just boot from a USB stick and you have plenty of linear dataspace. All (data-)streams can be stored and retrieved by accessing the cluster over HTTP so this is an ideal solution for rich content. The best part; all data is constantly integrity checked and when data is corrupted, or disk fail, detection and regeneration is swift and fully transparent.
See: http://www.caringo.com
@ rocky – check fowlers presentation on solaris roadmap and you’ll see that they are committed to zfs and even are pushing it as a major feature. also check the interview here: http://www.serverwatch.com/news/article.php/3897706/Oracle-Details-Upcoming-Solaris-11-Release.htm
opensolaris seems far from dead, just getting less public support by oracle. how this pans out we will see.
in any case there are numerous how-to’s for building a zfs-home-nas to be found online. its a cheap, proven to work solution unlike some of the vaporware advocated here.
ZFS is nice but you still need a separate backup as it sometimes loses an entire zpool (probably infrequent but there are many forum threads on this).
For archival use where the rate of change is very low, it’s worth looking at FSarchiver – this archives an entire filesystem at the file level, including all permissions and attributes, and can be restored into another filesystem type (e.g. archive NTFS restore into ext3). The interesting part is that it has block-level checksums. It doesn’t do any error correction, but you could simply have two FSarchives on separate disks – when one file fails to be retrieved due to checksum errors, you could retrieve the file from the other archive. Not as nice as RAID 1 but could also be used for write-once media.
I’m kind of late, but isn’t this the type of problem 520-byte-sector disks are designed to solve? Is there anything other than NAS/SAN that uses them?
I’ve thought about writing a script that creates a hash/checksum of files and saves them in a text file, then compares them periodically. That would at least detect bit rot, so you could restore from the last good backup.
All,
I’d asked the writer of the note to respond to the many replies, but didn’t hear from him until lately. Here’s his response:
The next version of Windows Home Server (Vail) will have CRC checksums for file integrity, but their built in file duplication technology is far too wasteful for large files like video files.
Yet I don’t want to deal with RAID – especially at home.
Ideal solution? Vail also supports iSCSI natively, and you can add iSCSI targets to the storage pool (!!)
So, The first time I can score me a Drobo Pro for $1100, I’m adding it to my $360 HP MediaSmart that I will be upgrading to Vail when it’s released.
Drobo for RAID-like protection of hard drive failures without the hassles of managing RAID arrays, and WHS Vail for bit rot protection. Backed up to the cloud by BackBlaze for $5 a month.
Can’t beat that!
Well, the obvious solution of ZFS has been mentioned multiple times, but the hiccup has always been the server hardware to make the viable from a cost, sound and energy point of view. I think that HP may have solved many of these issued with their new N36L microserver. The chassis cost is negligeable, especially when compared to most purpose built NAS products. Coupled with a ZFS capable OS (Nexenta or Sokaris Express) and ECC memory and you have something that addresses the archive issue quite nicely. And to be complete, a second instance off site with ZFS send receive replication and you have complete protection. With planned scrubs to ensure that the in disk data hasn’t been corrupted by bit rot you’re all set.