Would those be the same wagons that regularly lose copious amounts of data because the driver left the cab open when he stops to spend a penny? You only have to google ‘tape data loss’ for the amount of data going walkies through human error especially in transit.

Not to mention a whole raft of new cases coming out of the closet. It’s becoming ‘de rigueur’ for public organisations to lose data in the UK. I do not see that trend slowing whilst we rely so heavily of ultra-portable storage media. When reasonable amounts of money or precious jewels are moved around the carrier is usually cuffed to the briefcase or two scary looking kevlar suited chaps laden with CS gas canisters storm past you on way to their armored personnel carrier. When 65million confidential records are sent outside an agency in the UK some spotty teenager puts it in his back pocket making a mental note to pop down the post office after his fag break.

Why is a secure pipe for data transmission not a solution to consider? In either case encryption and strong policies on access are key.

On your other point I agree, it is all to easy for data to get wiped by an incompetent or disgruntled sysadmin. This why a disk-based archive solution simply needs to protect that data from being wiped out over the network, even by an administrator with a penchant for sudo commands and all the passwords under the sun.

Solutions, sometimes by individuals (or communities) going their own route; other times with some help from companies who have loftier goals than “lock-in”; always should start by letting you decide what hardware he wants to use both now and in the future (point 2, 3!)

Situation today from major vendors is: “You must take our software, our hardware, our support package, our future for the disk based archive”. Then you get shown a pretty little picture of shiny equipment; don’t ask questions such as “what happens next year” cos you know the answer “oh! We’ll support whatever comes out…” when you know darn well that they’ll only support what they want to, for as long as they want to, and at a price they want to.

Is there a place for any company to provide software that supports the ten points in the article, or will people always prefer to “build it themselves”, whatever effort that entails?

The answer/solution we came to is: let’s bolt a piece of generic software on to hardware that doesn’t force you to make hardware decisions; doesn’t force you to stop picking up new hardware solutions as they become available, and that does provide full support for all ten points in the article such as data delivery/security/checking/etc.

MatrixStore isn’t perfect – e.g., it only runs on MacOSX, but it does fully support you picking your own hardware (a lot of non-apple devices can be attached to apple h/w) and a full, build-it-yourself Linux version is close (Linux version is available now, but has to be compiled by the company). For security it strips down the OS (linux/macosx) so bare that even ssh is switched off, along with a few other tricks to lock data down. So, yes, as long as you want the software managing the communications (etc) to the devices, you have to run the software, but no hardware tie in and a lot of flexibility should you ever decide you don’t want to use the software for some reason (without ending up with hardware doorstops).

As for offsite backup – not sure if I do or don’t agree on that – data pipes (especially IP based ones) aren’t so expensive nowadays – are there any decent cost comparisons out there? Anyway, the (Uk) government are probably wishing they took encryption a LOT more seriously in whichever form offsiting data takes!!

We decided several years ago to host our online archives on NFS servers. We have at least 4 different vendors – depending on how you count the “who owns them this week” saga of the Snapserver family – and some major changes in technology. Most recently, we’re tossing our old NAS appliances for one Sun Thumper – still runs NFS, no changes to filesystem structure, more-or-less transparent swap out for the clients. To bad Sun can’t put a “stoner-on-the-night-shift-proof” user interface on Thumper-as-NFS-server. Maybe, someday Sun will get a clue about storage, but I’m not holding my breath.

We use a vendor-neutral format for our disaster recovery tapes – TAR. So if the Thumper goes poof, and Sun doesn’t make them any more, we can mount the new NFS server and restore the file system without any proprietary hardware or software.

EMC and the other big players will target the new, legal-department-driven, CYA archiving market. MatrixStore and others are targeting the video asset market. Both have more money to throw at these problems than we can afford. Neither market will know that they are missing lots of important features, and will be crying in their milk in a few years when their major chunks of their “archives” are corrupted or inaccessible due to RAID 5 write holes and other tricks the industry doesn’t like to talk about.

MediaStore claims to be a complete archive solution – as long as your disaster recovery plan includes setting up a distant mirror site (double the license fees to MediaStore) at the end of a *very* fat and expensive network pipe. No doubt EMC et al will push similar solutions too increase their margins.

For disaster recovery (an important element of a full Archiving solution), I prefer the value-bandwidth-security proposition of the proverbial “station wagon full of tapes screaming down the Interstate”. (Feel free to substitute your favorite portable media and physical transport mechanism). Security? How many times has a virus or operator error wiped out all of your offline media?