a

I’m a “prosumers” user, so I’m a little out of step with normal users, but I understand them. I had HD reliability issues on my main HD for my laptop. So what I did is bought a new 2nd HD, put it in the optical bay, and moved all my data there. My main HD filled up, and became less and less reliable (running WinXP BS, as I needed it for business school). Eventually, WinXP wouldn’t boot. Great! Got a 120GB replacement drive, and am now operating from the Ubuntu Linux CD. Haven’t bothered installing yet (actually, did, but ran into a stupid error 97% way done about GRUB install to MBR failing, then it borked out; haven’t had time to mess around with it yet). BTW, Ubuntu running from a CD boots up faster and loads programs faster than Winblows XP did for me for the last couple of years (from the HD). That’s really sad. And worse yet, apparently Win Vista is a downgrade from XP in terms of performance.

In the meanwhile, my data is pretty safe on an that data-drive, which is now wrapped in the HD static-proof bag that my new HD came with. And I have a lot of my photos on DVDs as well.

A few other comments…in RAID, drive failures tend to be correlated due to similar use-patterns. This greatly diminishes the value of RAID for home-users. It just doesn’t seem worth it. Because RAID isn’t a backup. So then, to really be protected, you need 3 hard-drives: 1 original, 1 for RAID, and 1 for backup. You’ll have a tough enough time convincing people to get 1 extra HD, let alone 2.

Ideally, backups to Gold Archival DVDs should be done whenever you know anything is fixed (e.g., not a work in progress), and have enough to fill up a DVD. But, although I’m willing to spend $80 for a 100-pack of gold archival DVDs, the average consumer will wonder why spend that kind of $$$, when a 100-pack of normal DVDs is like $20 or whatever.

There are other considerations when backing up, as well. A few ideas of mine:

* OS and apps should never be on the same HD as data. At the very least, they should be on a different partition. If on a different HD, the OS and apps would ideally be on a faster smaller HD…maybe even a small SAS drive instead of SATA for better performance. Also, if you want to make use of solid sate drives (like mtron), you can get small drives for your virtual memory, for your temporary files, and even for your /var files (if you’re a linux person).

*Your backup HD should be separated from your main HDs. Being inside the CPU case means more heat, which is bad for HDs. Also, if possible, it should be powered down when not in use by the automatic backup system (e.g., have software do this). That way, it has even less wear and tear.

*Use an ultra-high performance HD for your OS and apps, like a small Raptor or Cheatah.
*If possible, use an even better HD, like the mtron ssd’s, for your swap file, temp directory, /var directory, and anything else that gets a lot of read/writes all the time.
*Use an affordable high-performance high-capacity drive for your data, like the Hitachi DeskStar 7k1000, or Samsung Spinpoint F1. You can get these affordably in the 500, 750, or 1000GB sizes (by affordably, I mean more than 3.8 GB/$).
*Use an enterprise-class rated HD for your data backups. Something that’s rated for continuous 24×7 usage (even though you’ll have it powered down when not in use). E.g., the Hitachi Ultrastar, or Seagate Barracuda ES.2 SAS drives. These also come with 5 year warranties. Use them in an external enclosure. Or use an eSATA HD.

Ideally, all this stuff I’ve talked about should be done automatically, although that seems near-impossible. But Apple is always great with this stuff, making it easy for consumers. And they could even make all that hard-ware selection easy by selling premium reliability/performance grade macs that come with multiple HD’s, and the sys setup to use them appropriately for every-day use vs. backup.

PS: I find the arrogance displayed by some here flabbergasting. People are chastising the photographer woman for not knowing the difference between RAM and hard-drive? Ok, sure, she’s just a few notches above great-grandpa in terms of computer knowledge. But I bet a lot of the geeks harping about how imperative it is that people understand their computers, don’t know how to change a tire on a car, or deal with routine house-maintenance, or know a god-damned thing about their car for that matter. Also, I bet that that photographer — if she’s a professional — knows a lot more about Photoshop than anyone on this forum. Maybe if she saw people here doing some silly thing in Photoshop, she’d comment on how ignorant we were!

Not to go to far down Microsofts Road, I would imagine there are other vendors with similar products waiting for Microsoft to bloody the waters with the release of their usual unfinished product. They will then jump in with something Linux based that’s cheaper and has things like server backup built in.

The current selection of consumer NAS boxes are not worth the time to setup IMHO. They provide some useful services, but fail to provide true “server” abilities.

I made the following post originally on another Blog in response to a CAS post. No one ever replied. Maybe its worthless but I believe the “thousands of pictures” is very to the point.

Neither RAID nor backup would have helped without a Strategy and some Manageware skills.

Posted in response to:
“Oh, and one more thing. Robert Pearson’s first comment above ["CAS (Content Addressed Storage) requires organizing the Storage by Content. This requires some, if not a whole lot of, advance knowledge of the Information being stored."] truly invites a reaction, as the exact opposite is true. Real CAS explicitly does **not** require organizing Storage by Content.”

I guess I am a little confused by all of this?
You have my respect as an acknowledged expert in CAS.
I don’t claim to be.
My goal is to build consensus and make CAS better and more viable.

I fully agree with your statement about the Information, once it is in the CAS. My statement was directed to these areas:
1) Determining Stored Information that is a candidate for CAS
2) Getting it in the CAS
3) Once the Information is in the CAS dealing with the same
issues “non-CAS” Storage has like “hot-spots”, bottlenecks, updates, migrations, replications. and synchronizations.

Is CAS exempt from these “Management” issues?

How long would it take to migrate 30 TB of Seismic Information to CAS?
The same as for “regular” Storage?
I can tell you how long a 30 TB “snapshot” takes to commit and that is not being hashed for Content.

Here is an example:
A real world example is when my Uncle died. His wife preceded him by a few
years. My Uncle loved to take pictures. He had thousands of pictures.
He had some made into movies. He loved to show these at the family reunions. One whole room was his for the slide shows and the home theatre. And he took more pictures all the time.
He knew all the people in the pictures and had stories to tell about the person, the place, and the picture.

I helped his children transfer all these pictures to CD. Not every recipient had DVD capability. But we really didn’t know what to do with them. They seemed very valuable to us somehow. Like a valuable piece of history.
They should have been made with a VCR, or transferred to a VCR, so there would be audio but that technology wasn’t available until much later.
Between all of us we could probably identify 10% of the people and places.
We physically archived the pictures, movies and master DVD, made the CDs and disbanded.
That video Information has become valueless.
The technology is there to view them at any time. To what purpose?
Nobody derives any value from viewing them.

We put them on the CAS.
Suddenly they became hugely popular as background for TV shows. They were constantly being accessed. The TV people wanted to edit them and add comments.
Now I’ve got many versions of the originals that have become “fixed” that need to go on the CAS. I am out of CAS.
Does CAS offer versioning software?
Does it work with popular versioning software?

Then I got a chance to buy hundreds of hours of old TV sitcom video. We put it on the CAS figuring it would only be read.
Wrong! Thousands of “stills” have been edited out and produced.
These need to be stored in a related fashion to the CAS originals.
Is the hash on a still from a video the same as the video hash?
There is all kinds of Information related to the original Content that is stored on the CAS in a Content related fashion.
I keep hearing database. It was in a database.

Plus they want to insert “human realistic” pellet people in place of the
original actors to avoid paying any royalties. So now I have
the original, modified originals and “human realistic” copies. They are all the same content but play at different levels of value. Maybe these are not good candidates for CAS?

What about the level of Information High Availability? Information Integrity?
Disaster Recovery? Business Continuance?
Worst of all, Findability?
Each Unit of Information stored on the CAS is subject to all these demands.

“Vendor A” said CAS was the wrong solution. What I needed was “5 nines (99999)” active Storage. And lots of it. He bought me lunch.

p.s. Most dual failures happen when you buy a drive from a manufacturer that sells drives with higher failure rates. Some of these defects can be fixed with drive firmware updates, but unless that is an automatic process most consumers or even business will get the ugly failure.

p.p.s. Raid 6 does not fix the problem of “oops! All my drives have decided to stop reading or writing data”.

At home I just use a cheap $15 USB metal case with my old 5400 RPM hard drive. I use a XCOPY .bat script and cygwin + ssh to copy my linux tar backups to backup my data. However, while the solution is not optimal it’s all automated and has worked without any issues.

Does RAID have a place in your home env? Sure if you have the knowledge, enthusiasm, need for performance and particular RTO why not? But do my folks or friends or even myself need that only to keep spreadsheets and documents, pics and occasional vacation videos need RAID…I don’t think so.

Now the Drobo type thing is pretty cool since to a machine it’s just another disk drive but I don’t see myself or anyone in my family to be shelling out $400+ for a ‘reliable USB drive’

George, RAID 10 is not the same thing as RAID 01. Furthermore RAID 1 may have better performance if you you configure it for a particular application. An application that can do parallel reads and writes through two separate disk controllers to two separate RAID 1 volumes. However, as a general purpose read/write filesystem on a single controller you can’t beat RAID 10 otherwise the whole storage industry must be wrong.

High availability with RAID is a bit too much for home use - for me.

When it comes to home users and their data I would like to suggest that there are three main concerns:

1. protection against accidental deletion
2. protection against HD crash
3. protection against site/location disaster

and the solutions have to be

4. foolproof
5. proactive
6. affordable

1. this is probably the most common. A daily backup, if it is maintained, is the best defense. But since it is rarely maintained even at businesses, what would be ideal is some sort of snapshotting service, hopefully enabled by default on a turnkey solution.

2. protection against HD crash - this is probably the second most common cause of data loss and is the chief reason for the existence of RAID. Despite some of the comments above, RAID is in fact a pretty good defense against HD failure (else, uh, no-one would use it) with the advantages of proactive user alert for drive failure (on good devices, anyway) and - importantly - no downtime or reinstallations necessary if failure does occur.

3. Location disaster ie fire or flood - nothing can protect against this but a current offsite backup (I don’t share Robin’s faith in fire safes ..!)

and to the necessary attributes of the service …

4. Foolproof. Backup’s achilles heel. Backups, when they aren’t just a single HD copied over each night, require media changes. Testing. A labelling system. A schedule. Testing. Buying expensive new media when the old stuff wears out, which nobody notices anyway without …. Testing. Did I mention testing? Doesn’t really matter how many times I mention it actually, no home user will ever test their backup system.

Backups are absolutely necessary. They are also high-maintenance and require user attention. This is why I would basically not recommend anything except an online automated backup system. That’s the only way I can imagine Mrs “What’s RAM?” actually having an extant current backup somewhere…

5. Proactive - I like RAID because it’s proactive. When something goes wrong, it tells you, assuming it’s anything like a decent system. The problem with backup is it’s reactive. The typical user will be reminded of the fact she hasn’t updated her backups in 6 months by nothing but the actual failure of her disk. Software warnings are way too easy to ignore/dismiss, and it’ll become a habit.

6. Affordable - the tapes, fragile mechanical interiors and pure cost in time and worry of a home tape backup system will cost you more than a good RAID would anyway.

OK, this comment is getting too long, let me get to the point. The perfect home backup system, as I see it, is this:

- a RAIDed “big drive”, with some kind of version control/snapshotting system on it - choose your poison, VSC or whatever. I use svn ; ) .. all the important data goes on here. Bonus points for a bootable image for the client sitting on there as well, as well as full client backups. A prominent visible position, warning lights clearly visible and audio alerts on, and a spare drive sitting next to it.

- an automated “lazy realtime” (ie, leisurely asynchronous) online backup for the RAID (and the client backups on it!) If the online service can do versioning and snapshots, all the better.

The key is they need to be running all the time behind the scenes, and make a loud beeping sound if something goes wrong. If the backup company doesn’t receive any data for 24 hours the person’s phone should be ringing. If the RAID drops a drive it should sound like a smoke alarm.

That’s my opinion of pretty much the only way to guarantee a good backup. Sorry for taking so long, thanks for reading : )

There are multiple ways to lose both drives at once, as we’re finding. For example, in the case of a lightning strike, we had a customer lose both drives, and his mirrored data, even though the PC which was on a surge protector was unscathed. There’s also the case where if you accidentally delete or corrupt a file from one drive, the mirroring function with dutifully delete or corrupt it on the other. RAID cannot protect you from data loss in the event of fire, flood or theft. So I’m now of the opinion that a safer solution is to have an off-site strategy with drives that can be periodically cycled rather than using a RAID-only solution for home users.

It’s not that I don’t like RAID. I do use it myself. It’s just that sometimes we think of a hard drive crash as the only way to lose data and forget about all the other ways the data can be lost.