George Ou weighs in
Many good points have been made about the problems with the Tom’s Hardware flash SSD tests. My former colleague George Ou, late of ZDnet, weighed in with an excellent summary of the TH testing problems:

The tests are very flawed.  If you read the results, the SSDs with the worst power consumption aren’t the ones getting the worst battery life.  The ones with great performance and above average power consumption turn out to be the worst on battery life WITH THE TEST THEY RAN.
 
What this says is that Tomshardware’s measurements weren’t wrong, but what they were measuring was wrong.
 
The load test was not well controlled.  The SSDs with great performance allowed the benchmark to run faster which cranked the CPU more.  The difference in the CPU state is what explains the discrepancy in their data.
 
A proper measurement would have done a fixed amount of CPU work and a fixed amount of storage work and then you can see how long the battery lasts.  They could have simply played a movie off the storage system and let it play until the battery died.  Videos are great because they’re fixed computational workload and fixed storage workload.
 
This is yet another example of bad science by Tomshardware.

I don’t buy the “play a movie” test – that only tests playing a movie – but I do accept that Tom’s Hardware didn’t do a great job of testing. So what?

I’ll be returning to the testing issues shortly – after pausing for this disclosure.

Disclosure: I’m biased towards notebook flash drives
Unlike, AFAIK, any of the commenters – pro or con – I used a flash-based Windows notebook every day for 5 years and loved it. It had a 10 hour battery life, a full-size keyboard and a sleep mode that really worked. Bliss!

I also paid an extra 20% – $400 back when the dollar was worth something – for the dinky 10 MB CF card it used. It was worth every penny.

Based on my sample size of 1 (me) here’s WHY it was worth an extra 20%:

• Battery life. The Omnibook 300 went from 5 hours to 10 hours of battery life with flash.

Factors that didn’t matter:

• Performance: I never compared the disk to the flash, but the performance was “good enough” with either.
• Durability: nobody gets 5 years out of a notebook drive, but crashing wasn’t a liability since all docs were copied to an external system.
• Boot up time: sleep mode worked perfectly, so I’d reboot once a month at most. I did not care about boot time.
• Multi-media workloads: while I agree with George that a video provides a good fixed workload, notebook SSDs are aimed at business travelers whose workloads commonly allow drives to spin down. But this is a topic that deserves a deeper look.
• Capacity. The Omnibook had a compression utility that effectively doubled capacity to 20 MB. But it was easy to copy stuff off the ‘book – Laplink – so it never felt cramped.

Those are my biases. They may or may not be the biases of Mr. Road Warrior – but I suspect they are close. End disclosure.

Testing, testing, testing
Performance testing is a black art. That’s why test driving applications remains popular: there are so many variables that predictions based on benchmarks are close to useless.

Because of that I prefer to look at the preponderance of evidence rather than a single benchmark or set of tests. More data points paint a clearer picture.

For example, the single most positive SSD test I’ve found is Anandtech’s MacBook Air SSD. The similar results of another test is here.

Bottom line best case: a 17% improvement. Not zero but not, as most reviewers concluded, enough to justify the price.

Ars Technica also reviewed the MBA SSD and had mixed results. They concluded:

. . . I had high hopes for the battery life on the SSD model. Unfortunately, I was met with only moderate gains when there were any at all.

More Anandtech
Anandtech also tested a high-end Memoright SSD in a high-end MacBook Pro. Here are their results:

The StorageMojo take
All workload testing is a compromise – but the preponderance of the evidence is clear: significant – i.e. 40% or better – notebook power advantages just aren’t there. UMPCs that can’t afford a disk – flash will win. Notebooks? Hasta la vista, baby.

The one SSD advantage that is yet to be debunked is durability. Someone made a case that just the maintenance advantages alone justify SSDs for enterprise notebooks. And it may be that simple.

Yet even there, the issues of hard CapEx dollars against softer expense dollars will work against SSDs.

Maybe the next gen of flash controllers will solve all the problems and usher in the age of flash storage everywhere. But piddly 20-30 minute gains for an extra $300 bucks won’t do it.

Comments welcome, of course. Just so everyone knows: I haven’t done any work in the last few years for either flash drive or disk drive vendors. I wish them both the best.