Testing, testing, 1 2 3 . . .

by Robin Harris on Monday, 7 July, 2008

George Ou weighs in
Many good points have been made about the problems with the Tom’s Hardware flash SSD tests. My former colleague George Ou, late of ZDnet, weighed in with an excellent summary of the TH testing problems:

The tests are very flawed.  If you read the results, the SSDs with the worst power consumption aren’t the ones getting the worst battery life.  The ones with great performance and above average power consumption turn out to be the worst on battery life WITH THE TEST THEY RAN.
 
What this says is that Tomshardware’s measurements weren’t wrong, but what they were measuring was wrong.
 
The load test was not well controlled.  The SSDs with great performance allowed the benchmark to run faster which cranked the CPU more.  The difference in the CPU state is what explains the discrepancy in their data.
 
A proper measurement would have done a fixed amount of CPU work and a fixed amount of storage work and then you can see how long the battery lasts.  They could have simply played a movie off the storage system and let it play until the battery died.  Videos are great because they’re fixed computational workload and fixed storage workload.
 
This is yet another example of bad science by Tomshardware.

I don’t buy the “play a movie” test – that only tests playing a movie – but I do accept that Tom’s Hardware didn’t do a great job of testing. So what?

I’ll be returning to the testing issues shortly – after pausing for this disclosure.

Disclosure: I’m biased towards notebook flash drives
Unlike, AFAIK, any of the commenters – pro or con – I used a flash-based Windows notebook every day for 5 years and loved it. It had a 10 hour battery life, a full-size keyboard and a sleep mode that really worked. Bliss!

I also paid an extra 20% – $400 back when the dollar was worth something – for the dinky 10 MB CF card it used. It was worth every penny.

Based on my sample size of 1 (me) here’s WHY it was worth an extra 20%:

  • Battery life. The Omnibook 300 went from 5 hours to 10 hours of battery life with flash.

Factors that didn’t matter:

  • Performance: I never compared the disk to the flash, but the performance was “good enough” with either.
  • Durability: nobody gets 5 years out of a notebook drive, but crashing wasn’t a liability since all docs were copied to an external system.
  • Boot up time: sleep mode worked perfectly, so I’d reboot once a month at most. I did not care about boot time.
  • Multi-media workloads: while I agree with George that a video provides a good fixed workload, notebook SSDs are aimed at business travelers whose workloads commonly allow drives to spin down. But this is a topic that deserves a deeper look.
  • Capacity. The Omnibook had a compression utility that effectively doubled capacity to 20 MB. But it was easy to copy stuff off the ‘book – Laplink – so it never felt cramped.

Those are my biases. They may or may not be the biases of Mr. Road Warrior – but I suspect they are close. End disclosure.

Testing, testing, testing
Performance testing is a black art. That’s why test driving applications remains popular: there are so many variables that predictions based on benchmarks are close to useless.

Because of that I prefer to look at the preponderance of evidence rather than a single benchmark or set of tests. More data points paint a clearer picture.

For example, the single most positive SSD test I’ve found is Anandtech’s MacBook Air SSD. The similar results of another test is here.

Battery Life Test (H:MM) 80GB 4200RPM HDD 64GB SSD % Improvement
Wireless Internet + MP3 4:16 4:59 16.8%
DVD Playback 3:25 3:56 15.1%
Heavy Downloading + XviD + Web Browsing 2:26 2:42 11.0%

Bottom line best case: a 17% improvement. Not zero but not, as most reviewers concluded, enough to justify the price.

Ars Technica also reviewed the MBA SSD and had mixed results. They concluded:

. . . I had high hopes for the battery life on the SSD model. Unfortunately, I was met with only moderate gains when there were any at all.

More Anandtech
Anandtech also tested a high-end Memoright SSD in a high-end MacBook Pro. Here are their results:

Battery Life in Hours (Higher is Better) MacBook Pro (Hitachi 5400RPM) MacBook Pro (Memoright SSD)
Wireless Internet Browsing + MP3 Playback 5.13 hours 5.0 hours
DVD Playback 3.88 hours 3.58 hours
Heavy Downloading + XviD Playback + Web Browsing 3.38 hours 3.37 hours

The StorageMojo take
All workload testing is a compromise – but the preponderance of the evidence is clear: significant – i.e. 40% or better – notebook power advantages just aren’t there. UMPCs that can’t afford a disk – flash will win. Notebooks? Hasta la vista, baby.

The one SSD advantage that is yet to be debunked is durability. Someone made a case that just the maintenance advantages alone justify SSDs for enterprise notebooks. And it may be that simple.

Yet even there, the issues of hard CapEx dollars against softer expense dollars will work against SSDs.

Maybe the next gen of flash controllers will solve all the problems and usher in the age of flash storage everywhere. But piddly 20-30 minute gains for an extra $300 bucks won’t do it.

Comments welcome, of course. Just so everyone knows: I haven’t done any work in the last few years for either flash drive or disk drive vendors. I wish them both the best.

{ 2 comments… read them below or add one }

Martin July 8, 2008 at 3:50 pm

Take a look at Lenovo X300 – the brand new high-end, which provides very long battery hour by cutting down the processing power smartly. Lenovo offsets the performance loss with the help of SSD. That is a way of engineering.

Some SSDs in the market have problem in idle time power consumption. The root cause is their controller design, not NAND itself. Time will solve it.
Major vendors are already free from that issue.

Again, as well as in booting, SSD brings benefit in daily use cases. You will see hardly any ‘wait… icon’ in most of your working time.

And MLC products will be shipping soon, with the less than half of the current price level. The price will go down again and again in coming years.

Don’t be so pessimistic. It’s brighter out there.

Colin July 14, 2008 at 9:25 pm

Tom’s has updated their original review and one SSD really shines. It has a better controller design and much lower power consumption. I think it and the models that will come out to equal or best it will be quite popular with power users. Lower power consumption and better performance – the promise of SSDs has been delivered on.

And this is just the start, of course… We’re on second generation devices and many optimizations haven’t been thought of let alone implemented in hardware or software. Wait a year or two and everybody will want one.

Leave a Comment

Previous post:

Next post: