Maybe software will eat the world, but sometimes the physical world gives software indigestion. That fact was evident at the Flash Memory Summit this month.

As mentioned in Flash slaying the latency dragon? several companies were showing remote storage accesses – using NVMe and hopped up networks – in the 1.5 to 2.5µsec range. That’s roughly 500 times better than the ≈1msec averages seen on today’s flash storage.

That’s amazing. Really. But will it help?

Fusion-io investigated sharing storage – to help amortize the cost of their PCIe cards across multiple servers – for years, but the tools weren’t there to make it work. Now, with NVMe and PMC’s Switchtec PSX PCIe Gen3 or Avago’s ExpressFabric PCIe storage switches the hardware tools are there.

The software problem
But lopping off 998µsec from storage I/O isn’t the boost we’d like, because the storage stack is so freakin’ sl-o-o-w-w. How slow?

In a recent record-setting SPC-2 benchmark, an EMC VMAX 400k achieved 3.5ms response time with 64KiB transfers, 800 streams, and 4 IOs per stream.

However, looking at a recent TPC-C benchmark – which are at the application level, not the storage device level – we see that minimum response times are 110ms with maximum response times of almost 10 seconds. Clearly, 1ms doesn’t make much difference.

Granted, the TPC-C results include application and database overhead – in this case SAP – not just the storage stack. But with all-flash arrays averaging under 1ms SPC response times, performance improvements need to come from the software.

The StorageMojo take
The yawning chasm between SPC and TPC results calls into question the value of the SPC benchmarks. Great for vendor’s “plausible deniability” when customers complain about performance. But as such a small portion of total latency it’s obvious that software – and perhaps server hardware – are key to reduced latency and higher performance.

Software may be eating the world, but the days of easy performance boosts from new CPUs are over. Software has to step up to improve performance.

Courteous comments welcome, of course.