A hardware storage controller is an expensive guarantee that you’re using old technology to handle your most important data. Hardware specs are frozen early in the typical 18-24 month development cycle so by the time you get your “new” controller it is already 2 years old.
But it may not have to be that way. In Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization researchers Muli Ben-Yehuda, Michael Factor, Eran Rom, Avishay Traeger, Eran Borovik and Ben-Ami Yassour of IBM Research–Haifa wanted to find out if virtualized storage controller features are feasible.
Short answer: with some tweaking, yes.
The big question is overhead. Storage controllers are typically in the data path, so latency, as well as compute efficiency on out-of-date processors, are real concerns.
Unlike the gateway approach of virtual storage appliances (VSA), the team ran the VMs directly on storage controllers using the Linux KVM hypervisor.
Overhead
The team identified 3 sources of performance overhead:
- Base. System work such as virtual memory managment or process switching.
- External communication. Important if a new function is layered on top of the storage system, such as a file server.
- Internal communication. Virtual machine coordination and communication with the hardware controller.
Reducing overhead
Different techniques are used to limit each type of overhead.
Base They statically allocate CPU cores to the guest to ensure sufficient resources. Memory is also statically allocated to the VM to reduce translation overheads.
External Device assignment is the highest-performing approach as it eliminates hypervisor intervention for physical events. This requires assigning the network device directly to the guest using an SR-IOV (single root I/O virtualization) enabled adapter which allows the guest to send requests directly to the device.
Internal communications To reduce internal communication overhead, they modified KVM’s block driver to poll instead of interrupt. This gives a fast, exit-less, zero-copy transport.
Results
By using these techniques, we show no measurable difference in network latency between bare metal and virtualized I/O and under 5% difference in throughput. For internal communication, micro-benchmarks show 6.6μs latency overhead, read throughput of 357K IOPS, and write throughput of 284K IOPS; roughly seven times better than a base KVM implementation. In addition, an I/O intensive filer workload running in KVM incurs less than 0.4% runtime performance overhead compared to bare metal integration.
That sounds pretty good.
The StorageMojo take
While the static assignments may reduce flexibility, the win is updating storage functionality on the fly. But are there viable use cases? The arc of controller history suggests there are.
The earliest disk drives were directly controlled by the host CPU. Over the decades that and much other functionality migrated to controllers and to disks. Lately that trend has slowed because of large investments in existing standards.
This paper shows that it is possible to migrate more functionality to controllers without lengthy development cycles, enabling architects to make different tradeoffs.
For example, big data requires big pipes, and big pipes are expensive. If volume-reducing preprocessors could be added to file servers, existing bandwidth could be optimized.
More importantly, it suggests that by virtualizing the controller’s applications, the underlying hardware can be updated more frequently. To be fair, that’s not what the authors suggested, but it certainly seems possible based on their work.
Courteous comments welcome, of course. Jeff Darcy of Red Hat has his own list of favorite papers from FAST ’12 here.
Recent Comments