StoreRush: An Application-Level Approach to Harvesting Idle Storage in a Best Effort Environment

Document Type

Conference Proceeding

Publication Date

1-1-2017

Abstract

For a production HPC system where storage devices are shared between multiple applications and managed in a best effort manner, contention is often a major problem leading to some storage devices being more loaded than others and causing a significant reduction in I/O throughput. In this paper, we describe our latest efforts StoreRush to resolve this practical issue at the application level without requiring modification to the file and storage system. The proposed scheme uses a two-level messaging system to harvest idle storage via re-routing I/O requests to a less congested storage location so that write performance is improved while limiting the impact on read by throttling re-routing if deemed too much. An analytical model is derived to guide the setup of optimal throttling factor. The proposed scheme is verified against production applications Pixie3D, XGC1 and QMCPack during production windows, which very well demonstrated the effectiveness (e.g., up to 1.8x improvement in write) and scalability of our approach (up to 131,072 cores).

Identifier

85027349960 (Scopus)

Publication Title

Procedia Computer Science

External Full Text Location

https://doi.org/10.1016/j.procs.2017.05.005

e-ISSN

18770509

First Page

475

Last Page

484

Volume

108

This document is currently not available here.

Share

COinS