Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition
Document Type
Conference Proceeding
Publication Date
11-14-2021
Abstract
Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time the challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-The-Art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.
Identifier
85119970619 (Scopus)
ISBN
[9781450384421]
Publication Title
International Conference for High Performance Computing Networking Storage and Analysis Sc
External Full Text Location
https://doi.org/10.1145/3458817.3476179
e-ISSN
21674337
ISSN
21674329
Fund Ref
U.S. Department of Energy
Recommended Citation
Liang, Xin; Gong, Qian; Chen, Jieyang; Whitney, Ben; Wan, Lipeng; Liu, Qing; Pugmire, David; Archibald, Rick; Podhorszki, Norbert; and Klasky, Scott, "Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition" (2021). Faculty Publications. 3677.
https://digitalcommons.njit.edu/fac_pubs/3677