Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression

Document Type

Conference Proceeding

Publication Date

1-1-2021

Abstract

Lossy compression techniques have demonstrated promising results in significantly reducing the scientific data size while guaranteeing the compression error bounds. However, one important yet often neglected side effect of lossy scientific data compression is its impact on the performance of parallel I/O. Our key observation is that the compressed data size is often highly skewed across processes in lossy scientific compression. To understand this behavior, we conduct extensive experiments where we apply three lossy compressors MGARD, ZFP, and SZ, which are specifically designed and optimized for scientific data, to three real-world scientific applications Gray-Scott simulation, WarpX, and XGC. Our analysis result demonstrates that the size of the compressed data is always skewed even if the original data is evenly decomposed among processes. Such skewness widely exists in different scientific applications using different compressors as long as the information density of the data varies across processes. We then systematically study how this side effect of lossy scientific data compression impacts the performance of parallel I/O. We observe that the skewness in the sizes of the compressed data often leads to I/O imbalance, which can significantly reduce the efficiency of I/O bandwidth utilization if not properly handled. In addition, writing data concurrently to a single shared file through MPI-IO library is more sensitive to the unbalanced I/O loads. Therefore, we believe our research community should pay more attention to the unbalanced parallel I/O caused by lossy scientific data compression.

Identifier

85124513775 (Scopus)

ISBN

[9781728186726]

Publication Title

Proceedings of Drbsd 7 2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data Held in Conjunction with Sc 2021 the International Conference for High Performance Computing Networking Storage and Analysis

External Full Text Location

https://doi.org/10.1109/DRBSD754563.2021.00008

First Page

26

Last Page

32

Grant

CAREER-2048044

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS