High-Ratio Lossy Compression: Exploring the Autoencoder to Compress Scientific Data
Document Type
Article
Publication Date
2-1-2023
Abstract
Scientific simulations on high-performance computing (HPC) systems can generate large amounts of floating-point data per run. To mitigate the data storage bottleneck and lower the data volume, it is common for floating-point compressors to be employed. As compared to lossless compressors, lossy compressors, such as SZ and ZFP, can reduce data volume more aggressively while maintaining the usefulness of the data. However, a reduction ratio of more than two orders of magnitude is almost impossible without seriously distorting the data. In deep learning, the autoencoder technique has shown great potential for data compression, in particular with images. Whether the autoencoder can deliver similar performance on scientific data, however, is unknown. In this article, we for the first time conduct a comprehensive study on the use of autoencoders to compress real-world scientific data and illustrate several key findings on using autoencoders for scientific data reduction. We implement an autoencoder-based compression prototype to reduce floating-point data. Our study shows that the out-of-the-box implementation needs to be further tuned in order to achieve high compression ratios and satisfactory error bounds. Our evaluation results show that, for most of the test datasets, the tuned autoencoder outperforms SZ by up to 4X, and ZFP by up to 50X in compression ratios, respectively. Our practices and lessons learned in this work can direct future optimizations for using autoencoders to compress scientific data.
Identifier
85103163762 (Scopus)
Publication Title
IEEE Transactions on Big Data
External Full Text Location
https://doi.org/10.1109/TBDATA.2021.3066151
e-ISSN
23327790
First Page
22
Last Page
36
Issue
1
Volume
9
Grant
1812861
Fund Ref
National Science Foundation
Recommended Citation
Liu, Tong; Wang, Jinzhen; Liu, Qing; Alibhai, Shakeel; Lu, Tao; and He, Xubin, "High-Ratio Lossy Compression: Exploring the Autoencoder to Compress Scientific Data" (2023). Faculty Publications. 1958.
https://digitalcommons.njit.edu/fac_pubs/1958