Reducing the Training Overhead of the HPC Compression Autoencoder via Dataset Proportioning
Document Type
Conference Proceeding
Publication Date
1-1-2021
Abstract
As the storage overhead of high-performance computing (HPC) data reaches into the petabyte or even exabyte scale, it could be useful to find new methods of compressing such data. The compression autoencoder (CAE) has recently been proposed to compress HPC data with a very high compression ratio. However, this machine learning-based method suffers from the major drawback of lengthy training time. In this paper, we attempt to mitigate this problem by proposing a proportioning scheme to reduce the amount of data that is used for training relative to the amount of data to be compressed. We show that this method drastically reduces the training time without, in most cases, significantly increasing the error. We further explain how this scheme can even improve the accuracy of the CAE on certain datasets. Finally, we provide some guidance on how to determine a suitable proportion of the training dataset to use in order to train the CAE for a given dataset.
Identifier
85123221360 (Scopus)
ISBN
[9781728177441]
Publication Title
2021 IEEE International Conference on Networking Architecture and Storage Nas 2021 Proceedings
External Full Text Location
https://doi.org/10.1109/NAS51552.2021.9605407
Grant
CCF-1718297
Fund Ref
National Science Foundation
Recommended Citation
Liu, Tong; Alibhai, Shakeel; Wang, Jinzhen; Liu, Qing; and He, Xubin, "Reducing the Training Overhead of the HPC Compression Autoencoder via Dataset Proportioning" (2021). Faculty Publications. 4624.
https://digitalcommons.njit.edu/fac_pubs/4624