Reducing the Training Overhead of the HPC Compression Autoencoder via Dataset Proportioning

Document Type

Conference Proceeding

Publication Date

1-1-2021

Abstract

As the storage overhead of high-performance computing (HPC) data reaches into the petabyte or even exabyte scale, it could be useful to find new methods of compressing such data. The compression autoencoder (CAE) has recently been proposed to compress HPC data with a very high compression ratio. However, this machine learning-based method suffers from the major drawback of lengthy training time. In this paper, we attempt to mitigate this problem by proposing a proportioning scheme to reduce the amount of data that is used for training relative to the amount of data to be compressed. We show that this method drastically reduces the training time without, in most cases, significantly increasing the error. We further explain how this scheme can even improve the accuracy of the CAE on certain datasets. Finally, we provide some guidance on how to determine a suitable proportion of the training dataset to use in order to train the CAE for a given dataset.

Identifier

85123221360 (Scopus)

ISBN

[9781728177441]

Publication Title

2021 IEEE International Conference on Networking Architecture and Storage Nas 2021 Proceedings

External Full Text Location

https://doi.org/10.1109/NAS51552.2021.9605407

Grant

CCF-1718297

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS