Total Variation Reduction for Lossless Compression of HPC Applications

Document Type

Conference Proceeding

Publication Date

1-1-2021

Abstract

With the growing size of high-performance computing (HPC) applications, a major challenge that domain scientists are facing is how to efficiently store and analyze the vast volume of output data. Compression can reduce the amount of data that needs to be transferred and stored. However, most of the large datasets are of floating-point format, which exhibit high entropy. As a result, existing lossless compressors usually achieve a modest reduction ratio of less than 2X. To address this problem, we propose a total variation reduction method to improve the compression ratio of lossless compressors. In particular, we first try to exploit space-filling curve (SFC), a well-known technique to preserve data locality for a multi-dimensional HPC dataset. We show and explain why a raw SFC, such as Hilbert curve and Z-order curve, cannot improve the compression ratio. Then, we explore the opportunity and theoretical feasibility of the proposed total variation reduction based algorithm. The experiment results show the effectiveness of the proposed method. The compression ratios are improved by 20.6% for FPZIP, and 18.4% for FPC, on average.

Identifier

85127741670 (Scopus)

ISBN

[9781665429313]

Publication Title

International System on Chip Conference

External Full Text Location

https://doi.org/10.1109/SOCC52499.2021.9739467

e-ISSN

21641706

ISSN

21641676

First Page

129

Last Page

134

Volume

2021-September

Grant

61751204

Fund Ref

National Natural Science Foundation of China

This document is currently not available here.

Share

COinS