RAPIDS: Reconciling Availability, Accuracy, and Performance in Managing Geo-Distributed Scientific Data

Document Type

Conference Proceeding

Publication Date

8-7-2023

Abstract

In modern science, big data plays an increasingly important role. Many scientific applications, such as running simulations on supercomputers or conducting experiments on advanced instruments, produce huge amount of data at unprecedented speed. Analyzing and understanding such big data is the key for scientists to make scientific breakthroughs. However, data might become unavailable for scientists to access when outages or maintenance of the storage system occur, which severely hinders scientific discovery. To improve the data availability, data duplication and erasure coding (EC) are often used. But as the scientific data gets larger, using these two methods can cause considerable storage and network overhead. In this paper, we propose RAPIDS, a hybrid approach that combines the multigrid-based error-bounded lossy compression with erasure coding, to significantly reduce the storage and network overhead required for maintaining high data availability. Our experiments show that RAPIDS reduces the storage overhead by up to 7.5x and network overhead by up to 3x to achieve the same level of availability compared to the regular EC method. We improve RAPIDS by building two models to optimize the fault tolerance configurations and data gathering strategy. We demonstrate that RAPIDS significantly improves performance when running on many CPU cores in parallel or on GPUs.

Identifier

85169584306 (Scopus)

ISBN

[9798400701559]

Publication Title

Hpdc 2023 Proceedings of the 32nd International Symposium on High Performance Parallel and Distributed Computing

External Full Text Location

https://doi.org/10.1145/3588195.3592983

First Page

87

Last Page

100

Grant

DE-AC05-00OR22725

Fund Ref

U.S. Department of Energy

This document is currently not available here.

Share

COinS