Advising big data transfer over dedicated connections based on profiling optimization

Document Type

Conference Proceeding

Publication Date

12-1-2019

Abstract

Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.

Identifier

85077313347 (Scopus)

Publication Title

IEEE ACM Transactions on Networking

External Full Text Location

https://doi.org/10.1109/TNET.2019.2943884

e-ISSN

15582566

ISSN

10636692

First Page

2280

Last Page

2293

Issue

6

Volume

27

Grant

DE-AC05-00OR22725

Fund Ref

Oak Ridge National Laboratory

This document is currently not available here.

Share

COinS