Storage-aware task scheduling for performance optimization of big data workflows

Document Type

Conference Proceeding

Publication Date

7-2-2018

Abstract

Many large-scale applications in various domains are generating big data, which are increasingly processed and analyzed by MapReduce-based workflows deployed in Hadoop systems. In addition to computing time, the makespan of such data-intensive workflows is also largely affected by communication cost. Particularly, there are two levels of data movement during the execution of distributed workflows in Hadoop: i) from map tasks to reduce tasks within each individual MapReduce module and ii) between each pair of adjacent modules in the workflow. Traditionally, these two aspects of network traffic have been treated separately as data locality at the task and module or job level, respectively. However, the interactions between these two levels of data movement may create complicated dynamics and their compound effects remain largely unexplored. In this paper, we formulate a task scheduling problem that considers data movement at both levels to minimize the end-to-end delay of a MapReduce-based workflow. We show this problem to be NP-complete, and design a storage-aware big data workflow scheduling algorithm, referred to as SA-BWS, to optimize workflow performance in Hadoop environments. The performance superiority of SA-BWS is illustrated by extensive simulations in comparison with the default workflow engine in Hadoop and existing scheduling methods.

Identifier

85063912833 (Scopus)

ISBN

[9781728111414]

Publication Title

Proceedings 16th IEEE International Symposium on Parallel and Distributed Processing with Applications 17th IEEE International Conference on Ubiquitous Computing and Communications 8th IEEE International Conference on Big Data and Cloud Computing 11th IEEE International Conference on Social Computing and Networking and 8th IEEE International Conference on Sustainable Computing and Communications Ispa Iucc Bdcloud Socialcom Sustaincom 2018

External Full Text Location

https://doi.org/10.1109/BDCloud.2018.00163

First Page

1095

Last Page

1102

Grant

2017YFB1300301

Fund Ref

Northwest University

This document is currently not available here.

Share

COinS