On MapReduce Scheduling in Hadoop Yarn on Heterogeneous Clusters

Document Type

Conference Proceeding

Publication Date

9-5-2018

Abstract

Hadoop is a distributed computing system widely used for big data processing in various domains. As the data volume continues to increase rapidly, Hadoop systems have become a critical contributor to the success of many big data applications. The MapReduce scheduler is a key component that determines the overall performance of a Hadoop cluster. In this paper, we formulate and investigate a task scheduling problem in a heterogeneous Hadoop cluster to minimize the completion time of a batch of MapReduce jobs. We first design a prediction model to predict the end time of a task, which is used for placing the corresponding data block on a node in advance to reduce the data transmission time and the overall job completion time. Based on this prediction model, we propose a task matching-based scheduling algorithm, referred to as TMSA, to schedule the tasks in the task queue in Hadoop, by taking into account the real-time performance of each node in the cluster and the matching degree between nodes and tasks. Experimental results show that the prediction model achieves high accuracy and TMSA significantly reduces the completion time of a batch of MapReduce jobs compared to existing schedulers.

Identifier

85054085042 (Scopus)

ISBN

[9781538643877]

Publication Title

Proceedings 17th IEEE International Conference on Trust Security and Privacy in Computing and Communications and 12th IEEE International Conference on Big Data Science and Engineering Trustcom Bigdatase 2018

External Full Text Location

https://doi.org/10.1109/TrustCom/BigDataSE.2018.00264

First Page

1747

Last Page

1754

Grant

CNS-1560698

Fund Ref

National Aerospace Science Foundation of China

This document is currently not available here.

Share

COinS