Energy-efficient mapping of big data workflows under deadline constraints?

Document Type

Conference Proceeding

Publication Date

1-1-2016

Abstract

Large-scale workflows for big data analytics have become a main consumer of energy in data centers where moldable parallel computing models such as MapReduce are widely applied to meet high computational demands with timevarying computing resources. The granularity of task partitioning in each moldable job of such big data workflows has a significant impact on energy efficiency, which remains largely unexplored. In this paper, we analyze the properties of moldable jobs and formulate a workflow mapping problem to minimize the dynamic energy consumption of a given workflow request under a deadline constraint. Since this problem is strongly NP-hard, we design a fully polynomialtime approximation scheme (FPTAS) for a special case with a pipeline-structured workflow on a homogeneous cluster and a heuristic for the generalized problem with an arbitrary workflow on a heterogeneous cluster. The performance superiority of the proposed solution in terms of dynamic energy saving and deadline missing rate is illustrated by extensive simulation results in Hadoop/YARN in comparison with existing algorithms.

Identifier

85016286458 (Scopus)

Publication Title

Ceur Workshop Proceedings

ISSN

16130073

First Page

34

Last Page

43

Volume

1800

Grant

CNS-1560698

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS