Performance Modeling and Prediction of Big Data Workflows: An Exploratory Analysis

Document Type

Conference Proceeding

Publication Date

8-1-2020

Abstract

Many next-generation scientific and business applications feature large-scale data-intensive workflows, which require massive computing resources for execution on high-performance clusters in cloud environments. Such computing resources (e.g., VCores and virtual memory) requested through parameter setting in big data systems, if not fully utilized by workloads, are simply wasted due to the nature of exclusive access made possible by containerization. This necessitates accurate modeling and prediction of workflow performance to make an effective recommendation of appropriate parameter settings to end users. However, it is challenging to determine optimal workflow and system configurations due to the large parameter space and the interaction between various technology layers of big data systems. Towards this goal, we propose a machine learning-based feature selection method to identify influential parameters based on historical performance measurements of Spark-based computing workloads executed in big data systems with YARN. We first identify a comprehensive set of parameters across multiple layers in the big data technology stack including workflow input structure, Spark computing engine, and YARN resource management. We then conduct an in-depth exploratory analysis of their individual and coupled impact on workflow performance, and develop a performance-influence model using random forest for prediction. Experimental results show that the proposed approach identifies important features for performance modeling and achieves high accuracy in performance prediction.

Identifier

85093855591 (Scopus)

ISBN

[9781728166070]

Publication Title

Proceedings International Conference on Computer Communications and Networks ICCCN

External Full Text Location

https://doi.org/10.1109/ICCCN49398.2020.9209715

ISSN

10952055

Volume

2020-August

Grant

1828123

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS