Document Type
Dissertation
Date of Award
12-31-2021
Degree Name
Doctor of Philosophy in Computing Sciences - (Ph.D.)
Department
Computer Science
First Advisor
Chase Qishi Wu
Second Advisor
Ali Mili
Third Advisor
Zhi Wei
Fourth Advisor
Jing Li
Fifth Advisor
Dantong Yu
Abstract
A wide spectrum of big data applications in science, engineering, and industry generate large datasets, which must be managed and processed in a timely and reliable manner for knowledge discovery. These tasks are now commonly executed in big data computing systems exemplified by Hadoop based on parallel processing and distributed storage and management. For example, many companies and research institutions have developed and deployed big data systems on top of NoSQL databases such as HBase and MongoDB, and parallel computing frameworks such as MapReduce and Spark, to ensure timely data analyses and efficient result delivery for decision making and business intelligence.
This dissertation investigates and addresses two main challenges in such big data systems: i) performance optimization for distributed information composition, and ii) performance modeling and prediction of big data applications. To address the first challenge, analytical cost models are constructed to formulate a Distributed Information Composition problem in Big Data Systems, referred to as DIC-BDS, to aggregate multiple datasets stored as data blocks in Hadoop Distributed File System (HDFS) using a composition operator of specific complexity to produce one final output. DIC-BDS is rigorously proved to be NP-complete, and two heuristic algorithms are proposed. Extensive experiments are conducted with various compo-sition operators of commonly considered degrees of complexity, and experimental results illustrate the performance superiority of the proposed solutions over existing methods. To address the second challenge, a class of regression-based machine learning models is proposed to predict the execution performance of Spark-HBase applications in Hadoop. Accurate performance modeling and prediction are critical to optimizing application performance through strategic resource allocation with suitable parameter settings and also to providing an effective recommendation of optimal system configurations to end users. An in-depth exploratory analysis is conducted to identify an exhaustive set of system parameters across multiple technology layers including Spark and HBase, and examine their effects on the execution time of Spark-HBase applications. Based on these analysis results, a subset of critical parameters is selected to develop a performance predictor using regression-based machine learning. Experimental results show that the resulted predictor achieves high accuracy with different algorithms in comparison.
Recommended Citation
AlQuwaiee, Haifa, "On performance optimization and prediction of parallel computing frameworks in big data systems" (2021). Dissertations. 1564.
https://digitalcommons.njit.edu/dissertations/1564
