"Enhancing Proportional IO Sharing on Containerized Big Data File Syste" by Dan Huang, Jun Wang et al.
 

Enhancing Proportional IO Sharing on Containerized Big Data File Systems

Document Type

Article

Publication Date

12-1-2021

Abstract

Big Data platforms recently employ resource management systems, such as YARN, Mesos, and Google Borg, to provision computational resources. These systems adopt containerization to share the computing resources in a multi-tenant setting with low performance overhead and interference. However, it may be observed that tenants often interfere with each other on the underlying Big Data File Systems (BDFS), e.g., Hadoop File System, which have been widely deployed as a persistent layer in current data centers. A solution with systematic generality is to containerize BDFS itself to isolate and allocate its IO sources to multiple tenants. To this end, we conduct analysis on the ineffectiveness of proportionally sharing BDFS IO resource via containerization. This ineffectiveness is due to the scheduler of containerization in 'pseudo-starvation' status, in which most of IO requests are backlogged in BDFS rather than in containerization scheduler. Without enough backlogged IO requests, existing schedulers might have to maximize device utilization rather than enforce proportional sharing policy. To resolve this ineffectiveness issue, we develop a cross-layer system called BDFS-Container, which containerizes BDFS at the Linux block IO level. Central to BDFS-Container, we propose and design a proactive IOPS throttling-based mechanism named IOPS Regulator, which achieves a trade-off between maximizing IO utilization and accurately proportional IO sharing. The evaluation results show that our method can improve proportionally sharing BDFS IO resources by 74.4 percent on average.

Identifier

85098783193 (Scopus)

Publication Title

IEEE Transactions on Computers

External Full Text Location

https://doi.org/10.1109/TC.2020.3037078

e-ISSN

15579956

ISSN

00189340

First Page

2083

Last Page

2097

Issue

12

Volume

70

Grant

1907765

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS