DFS-container: Achieving containerized block I/O for distributed file systems

Document Type

Conference Proceeding

Publication Date

9-24-2017

Abstract

Today BigData systems commonly use resource management systems such as TORQUE, Mesos, and Google Borg to share the physical resources among users or applications. Enabled by virtualization, users can run their applications on the same node with low mutual interference. Container-based virtualizations (e.g., Docker and Linux Containers) offer a lightweight virtualization layer, which promises a near-native performance and is adopted by some Big-Data resource sharing platforms such as Mesos. Nevertheless, using containers to consolidate the I/O resources of shared storage systems is still at an early stage, especially in a distributed file system (DFS) such as Hadoop File System (HDFS). To overcome this issue, we propose a distributed middleware system, DFS-Container, by further containerizing DFS.We also evaluate and analyze the unfairness of using containers to proportionally allocate the I/O resource of DFS. Based on these analyses and evaluations, we propose and implement a new mechanism, IOPS-Regulator, which improve the fairness of proportional allocation by 74.4% on average.

Identifier

85032439257 (Scopus)

ISBN

[9781450350280]

Publication Title

Socc 2017 Proceedings of the 2017 Symposium on Cloud Computing

External Full Text Location

https://doi.org/10.1145/3127479.3132568

First Page

660

This document is currently not available here.

Share

COinS