BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Document Type

Conference Proceeding

Publication Date

7-2-2018

Abstract

Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.

Identifier

85063320038 (Scopus)

ISBN

[9781728101941]

Publication Title

Proceedings of Indis 2018 Innovating the Network for Data Intensive Science Held in Conjunction with Sc 2018 the International Conference for High Performance Computing Networking Storage and Analysis

External Full Text Location

https://doi.org/10.1109/INDIS.2018.00011

First Page

75

Last Page

84

Fund Ref

U.S. Department of Energy

This document is currently not available here.

Share

COinS