"Large Scale String Analytics in Arkouda" by Zhihui Du, Oliver Alvarado Rodriguez et al.
 

Large Scale String Analytics in Arkouda

Document Type

Conference Proceeding

Publication Date

1-1-2021

Abstract

Large scale data sets from the web, social networks, and bioinformatics are widely available and can often be rep-resented by strings and suffix arrays are highly efficient data structures enabling string analysis. But, our personal devices and corresponding exploratory data analysis (EDA) tools cannot handle big data sets beyond the local memory. Arkouda is a framework under early development that brings together the productivity of Python at the user side with the high-performance of Chapel at the server-side. In this paper, an efficient suffix array data structure design and integration method are given first. A suffix array algorithm library integration method instead of one single suffix algorithm is presented to enable runtime performance optimization in Arkouda since different suffix array algorithms may have very different practical performances for strings in various applications. A parallel suffix array construction algorithm framework is given to further exploit hierarchical parallelism on multiple locales in Chapel. A corresponding benchmark is developed to evaluate the feasibility of the provided suffix array integration method and measure the end-To-end performance. Experimental results show that the proposed solution can provide data scientists an easy and efficient method to build suffix arrays with high performance in Python. All our codes are open source and available from GitHub (https://github.com/Bader-Research/arkouda/tree/string-suffix-Array-functionality).

Identifier

85123503545 (Scopus)

ISBN

[9781665423694]

Publication Title

2021 IEEE High Performance Extreme Computing Conference Hpec 2021

External Full Text Location

https://doi.org/10.1109/HPEC49654.2021.9622810

Grant

CCF-2109988

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS