Large Scale String Analytics in Arkouda
Document Type
Conference Proceeding
Publication Date
1-1-2021
Abstract
Large scale data sets from the web, social networks, and bioinformatics are widely available and can often be rep-resented by strings and suffix arrays are highly efficient data structures enabling string analysis. But, our personal devices and corresponding exploratory data analysis (EDA) tools cannot handle big data sets beyond the local memory. Arkouda is a framework under early development that brings together the productivity of Python at the user side with the high-performance of Chapel at the server-side. In this paper, an efficient suffix array data structure design and integration method are given first. A suffix array algorithm library integration method instead of one single suffix algorithm is presented to enable runtime performance optimization in Arkouda since different suffix array algorithms may have very different practical performances for strings in various applications. A parallel suffix array construction algorithm framework is given to further exploit hierarchical parallelism on multiple locales in Chapel. A corresponding benchmark is developed to evaluate the feasibility of the provided suffix array integration method and measure the end-To-end performance. Experimental results show that the proposed solution can provide data scientists an easy and efficient method to build suffix arrays with high performance in Python. All our codes are open source and available from GitHub (https://github.com/Bader-Research/arkouda/tree/string-suffix-Array-functionality).
Identifier
85123503545 (Scopus)
ISBN
[9781665423694]
Publication Title
2021 IEEE High Performance Extreme Computing Conference Hpec 2021
External Full Text Location
https://doi.org/10.1109/HPEC49654.2021.9622810
Grant
CCF-2109988
Fund Ref
National Science Foundation
Recommended Citation
Du, Zhihui; Rodriguez, Oliver Alvarado; and Bader, David A., "Large Scale String Analytics in Arkouda" (2021). Faculty Publications. 4503.
https://digitalcommons.njit.edu/fac_pubs/4503