REMOLD: An efficient model-based clustering algorithm for large datasets with spark
Document Type
Conference Proceeding
Publication Date
7-2-2017
Abstract
Density-based clustering algorithms have the distinctive advantage of discovering arbitrarily shaped clusters, but they usually require a procedure to compute the distance between every pair of data points, and this procedure is prohibitive for large datasets since it has quadratic computation complexity. In this paper, we propose a new distributed clustering algorithm, named REstore MOdel with Local Density estimation (REMOLD). Firstly, REMODL applies a balanced partitioning method to evenly divide an large dataset based on Local Sensitive Hashing (LSH). Then, it locally clusters each partition of the dataset, and uses a Gaussian model to represent each local cluster based on the observation that the density distribution of each local cluster shares similar shape with Gaussian distribution. Finally, these models are aggregated on a server where REMOLD restores global clusters based on these local Gaussian models. More specifically, model connection, which measures the density connectivity between two models, are defined to merge local models with an optimized procedure. In this aggregation, REMOLD requires low cost of network transmission for local Gaussian models, since the number of Gaussian models is often less than that of core objects for each partition. We evaluate REMOLD on three synthetic datasets and three real-world datasets on Spark, and the experiment results demonstrate that REMOLD is efficient and effective to find out clusters with complex shapes and it outperforms the established methods.
Identifier
85048373524 (Scopus)
ISBN
[9781538621295]
Publication Title
Proceedings of the International Conference on Parallel and Distributed Systems ICPADS
External Full Text Location
https://doi.org/10.1109/ICPADS.2017.00057
ISSN
15219097
First Page
376
Last Page
383
Volume
2017-December
Grant
17511102900
Fund Ref
Natural Science Foundation of Beijing Municipality
Recommended Citation
Liang, Mingfei; Li, Qingyong; Geng, Yangli Ao; Wang, Jianzhu; and Wei, Zhi, "REMOLD: An efficient model-based clustering algorithm for large datasets with spark" (2017). Faculty Publications. 9455.
https://digitalcommons.njit.edu/fac_pubs/9455
