Optimal subspace dimensionality for k-NN search on clustered datasets
Document Type
Article
Publication Date
1-1-2004
Abstract
Content based retrieval is an important paradigm in multimedia applications. It heavily relies on k-Nearest-Neighbor (k-NN) queries applied to high dimensional feature vectors representing objects. Dimensionality Reduction (DR) of high-dimensional datasets via Principal Component Analysis - PCA is an effective method to reduce the cost of processing k-NN queries on multi-dimensional indices. The distance information loss is quantifiable by the Normalized Mean Square Error (NMSE), which is determined by the number of retained dimensions (n). For smaller n the cost of accessing the index (an SR-tree is used in our study) for k-NN search is lower, but the postprocessing cost to achieve exact query processing is higher. The optimum value nopt can be determined experimentally by considering cost as a function of n. We concern ourselves with a local DR method, which applies DR to clusters of the original dataset. Clusters are obtained via a PCA-friendly clustering method, which also determines the number of clusters. For a given NMSE we use an algorithm developed in conjunction with the Clustered SVD - CSVD method to determine the vector of the number of dimensions retained in all clusters (n). The NMSE is varied to determine the optimum n, which minimizes the number of accessed pages. To verify the robustness of our methodology we experimented with one synthetic and three real-world datasets. It is observed that the NMSE yielding the optimum n varies over a narrow range and that the optimal value is expected to be applicable to datasets with similar characteristics. © Springer-Verlag Berlin Heidelberg 2004.
Identifier
35048843866 (Scopus)
ISBN
[3540229361, 9783540229360]
Publication Title
Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics
External Full Text Location
https://doi.org/10.1007/978-3-540-30075-5_20
e-ISSN
16113349
ISSN
03029743
First Page
201
Last Page
211
Volume
3180
Recommended Citation
Li, Yue; Thomasian, Alexander; and Zhang, Lijuan, "Optimal subspace dimensionality for k-NN search on clustered datasets" (2004). Faculty Publications. 20557.
https://digitalcommons.njit.edu/fac_pubs/20557
