DeAnomalyzer: Improving Determinism and Consistency in Anomaly Detection Implementations
Document Type
Conference Proceeding
Publication Date
1-1-2023
Abstract
Anomaly Detection (AD) is a popular unsupervised learning technique, but AD implementations are difficult to test, understand, and ultimately improve. Contributing factors for these difficulties include the lack of a specification to test against, output differences (on the same input) between toolkits that supposedly implement the same AD algorithm, and no linkage between learning parameters and undesirable outcomes. We have implemented DeAnomalyzer, a black-box tool that improves AD reliability by addressing two issues: nondeterminism (wide output variations across repeated runs of the same implementation on the same dataset) and inconsistency (wide output variations between toolkits on the same dataset). Specifically, DeAnomalyzer uses a feedback-directed, gradient descent-like approach to search for toolkit parameter settings that maximize determinism and consistency. DeAnomalyzer can operate in two modes: univariate, without ground truth, targeted to general users, and bivariate, with ground truth, targeted to algorithm designers and developers. We evaluated DeAnomalyzer on 54 AD datasets and the implementations of four AD algorithms in three popular ML toolkits: MATLAB, R, and Scikit-learn. The evaluation has revealed that DeAnomalyzer is effective at increasing determinism and consistency without sacrificing performance, and can even improve performance.
Identifier
85172272695 (Scopus)
ISBN
[9798350336290]
Publication Title
Proceedings 5th IEEE International Conference on Artificial Intelligence Testing Aitest 2023
External Full Text Location
https://doi.org/10.1109/AITest58265.2023.00012
First Page
17
Last Page
25
Grant
CCF-2007730
Fund Ref
National Science Foundation
Recommended Citation
Ahmed, Muyeed and Neamtiu, Iulian, "DeAnomalyzer: Improving Determinism and Consistency in Anomaly Detection Implementations" (2023). Faculty Publications. 2052.
https://digitalcommons.njit.edu/fac_pubs/2052
