Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations

Document Type

Conference Proceeding

Publication Date

8-1-2021

Abstract

Self-organizing maps (SOMs) are a popular approach for neural network-based unsupervised learning. However the reliability of self-organizing map implementations has not been investigated. Using internal and external metrics, we define and check two basic SOM properties. First, determinism: A given SOM implementation should produce the same SOM when run repeatedly on the same training dataset. Second, consistency: Two SOM implementations should produce similar SOMs when presented with the same training dataset. We check these properties in four popular SOM implementations. We ran our approach on 381 popular datasets used in health, medicine, and other critical domains. We found that implementations violate these basic properties. For example, 375 out of 381 datasets have nondeterministic outcomes; for 51-92% of datasets, toolkits yield significantly different SOM clusterings; and clustering accuracy might be so inconsistent as to vary by a factor of four between toolkits. This undermines SOM reliability, and the reliability of results obtained via SOMs. Our study shines a light on what to expect, in practice, when running actual SOM implementations. Our findings suggest that for critical applications, SOM users should not take reliability for granted; rather, multiple runs and different toolkits should be considered and compared.

Identifier

85118797852 (Scopus)

ISBN

[9781665434812]

Publication Title

Proceedings 3rd IEEE International Conference on Artificial Intelligence Testing Aitest 2021

External Full Text Location

https://doi.org/10.1109/AITEST52744.2021.00026

First Page

85

Last Page

92

Grant

CCF-2007730

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS