A study of terminology auditors' performance for UMLS semantic type assignments

Document Type

Article

Publication Date

12-1-2012

Abstract

Auditing healthcare terminologies for errors requires human experts. In this paper, we present a study of the performance of auditors looking for errors in the semantic type assignments of complex UMLS concepts. In this study, concepts are considered complex whenever they are assigned combinations of semantic types. Past research has shown that complex concepts have a higher likelihood of errors. The results of this study indicate that individual auditors are not reliable when auditing such concepts and their performance is low, according to various metrics. These results confirm the outcomes of an earlier pilot study. They imply that to achieve an acceptable level of reliability and performance, when auditing such concepts of the UMLS, several auditors need to be assigned the same task. A mechanism is then needed to combine the possibly differing opinions of the different auditors into a final determination. In the current study, in contrast to our previous work, we used a majority mechanism for this purpose. For a sample of 232 complex UMLS concepts, the majority opinion was found reliable and its performance for accuracy, recall, precision and the F-measure was found statistically significantly higher than the average performance of individual auditors. © 2012 Elsevier Inc.

Identifier

84869872304 (Scopus)

Publication Title

Journal of Biomedical Informatics

External Full Text Location

https://doi.org/10.1016/j.jbi.2012.05.006

ISSN

15320464

PubMed ID

22687822

First Page

1042

Last Page

1048

Issue

6

Volume

45

Grant

R01LM008445-01A2

Fund Ref

U.S. National Library of Medicine

This document is currently not available here.

Share

COinS