Transfer Learning from BERT to Support Insertion of New Concepts into SNOMED CT

Document Type

Article

Publication Date

1-1-2019

Abstract

With advances in Machine Learning (ML), neural network-based methods, such as Convolutional/Recurrent Neural Networks, have been proposed to assist terminology curators in the development and maintenance of terminologies. Bidirectional Encoder Representations from Transformers (BERT), a new language representation model, obtains state-of-the-art results on a wide array of general English NLP tasks. We explore BERT's applicability to medical terminology-related tasks. Utilizing the "next sentence prediction" capability of BERT, we show that the Fine-tuning strategy of Transfer Learning (TL) from the BERTBASE model can address a challenging problem in automatic terminology enrichment - insertion of new concepts. Adding a pre-training strategy enhances the results. We apply our strategies to the two largest hierarchies of SNOMED CT, with one release as training data and the following release as test data. The performance of the combined two proposed TL models achieves an average F1 score of 0.85 and 0.86 for the two hierarchies, respectively.

Identifier

85083755680 (Scopus)

Publication Title

AMIA Annual Symposium Proceedings AMIA Symposium

e-ISSN

1942597X

PubMed ID

32308910

First Page

1129

Last Page

1138

Volume

2019

This document is currently not available here.

Share

COinS