Using Generative Large Language Models for Hierarchical Relationship Prediction in Medical Ontologies

Document Type

Conference Proceeding

Publication Date

1-1-2024

Abstract

This study extends the exploration of ontology enrichment by evaluating the performance of various open-sourced Large Language Models (LLMs) on the task of predicting hierarchical relationships (IS-A) in medical ontologies including SNOMED CT Clinical Finding and Procedure hierarchies and the human Disease Ontology. With the previous finetuned BERT models for hierarchical relationship prediction as the baseline, we assessed eight open-source generative LLMs for the same task. We observed only three models, without finetuning, demonstrated comparable or superior performance compared to the baseline BERT -based models. The best performance model OpenChat achieved a macro average F1 score of 0.96 (0.95) on SNOMED CT Clinical Finding (Procedure) hierarchy, an increase over 7% from the baseline 0.89 (0.85). On human Disease Ontology, OpenChat excels with an F1 score of 0.91, outperforming the second-best performance model Vicuna (0.84). Notably, some LLMs prove unsuitable for hierarchical relationship prediction tasks or appliable for concept placement of medical ontologies. We also explored various prompt templates and ensemble techniques to uncover potential confounding factors in applying LLMs for IS-A relation predictions for medical ontologies.

Identifier

85203690166 (Scopus)

ISBN

[9798350383737]

Publication Title

Proceedings - 2024 IEEE 12th International Conference on Healthcare Informatics, ICHI 2024

External Full Text Location

https://doi.org/10.1109/ICHI61247.2024.00040

First Page

248

Last Page

256

Grant

2018575

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS