Mitigating Shortcuts in Language Models with Soft Label Encoding
Document Type
Conference Proceeding
Publication Date
1-1-2024
Abstract
Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). First, we train a teacher model to quantify each sample's degree of relying on shortcuts. Then, we encode this shortcut degree into a dummy class and use it to smooth the original ground truth labels, generating soft labels. These soft labels are used to train a more robust student model that reduces spurious correlations between shortcut features and certain classes. Extensive experiments on two NLU benchmark tasks via two language models demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy. Our code is available at https://github.com/ZiruiHE99/sle.
Identifier
85195900515 (Scopus)
ISBN
[9782493814104]
Publication Title
2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings
First Page
11341
Last Page
11348
Recommended Citation
He, Zirui; Deng, Huiqi; Zhao, Haiyan; Liu, Ninghao; and Du, Mengnan, "Mitigating Shortcuts in Language Models with Soft Label Encoding" (2024). Faculty Publications. 975.
https://digitalcommons.njit.edu/fac_pubs/975
