Skill-Transferring Knowledge Distillation Method
Document Type
Article
Publication Date
11-1-2023
Abstract
Knowledge distillation is a deep learning method that mimics the way that humans teach, i.e., a teacher network is used to guide the training of a student one. Knowledge distillation can generate an efficient student network to facilitate deployment in resource-constrained edge computing devices. Existing studies have typically mined knowledge from a teacher network and transferred it to a student one. The latter can only passively receive knowledge but cannot understand how the former acquires the knowledge, thus limiting the latter's performance improvement. Inspired by the old Chinese saying 'Give a man a fish and you feed him for a day; teach a man how to fish and you feed him for a lifetime,' this work proposes a Skill-transferring Knowledge Distillation (SKD) method to boost a student network's ability to create new valuable knowledge. SKD consists of two main meta-learning networks: Teacher Behavior Teaching and Teacher Experience Teaching. The former captures the process of a teacher network's learning behavior in the hidden layers and can predict the teacher network's subsequent behavior based on previous ones. The latter models the optimal empirical knowledge of a teacher network's output layer at each learning stage. With their help, a teacher network can provide its actions to a student one in the subsequent behavior and its optimal empirical knowledge in the current stage. SKD's performance is verified through its application to multiple object recognition tasks and comparison with the state of the art.
Identifier
85159703255 (Scopus)
Publication Title
IEEE Transactions on Circuits and Systems for Video Technology
External Full Text Location
https://doi.org/10.1109/TCSVT.2023.3271124
e-ISSN
15582205
ISSN
10518215
First Page
6487
Last Page
6502
Issue
11
Volume
33
Grant
6020210083K
Fund Ref
National Natural Science Foundation of China
Recommended Citation
Yang, Shunzhi; Xu, Liuchi; Zhou, Mengchu; Yang, Xiong; Yang, Jinfeng; and Huang, Zhenhua, "Skill-Transferring Knowledge Distillation Method" (2023). Faculty Publications. 1350.
https://digitalcommons.njit.edu/fac_pubs/1350
