To address the challenges of deploying deep learning models in resource-constrained environments and to ensure the compatibility between diagnostic accuracy and compression ratio, we propose a lightweight Multi-Teacher Knowledge Distillation Fault Diagnosis (BFD-KD) method for rolling bearing fault diagnosis. This method employs a dynamic weighting mechanism based on cross-entropy loss between teacher model predictions and true labels, while integrating intermediate-layer features from multiple teacher models to effectively resolve issues of knowledge competition and fusion. Experimental results on the publicly available Huazhong University of Science and Technology bearing data set demonstrate, in the case of small samples, the BFD-KD achieves a significantly higher compression ratio while maintaining superior diagnostic accuracy compared to existing multi-teacher knowledge distillation methods. Furthermore, its lightweight design and robustness are validated on the UORED-VAFCLS industrial data set, confirming its effectiveness and practicality for real-world deployment in resource-limited devices.
BajestaniMFYangY (2020) TKD: Temporal knowledge distillation for active perception. In: Proceedings of the IEEE winter conference on applications of computer vision, Snowmass Village, CO, 1–5 March, pp. 953–962. New York: IEEE.
2.
BhardwajSSrinivasanMKhapraM, et al. (2019) Efficient video classification using fewer frames. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, 15–20 June, pp. 354–363. New York: IEEE.
3.
ChenHWangYXuC, et al. (2019) Data-free learning of student networks. In: Proceedings of the IEEE/CVF international conference on computer vision, Seoul, South Korea, 27 October–2 November, pp. 3514–3522. New York: IEEE.
4.
DaiXJiangZWuZ, et al. (2021) General instance distillation for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, virtual event, 19–25 June, pp. 7842–7851. New York: IEEE.
5.
DuSYouSLiX, et al. (2020) Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. Advances in Neural Information Processing Systems33: 12345–12355.
HintonGVinyalsODeanJ (2015) Distilling the knowledge in a neural network. Computer Science14(7): 38–39.
8.
HuangZYangSLinW, et al. (2022) Summary of knowledge distillation research. Chinese Journal of Computers45(3): 624–653.
9.
LiQJinSYanJ, et al. (2017) Mimicking very efficient network for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, 21–26 July, pp. 6356–6364. New York: IEEE.
10.
LiuXHePChenW, et al. (2020) Improving multi-task deep neural networks via knowledge distillation for natural language understanding. In: Proceedings of the IEEE international conference on acoustics, speech, and signal processing (ICASSP), Barcelona, 20 April, pp. 7419–7423. New York: IEEE.
11.
LuRLiuSGongZ, et al. (2024) Lightweight knowledge distillation-based transfer learning framework for rolling bearing fault diagnosis. Sensors24(6): 1758.
12.
SehriMDumondP (2023) University of Ottawa rolling-element dataset: Vibration and acoustic faults under constant load and speed conditions (UORED-VAFCLS). Available at: https://data.mendeley.com/datasets/y2px5tg92h/5
13.
ShenCXueMWangX, et al. (2019) Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. In: Proceedings of the IEEE international conference on computer vision, Seoul, South Korea, 20 August, pp. 3504–3513. New York: IEEE.
14.
TangJ (2023) Diagnosis of steel surface defects based on multi-teacher knowledge distillation network. Information and Computer35(11): 217–220.
15.
TranLVeelingBSRothK, et al. (2021) Hydra: Preserving ensemble diversity for model distillation. In: Proceedings of the 2021 conference on empirical methods in natural language processing (EMNLP), Virtual Event / Punta Cana, Dominican Republic, 711 November, pp. 4093–4107. Vienna: ACL Press.
16.
WangFVSuW (2018) Fault Diagnosis and Life Prediction of Rolling Bearing. Beijing, China: Science Press.
17.
XiaYZhangSWangJ, et al. (2022) One-stage self-distillation guided knowledge transfer for long-tailed visual recognition. International Journal of Intelligent Systems37(12): 11893–11908.
18.
XiaoGYangCChenY, et al. (2020) Convolutional neural network quantization algorithm based on weight interaction idea. Application of Electronic Technology46(10): 39–41.
19.
YeRShaoJZhanW, et al. (2023) Research on knowledge distillation method of news text classification based on BERT-CNN. Application of Electronic Technology49(1): 8–13.
20.
YeXJiangRTianX, et al. (2024) Knowledge distillation via multi-teacher feature ensemble. IEEE Signal Processing Letters31: 566–570.
21.
YouSXuCXuC, et al. (2017) Learning from multiple teacher networks. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, Halifax, NS, Canada, 13–17 August, pp. 1285–1294. New York: IEEE.
22.
ZhangHChenDWangC, et al. (2022) Confidence-aware multi-teacher knowledge distillation. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, 23–27 May, pp. 4498–4502. New York: IEEE.
23.
ZhaoCZioEShenW, et al. (2024) Domain generalization for cross-domain fault diagnosis: An application-oriented perspective and a benchmark study. Reliability Engineering & System Safety245: 109964.
