Abstract
Knowledge Distillation (KD) is a widespread technique for compressing the knowledge of large models into more compact and efficient models. KD has proved to be highly effective in building well-performing low-complexity Acoustic Scene Classification (ASC) systems and was used in all the top-ranked submissions to this task of the annual DCASE challenge in the past three years. There is extensive research available on establishing the KD process, designing efficient student models, and forming well-performing teacher ensembles. However, less research has been conducted on investigating which teacher model attributes are beneficial for low-complexity students. In this work, we try to close this gap by studying the effects on the student's performance when using different teacher network architectures, varying the teacher model size, training them with different device generalization methods, and applying different ensembling strategies. The results show that teacher model sizes, device generalization methods, the ensembling strategy and the ensemble size are key factors for a well-performing student network.
Abstract (translated)
知识蒸馏（KD）是一种将大型模型的知识压缩到更紧凑、高效的模型中的广泛技术。在过去的三年里，KD已被证明是构建高性能低复杂度的声场景分类（ASC）系统的一种非常有效的方法，并且被年度DCASE挑战赛中该任务的所有顶级提交所采用。已有大量研究探讨了建立蒸馏过程、设计高效的学生模型以及形成表现良好的教师模型组合的方法。然而，关于探究哪些教师模型属性对低复杂度学生有益的研究较少。在这项工作中，我们尝试填补这一空白，通过使用不同的教师网络架构、改变教师模型大小、用不同的设备泛化方法对其进行训练，并应用不同的集合策略来研究对学生性能的影响。结果表明，教师模型的规模、设备泛化方法、组合策略和组合大小是构建表现良好的学生网络的关键因素。
URL
https://arxiv.org/abs/2503.11363
