Abstract
Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as `Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense this http URL integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during this http URL existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student this http URL conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models.
Abstract (translated)
密集特征，针对不同的业务场景进行定制，是短时视频分类的基本要求。然而，它们的复杂性、特定的适应要求和高的计算成本使它们变得资源密集且难以在线推理时使用。因此，这些密集特征被归类为“特权密集特征”。与此同时，端到端多模态模型已经在许多计算机视觉任务中取得了良好的结果。在工业应用中，优先考虑端到端多模态特征，可以提高效率，但往往会导致历史特权密集特征的宝贵信息丢失。为了同时保持效率和可管理资源成本，我们提出了Confidence-aware Privileged Feature Distillation（CPFD），它通过在现有的特权特征提取方法中自适应地蒸馏特权特征，使端到端多模态模型的特征具有可塑性。通过应用教师模型的置信分数，CPFD可以动态地减轻学生模型的性能方差。我们在五个多样化的任务上进行了广泛的离线实验，结果表明，与端到端多模态模型（X-VLM）相比，CPFD可以将视频分类F1得分提高6.76%；与普通PFD相比，CPFD可以将性能差距减少84.6%。在实际应用中，CPFD的效果得到了进一步的验证，我们的框架已经被部署在多个生产系统上，支持了数十个模型。
URL
https://arxiv.org/abs/2410.03038
