Abstract
Challenges in cross-learning involve inhomogeneous or even inadequate amount of training data and lack of resources for retraining large pretrained models. Inspired by transfer learning techniques in NLP, adapters and prefix tuning, this paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T, that adapts transformer-based models to new or missing information. We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning. Training becomes substantially efficient as the backbone and other plugins do not need to be finetuned along with these additions. Comparative and ablation studies on three datasets Epic-Kitchens-100, MPIIGroupInteraction and UDIVA v0.5 show efficacy of this framework on different recording settings and tasks. With only 12.8% trainable parameters compared to the backbone to process video input and only 22.3% trainable parameters for two additional modalities, we achieve comparable and even better results than the state-of-the-art. CM3T has no specific requirements for training or pretraining and is a step towards bridging the gap between a general model and specific practical applications of video classification.
Abstract (translated)
跨领域学习面临的挑战包括异质或甚至不足的训练数据，以及重新训练大型预训练模型资源匮乏的问题。受自然语言处理（NLP）中迁移学习技术的启发，本文提出了一种新的与模型无关的插件架构——CM3T，用于跨领域学习，该架构可以使基于Transformer的模型适应新出现或缺失的信息。我们引入了两种适配器模块：多头视觉适配器用于转移学习和跨注意力适配器用于多模态学习。由于不需要对骨干网络和其他插件进行微调，因此训练效率显著提高。在Epic-Kitchens-100、MPIIGroupInteraction 和 UDIVA v0.5 数据集上的比较与消融研究表明，该框架在不同的记录设置和任务上均表现出良好的效果。仅使用占骨干网络12.8%的可训练参数来处理视频输入，并且针对两种额外模态只需22.3%的可训练参数，CM3T就能达到与当前最佳方法相当甚至更好的结果。此外，CM3T对于训练或预训练没有特定要求，并为通用模型和具体应用场景之间的差距搭建了一座桥梁。
URL
https://arxiv.org/abs/2501.03332
