Abstract
Audio-visual Zero-Shot Learning (ZSL) has attracted significant attention for its ability to identify unseen classes and perform well in video classification tasks. However, modal imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Some studies have attempted to address this issue by modifying parameter gradients, but two challenges still remain: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where sample contributions within a modality vary significantly. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual ZSL. Our approach introduces a Quality-Discrepancy Mitigation Attention (QDMA) unit to minimize redundant information in the high-quality modality and a Contrastive Sample-level Gradient Modulation (CSGM) block to adjust gradient magnitudes and balance content discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrates DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules.
Abstract (translated)
音频视觉零样本学习（ZSL）因其能够识别未见过的类别并在视频分类任务中表现出色而受到了广泛关注。然而，(G)ZSL中的模态不平衡问题导致过度依赖最优模态，降低了对未见类别的辨别能力。一些研究试图通过修改参数梯度来解决这个问题，但仍存在两个挑战：(a) 质量差异，即对于同一概念，不同模态提供的信息数量和质量有所不同。(b) 内容差异，在同一个模态内样本贡献的变化很大。为了解决这些问题，我们提出了一种差异感知注意力网络（DAAN），用于增强音频视觉ZSL。我们的方法引入了一个质量差异缓解注意单元（QDMA）来减少高质量模态中的冗余信息，并且使用对比样本级梯度调节模块（CSGM）来调整梯度大小并平衡内容差异。我们在CSGM中通过整合优化和收敛率来量化模态贡献，实现更精确的梯度调节。实验表明，DAAN在基准数据集上实现了最先进的性能，消融研究验证了各个模块的有效性。
URL
https://arxiv.org/abs/2412.11715
