Abstract
Large Multimodal Models (LMMs) struggle to adapt varying computational budgets due to numerous visual tokens. Previous methods attempted to reduce the number of visual tokens before or within LLMs. However, these strategies inevitably result in the loss of visual semantic. To address these issues, we introduce FMVR, a plug-and-play and extremely simple Frequency-Modulated Visual Restoration strategy to boost the reasoning ability of LMMs under visual token reduction. Specifically, FMVR disentangles the visual representation of fewer visual tokens into low- and high-frequency components through AvgPool and MaxPool. The derived frequencies are subsequently modulated using lightweight learnable parameters. The high-frequency from AvgPool acts as a saliency filter to enhance saliency visual semantics, while the low-frequency from MaxPool acts as an anti-saliency filter to strengthen weak visual semantics. It enables the preservation of visual semantics dominated by few visual tokens and the restoration of diluted visual semantics. Additionally, we inject FMVR into Matryoshka Representation Learning to learn coarse-to-fine visual token sets, thus enabling to elastically adjust the number of visual tokens during inference while maintaining comparable performance. Experiments across 10 image-based and 4 video-based bench marks demonstrate that FMVR-LLaVA reduce the FLOPs of LLaVA-1.5-7B by 89%, while maintaining almost 100% of the original accuracy. The code will be open.
Abstract (translated)
大型多模态模型（LMMs）在适应不同的计算预算时面临挑战，这主要是由于存在大量的视觉标记。之前的方法试图通过减少输入到LLMs之前的视觉标记数量来解决这个问题。然而，这些策略不可避免地会导致视觉语义的丢失。为了解决这些问题，我们引入了FMVR（Frequency-Modulated Visual Restoration），这是一种插件式且极其简单的策略，旨在增强在视觉标记减少情况下的LMM推理能力。 具体来说，FMVR通过使用AvgPool和MaxPool将较少视觉标记的视觉表示分解为低频和高频分量。随后，利用轻量级可学习参数对这些频率进行调制。来自AvgPool的高频部分充当了显著性过滤器，用于增强显著性的视觉语义；而来自MaxPool的低频部分则充当非显著性过滤器，用来加强弱化的视觉语义。这使得在使用少量视觉标记的情况下保留主导视觉语义，并恢复稀释的视觉语义成为可能。 此外，我们将FMVR整合到Matryoshka表示学习中，以从粗到细地学习视觉令牌集，从而能够在推理过程中灵活调整视觉令牌的数量，同时保持与原模型相当的表现力。跨10个基于图像和4个基于视频的基准测试的实验表明，在减少LLaVA-1.5-7B的FLOPs（89%）的同时，FMVR-LLaVA能够几乎维持原始准确性的100%不变。代码将会公开。
URL
https://arxiv.org/abs/2603.11220
