Abstract
Multimodal large language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length token compression, disrupting volumetric continuity and obscuring subtle findings. We present Photon, a framework that represents 3D medical volumes with token sequences of variable length. Photon introduces instruction-conditioned token scheduling and surrogate gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use of visual evidence, Photon further applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.
Abstract (translated)
多模态大语言模型在临床视觉问答任务中前景广阔，但扩展到3D影像时受限于高昂的计算成本。现有方法常依赖2D切片或固定长度的令牌压缩，破坏了体积连续性并掩盖了细微病灶。我们提出Photon框架，该框架以可变长度的令牌序列表示3D医学影像。Photon引入了指令条件化令牌调度与替代梯度传播机制，可在训练和推理阶段自适应减少令牌数量，从而在降低计算成本的同时缓解冗余令牌导致的注意力稀释问题。其采用自定义反向传播规则与梯度恢复技术，使离散令牌丢弃过程仍可实现可微优化。为稳定令牌压缩并确保视觉证据的可靠利用，Photon进一步应用正则化目标来减轻纯语言偏差并提升可靠性。在多种医学视觉问答任务上的实验表明，Photon在减少资源消耗、加速训练与推理的同时，实现了最先进的准确率。
URL
https://arxiv.org/abs/2603.25155
