Abstract
Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.
Abstract (translated)
近年来人工智能的进展激励了众多通过脑信号理解和解码人类视觉系统的尝试。这些先前工作通常使用预训练的视觉模型从图像中提取语义和感知特征，并分别与神经活动进行对齐。然而，它们却未能解决两个关键挑战：（1）脑信号与图像在表征信息层级上存在天然差异而产生的模态差距；（2）语义和感知特征在神经活动中高度纠缠的事实。为应对这些问题，我们利用特别适合考虑信息量差异且具有几何特性的双曲空间——其两点间的测地线会自然向表征容量较低的原点弯曲。基于这些特性，我们提出了一种新颖框架——双曲特征插值（HyFI），该框架沿双曲测地线在语义与感知视觉特征间进行插值。这实现了感知与语义信息的融合与压缩，有效反映了脑信号有限的表达能力以及这些特征的纠缠本质，从而促进了脑特征与视觉特征间更优的对齐。我们证明HyFI在零样本脑到图像检索任务中达到了最先进性能，在THINGS-EEG数据集上Top-1准确率最高提升+17.3%，在THINGS-MEG数据集上最高提升+9.1%，超越了先前方法。
URL
https://arxiv.org/abs/2603.22721
