Abstract
Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D image slices from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R2 up to 0.941) and recover up to 87.8% of downstream performance using only 10 features (99.4% dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation. (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: this https URL.
Abstract (translated)
视觉基础模型（FMs）在医学影像领域达到了最先进的性能。然而，它们将信息编码在抽象的潜在表示中，临床医生无法对此进行 interrogate 或验证。本研究旨在探究稀疏自编码器（SAEs），用人类可解释的稀疏特征替代不透明的FM图像表示。我们在BiomedParse（生物医学）和DINOv3（通用）的嵌入表示上训练SAEs，使用TotalSegmentator数据集中的909,873张CT和MRI二维图像切片。我们发现学习到的稀疏特征：（a）以高保真度重建原始嵌入表示（R²最高达0.941），且仅使用10个特征（降低99.4%维度）即可恢复高达87.8%的下游任务性能；（b）在图像检索任务中保持语义保真度；（c）对应可通过基于大语言模型（LLM）的自动解释用语言表达的具体概念；（d）在零样本语言驱动图像检索中连接临床语言与抽象潜在表示。我们的工作表明SAEs是通向可解释的、概念驱动的医学视觉系统的一条有前景的路径。代码仓库：此https链接。
URL
https://arxiv.org/abs/2603.23794
