Abstract
Self-supervised and multimodal vision encoders learn strong visual representations that are widely adopted in downstream vision tasks and large vision-language models (LVLMs). However, downstream users often rely on third-party pretrained encoders with uncertain provenance, exposing them to backdoor attacks. In this work, we propose BackdoorIDS, a simple yet effective zero-shot, inference-time backdoor samples detection method for pretrained vision encoders. BackdoorIDS is motivated by two observations: Attention Hijacking and Restoration. Under progressive input masking, a backdoored image initially concentrates attention on malicious trigger features. Once the masking ratio exceeds the trigger's robustness threshold, the trigger is deactivated, and attention rapidly shifts to benign content. This transition induces a pronounced change in the image embedding, whereas embeddings of clean images evolve more smoothly across masking progress. BackdoorIDS operationalizes this signal by extracting an embedding sequence along the masking trajectory and applying density-based clustering such as DBSCAN. An input is flagged as backdoored if its embedding sequence forms more than one cluster. Extensive experiments show that BackdoorIDS consistently outperforms existing defenses across diverse attack types, datasets, and model families. Notably, it is a plug-and-play approach that requires no retraining and operates fully zero-shot at inference time, making it compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.
Abstract (translated)
自监督和多模态视觉编码器学习到的强大视觉表示在下游视觉任务和大型视觉语言模型（LVLM）中被广泛采用。然而，下游用户经常依赖来源不明的第三方预训练编码器，这使他们容易受到后门攻击。在这项工作中，我们提出了BackdoorIDS，这是一种简单而有效的零样本、推理时检测预训练视觉编码器中的后门样本的方法。 BackdoorIDS的设计动机来自两个观察结果：注意力劫持和恢复。在逐步输入屏蔽过程中，含有恶意触发特征的被篡改图像最初会集中注意于这些恶意特征上。一旦屏蔽比例超过了触发特征的鲁棒性阈值，该触发特征就会失效，注意力迅速转移到正常的内容上。这一转变会在图像嵌入中引起显著的变化，而干净图像的嵌入在屏蔽过程中则变化较为平缓。 BackdoorIDS通过沿着屏蔽轨迹提取一个嵌入序列并应用基于密度的聚类（如DBSCAN）来实现这一点。如果某个输入样本的嵌入序列形成了超过一个簇，则将其标记为被篡改。广泛的实验表明，无论攻击类型、数据集还是模型家族如何，BackdoorIDS都能持续超越现有的防御方法。 特别值得注意的是，BackdoorIDS是一个即插即用的方法，不需要重新训练，并且能够在推理时完全零样本操作，使其兼容包括CNNs、ViTs、CLIP和LLaVA-1.5在内的各种编码器架构。
URL
https://arxiv.org/abs/2603.11664
