Abstract
We present a semantics modulated, multi scale Transformer for 3D gaze estimation. Our model conditions CLIP global features with learnable prototype banks (illumination, head pose, background, direction), fuses these prototype-enriched global vectors with CLIP patch tokens and high-resolution CNN tokens in a unified attention space, and replaces several FFN blocks with routed/shared Mixture of Experts to increase conditional capacity. Evaluated on MPIIFaceGaze, EYEDIAP, Gaze360 and ETH-XGaze, our model achieves new state of the art angular errors of 2.49°, 3.22°, 10.16°, and 1.44°, demonstrating up to a 64% relative improvement over previously reported results. ablations attribute gains to prototype conditioning, cross scale fusion, MoE and hyperparameter. Our code is publicly available at https://github. com/AIPMLab/Gazeformer.
Abstract (translated)
我们提出了一种语义调节的多尺度Transformer，用于3D注视点估计。我们的模型使用可学习的原型库（光照、头部姿态、背景和方向）来调整CLIP全局特征，并将这些经过丰富处理的全局向量与CLIP补丁标记和高分辨率CNN令牌在统一注意空间中融合在一起。此外，我们用路由/共享专家混合体替换了若干前馈网络块以增加条件容量。 我们在MPIIFaceGaze、EYEDIAP、Gaze360和ETH-XGaze数据集上对模型进行了评估，我们的模型分别实现了2.49°、3.22°、10.16°和1.44°的新最佳角误差，相较于先前报告的结果有了高达64%的相对改进。消融研究表明，原型调节、跨尺度融合、专家混合体以及超参数调整是性能提升的关键因素。 我们的代码在 https://github.com/AIPMLab/Gazeformer 上公开可用。
URL
https://arxiv.org/abs/2601.12316
