Abstract
Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at this https URL
Abstract (translated)
从可见图像中估计人类视线目标是机器人理解人类注意力的关键任务，然而开发通用的神经网络架构和训练模式仍然具有挑战性。虽然近期在预训练视觉基础模型方面的进展为定位注视点提供了有前景的方法，但整合多模态线索——包括眼睛、头部姿态、手势及上下文特征——则需要灵活高效的解码机制。 受混合专家（Mixture-of-Experts, MoE）在大型视觉语言模型中适应领域专业知识的启发，我们提出了GazeMoE，这是一种新颖的端到端框架。该框架通过MoE模块选择性地利用一个冻结的基础模型中的与注视点相关的线索。为了应对视线目标分类中的类别不平衡问题（即画面内与画面外）以及增强其鲁棒性，GazeMoE引入了类平衡辅助损失，并采用了特定区域裁剪和光度变换等策略性的数据增强方法。 在基准数据集上进行的广泛实验表明，我们的GazeMoE框架实现了最先进的性能，在具有挑战性的视线估计任务中超越现有方法。相关代码与预训练模型可以在以下网址获取：[此 URL]
URL
https://arxiv.org/abs/2603.06256
