Abstract
Current 3D gaze estimation methods struggle to generalize across diverse data domains, primarily due to i) the scarcity of annotated datasets, and ii) the insufficient diversity of labeled data. In this work, we present OmniGaze, a semi-supervised framework for 3D gaze estimation, which utilizes large-scale unlabeled data collected from diverse and unconstrained real-world environments to mitigate domain bias and generalize gaze estimation in the wild. First, we build a diverse collection of unlabeled facial images, varying in facial appearances, background environments, illumination conditions, head poses, and eye occlusions. In order to leverage unlabeled data spanning a broader distribution, OmniGaze adopts a standard pseudo-labeling strategy and devises a reward model to assess the reliability of pseudo labels. Beyond pseudo labels as 3D direction vectors, the reward model also incorporates visual embeddings extracted by an off-the-shelf visual encoder and semantic cues from gaze perspective generated by prompting a Multimodal Large Language Model to compute confidence scores. Then, these scores are utilized to select high-quality pseudo labels and weight them for loss computation. Extensive experiments demonstrate that OmniGaze achieves state-of-the-art performance on five datasets under both in-domain and cross-domain settings. Furthermore, we also evaluate the efficacy of OmniGaze as a scalable data engine for gaze estimation, which exhibits robust zero-shot generalization on four unseen datasets.
Abstract (translated)
目前的3D凝视估计方法在不同数据域之间难以泛化，主要原因是：i) 标注数据集稀少；ii) 标记数据种类不足。为此，我们提出了一种半监督框架OmniGaze，用于解决这一问题。该框架利用大规模未标记的真实世界面部图像来减轻领域偏差并提高凝视估计的泛化能力。首先，我们构建了一个多样化的未标记面部图像集合，这些图像是在不同面部外观、背景环境、光照条件、头部姿态和眼睛遮挡下采集到的。 为了充分利用广泛的分布范围内的未标注数据，OmniGaze采用了标准的伪标签策略，并设计了一个奖励模型来评估伪标签的可靠性。除了作为3D方向向量的伪标签外，该奖励模型还结合了现成视觉编码器提取出的视觉嵌入以及由多模态大型语言模型生成的凝视视角语义线索以计算置信度分数。然后利用这些分数选择高质量的伪标签，并为损失计算加权。 广泛的实验表明，OmniGaze在五种数据集上（无论是领域内还是跨域）均达到了最先进的性能水平。此外，我们还评估了OmniGaze作为凝视估计可扩展数据引擎的有效性，在四个未见过的数据集中表现出稳健的零样本泛化能力。
URL
https://arxiv.org/abs/2510.13660
