Abstract
We introduce GazeD, a new 3D gaze estimation method that jointly provides 3D gaze and human pose from a single RGB image. Leveraging the ability of diffusion models to deal with uncertainty, it generates multiple plausible 3D gaze and pose hypotheses based on the 2D context information extracted from the input image. Specifically, we condition the denoising process on the 2D pose, the surroundings of the subject, and the context of the scene. With GazeD we also introduce a novel way of representing the 3D gaze by positioning it as an additional body joint at a fixed distance from the eyes. The rationale is that the gaze is usually closely related to the pose, and thus it can benefit from being jointly denoised during the diffusion process. Evaluations across three benchmark datasets demonstrate that GazeD achieves state-of-the-art performance in 3D gaze estimation, even surpassing methods that rely on temporal information. Project details will be available at this https URL.
Abstract (translated)
我们介绍了GazeD，这是一种新的3D凝视估计方法，可以从单个RGB图像中同时提供3D凝视方向和人体姿态。通过利用扩散模型处理不确定性的能力，它基于从输入图像提取的2D上下文信息生成多个合理的3D凝视方向和姿势假设。具体来说，我们在去噪过程中以2D姿态、主体周围的环境以及场景背景作为条件进行操作。使用GazeD，我们还引入了一种新的表示3D凝视的方法，即将其定位为距眼睛固定距离的一个额外身体关节。理由是凝视通常与姿态密切相关，因此在扩散处理中共同去噪可以获益。跨三个基准数据集的评估表明，GazeD在3D凝视估计方面达到了最先进的性能，甚至超越了依赖于时间信息的方法。项目详情将在[此处](https://this https URL)提供。
URL
https://arxiv.org/abs/2601.12948
