Abstract
Analyzing hand-object interaction in egocentric vision facilitates VR/AR applications and human-robot policy transfer. Existing research has mostly focused on modeling the behavior paradigm of interactive actions (i.e., ``how to interact''). However, the more challenging and fine-grained problem of capturing the critical moments of contact and separation between the hand and the target object (i.e., ``when to interact'') is still underexplored, which is crucial for immersive interactive experiences in mixed reality and robotic motion planning. Therefore, we formulate this problem as temporal interaction localization (TIL). Some recent works extract semantic masks as TIL references, but suffer from inaccurate object grounding and cluttered scenarios. Although current temporal action localization (TAL) methods perform well in detecting verb-noun action segments, they rely on category annotations during training and exhibit limited precision in localizing hand-object contact/separation moments. To address these issues, we propose a novel zero-shot approach dubbed EgoLoc to localize hand-object contact and separation timestamps in egocentric videos. EgoLoc introduces hand-dynamics-guided sampling to generate high-quality visual prompts. It exploits the vision-language model to identify contact/separation attributes, localize specific timestamps, and provide closed-loop feedback for further refinement. EgoLoc eliminates the need for object masks and verb-noun taxonomies, leading to generalizable zero-shot implementation. Comprehensive experiments on the public dataset and our novel benchmarks demonstrate that EgoLoc achieves plausible TIL for egocentric videos. It is also validated to effectively facilitate multiple downstream applications in egocentric vision and robotic manipulation tasks. Code and relevant data will be released at this https URL.
Abstract (translated)
在第一人称视觉中分析手与物体的交互，有助于虚拟现实（VR）和增强现实（AR）应用以及人机政策转移。现有研究主要集中在互动行为模式的建模上（即“如何进行互动”）。然而，更复杂且精细的问题是捕捉手与目标物之间接触和分离的关键时刻（即“何时进行互动”），这个问题至今仍未得到充分探索，但对于混合现实中的沉浸式交互体验以及机器人运动规划至关重要。因此，我们将该问题定义为时间互动定位（TIL）。 一些最近的研究提取语义掩码作为 TIL 的参考，但存在物体对齐不准确和复杂场景的问题。尽管当前的时间动作定位 (TAL) 方法在检测动词-名词行为片段方面表现出色，但在训练过程中依赖于类别标注，并且在定位手与物体接触/分离时刻的精度有限。 为了解决这些问题，我们提出了一种新颖的零样本方法 EgoLoc，用于定位第一人称视频中手与物体的接触和分离时间戳。EgoLoc 引入了以手部动态为导向的采样技术来生成高质量的视觉提示。它利用视觉-语言模型识别接触/分离属性、精确定位特定时间戳，并提供闭环反馈进行进一步优化。 通过使用 EgoLoc，可以消除对物体掩码和动词-名词分类法的需求，从而实现通用化的零样本实施。在公共数据集以及我们自己创建的新基准上的综合实验表明，EgoLoc 在第一人称视频中实现了合理的 TIL 结果，并且已被证明能够有效促进多个下游应用，包括第一人称视觉任务及机器人操作任务。 代码及相关数据将在此网址发布：[此链接]（请将 [此链接] 替换为实际的网址）。
URL
https://arxiv.org/abs/2508.12349
