Abstract
Locating human-object interaction (HOI) actions within video serves as the foundation for multiple downstream tasks, such as human behavior analysis and human-robot skill transfer. Current temporal action localization methods typically rely on annotated action and object categories of interactions for optimization, which leads to domain bias and low deployment efficiency. Although some recent works have achieved zero-shot temporal action localization (ZS-TAL) with large vision-language models (VLMs), their coarse-grained estimations and open-loop pipelines hinder further performance improvements for temporal interaction localization (TIL). To address these issues, we propose a novel zero-shot TIL approach dubbed EgoLoc to locate the timings of grasp actions for human-object interaction in egocentric videos. EgoLoc introduces a self-adaptive sampling strategy to generate reasonable visual prompts for VLM reasoning. By absorbing both 2D and 3D observations, it directly samples high-quality initial guesses around the possible contact/separation timestamps of HOI according to 3D hand velocities, leading to high inference accuracy and efficiency. In addition, EgoLoc generates closed-loop feedback from visual and dynamic cues to further refine the localization results. Comprehensive experiments on the publicly available dataset and our newly proposed benchmark demonstrate that EgoLoc achieves better temporal interaction localization for egocentric videos compared to state-of-the-art baselines. We will release our code and relevant data as open-source at this https URL.
Abstract (translated)
在视频中定位人与物体的交互（HOI）动作是多个下游任务的基础，如人类行为分析和人机技能转移。当前的时间行动定位方法通常依赖于标注的动作和对象类别来进行优化，这会导致领域偏差并降低部署效率。尽管最近有一些研究通过大型视觉语言模型（VLMs）实现了零样本时间行动定位（ZS-TAL），但它们粗略的估计结果和开环管道阻碍了人与物体交互时间定位（TIL）性能进一步提升。 为了应对这些问题，我们提出了一种新颖的零样本TIL方法，称为EgoLoc，用于在第一视角视频中定位人与物体互动中的抓握动作的时间。EgoLoc引入了一个自适应采样策略来为VLM推理生成合理的视觉提示。通过吸收二维和三维观察数据，它根据三维手的速度直接从可能的接触/分离时间戳周围采样高质量的初始猜测，从而提高推理准确性和效率。 此外，EgoLoc还通过视觉和动态线索产生闭环反馈，进一步细化定位结果。在公开可用的数据集以及我们新提出的基准测试上的全面实验表明，与最先进的基线相比，EgoLoc在第一视角视频的人际互动时间定位方面表现出更好的性能。我们将开源我们的代码及相关数据，请访问此链接以获取更多详情：[该URL需由发布者提供]。
URL
https://arxiv.org/abs/2506.03662
