Abstract
Language-driven action localization in videos requires not only semantic alignment between language query and video segment, but also prediction of action boundaries. However, the language query primarily describes the main content of an action and usually lacks specific details of action start and end boundaries, which increases the subjectivity of manual boundary annotation and leads to boundary uncertainty in training data. In this paper, on one hand, we propose to expand the original query by generating textual descriptions of the action start and end boundaries through LLMs, which can provide more detailed boundary cues for localization and thus reduce the impact of boundary uncertainty. On the other hand, to enhance the tolerance to boundary uncertainty during training, we propose to model probability scores of action boundaries by calculating the semantic similarities between frames and the expanded query as well as the temporal distances between frames and the annotated boundary frames. They can provide more consistent boundary supervision, thus improving the stability of training. Our method is model-agnostic and can be seamlessly and easily integrated into any existing models of language-driven action localization in an off-the-shelf manner. Experimental results on several datasets demonstrate the effectiveness of our method.
Abstract (translated)
视频中的语言驱动动作定位不仅需要在语言查询和视频片段之间进行语义对齐，还需要预测动作边界。然而，语言查询主要描述了动作的主要内容，通常缺乏关于动作开始和结束的具体细节，这增加了人工边界标注的主观性，并导致训练数据中的边界不确定性。 在这篇论文中，一方面，我们提出通过大型语言模型（LLMs）生成动作起始和结束边界的文本描述来扩展原始查询。这种方法可以为定位提供更多详细的边界线索，从而减少边界不确定性的负面影响。另一方面，为了增强模型在面对边界不确定性时的训练鲁棒性，我们建议通过对帧与扩展后的查询之间的语义相似性和帧与标注边界帧之间的时间距离进行计算，以建立动作边界的概率分数。这些概率分数可以提供更一致的边界监督信息，从而提高训练过程中的稳定性。 我们的方法是模型无关的，并且可以通过现成的方式无缝集成到任何现有的语言驱动的动作定位模型中。在多个数据集上的实验结果验证了我们方法的有效性。
URL
https://arxiv.org/abs/2505.24282
