Abstract
Existing skeleton-based human action classification models rely on well-trimmed action-specific skeleton videos for both training and testing, precluding their scalability to real-world applications where untrimmed videos exhibiting concatenated actions are predominant. To overcome this limitation, recently introduced skeleton action segmentation models involve un-trimmed skeleton videos into end-to-end training. The model is optimized to provide frame-wise predictions for any length of testing videos, simultaneously realizing action localization and classification. Yet, achieving such an improvement im-poses frame-wise annotated skeleton videos, which remains time-consuming in practice. This paper features a novel framework for skeleton-based action segmentation trained on short trimmed skeleton videos, but that can run on longer un-trimmed videos. The approach is implemented in three steps: Stitch, Contrast, and Segment. First, Stitch proposes a tem-poral skeleton stitching scheme that treats trimmed skeleton videos as elementary human motions that compose a semantic space and can be sampled to generate multi-action stitched se-quences. Contrast learns contrastive representations from stitched sequences with a novel discrimination pretext task that enables a skeleton encoder to learn meaningful action-temporal contexts to improve action segmentation. Finally, Segment relates the proposed method to action segmentation by learning a segmentation layer while handling particular da-ta availability. Experiments involve a trimmed source dataset and an untrimmed target dataset in an adaptation formulation for real-world skeleton-based human action segmentation to evaluate the effectiveness of the proposed method.
Abstract (translated)
现有的基于骨架的人类动作分类模型依赖于精心修剪的动作特定骨架视频来进行训练和测试，这阻碍了它们在现实世界应用中的扩展能力，在这些应用场景中，包含拼接动作的未修剪视频占主导地位。为了克服这一局限性，最近引入的骨架动作分割模型将未修剪的骨架视频纳入端到端的训练过程。该模型被优化为能够对任何长度的测试视频提供逐帧预测，同时实现动作定位和分类。然而，要达到这种改进需要逐帧标注的骨架视频，这在实践中仍然耗时费力。本文介绍了一种新颖的基于骨架的动作分割框架，它可以在短修剪过的骨架视频上进行训练，但在较长未修剪的视频上运行。该方法分为三个步骤：拼接、对比和分割。首先，“拼接”提出了一种时间骨架拼接方案，将修剪后的骨架视频视为构成语义空间的基本人体运动，并可以采样生成包含多个动作的拼接序列。“对比”从拼接序列中学习对比表示，使用一种新颖的判别预训练任务，使骨架编码器能够学习有意义的动作-时间上下文以改进动作分割。最后，“分割”通过学习分割层来将提出的方法应用于动作分割，并处理特定的数据可用性问题。实验采用修剪过的源数据集和未修剪的目标数据集，在适应现实世界基于骨架的人类动作分割的设定下，评估了所提方法的有效性。
URL
https://arxiv.org/abs/2412.14988
