Abstract
Despite LiDAR (Light Detection and Ranging) being an effective privacy-preserving alternative to RGB cameras to perceive human activities, it remains largely underexplored in the context of multi-modal contrastive pre-training for human activity understanding (e.g., human activity recognition (HAR), retrieval, or person re-identification (RE-ID)). To close this gap, our work explores learning the correspondence between LiDAR point clouds, human skeleton poses, IMU data, and text in a joint embedding space. More specifically, we present DeSPITE, a Deep Skeleton-Pointcloud-IMU-Text Embedding model, which effectively learns a joint embedding space across these four modalities through noise contrastive estimation. At the heart of our empirical exploration, we have combined the existing LIPD and Babel datasets, which enabled us to synchronize data of all four modalities, allowing us to explore the learning of a new joint embedding space. Our experiments demonstrate novel human activity understanding tasks for point cloud sequences enabled through DeSPITE, including Skeleton<->Pointcloud<->IMU matching, retrieval, and temporal moment retrieval. Furthermore, we show that DeSPITE is an effective pre-training strategy for point cloud HAR through experiments in MSR-Action3D and HMPEAR.
Abstract (translated)
尽管激光雷达（LiDAR，Light Detection And Ranging）作为一种有效的隐私保护替代方案，在感知人类活动方面比RGB相机更加有效，但在多模态对比预训练以理解人类活动中（例如，人体动作识别(Human Activity Recognition, HAR)、检索或行人再识别(Person Re-Identification, RE-ID)等任务）的应用仍然较少。为了弥补这一不足，我们的工作探索了在联合嵌入空间中学习激光雷达点云、人体骨架姿势、惯性测量单元(IMU)数据和文本之间对应关系的方法。 具体来说，我们提出了DeSPITE（Deep Skeleton-Pointcloud-IMU-Text Embedding）模型，这是一种深度骨架-点云-IMU-文本嵌入模型。该模型能够通过噪声对比估计方法有效地学习这四种模态之间的联合嵌入空间。在我们的经验研究中，我们将现有的LIPD和Babel数据集结合起来使用，从而实现了所有四类数据的同步，使我们能够在新的联合嵌入空间中进行探索。 实验表明，DeSPITE为点云序列中的新的人体活动理解任务提供了可能，包括骨架<->点云<->IMU匹配、检索以及时间段检索。此外，通过在MSR-Action3D和HMPEAR数据集上的实验，我们证明了DeSPITE是一种有效的预训练策略，可以应用于基于点云的动作识别任务中。
URL
https://arxiv.org/abs/2506.13897
