Abstract
Detecting and interpreting operator actions, engagement, and object interactions in dynamic industrial workflows remains a significant challenge in human-robot collaboration research, especially within complex, real-world environments. Traditional unimodal methods often fall short of capturing the intricacies of these unstructured industrial settings. To address this gap, we present a novel Multimodal Industrial Activity Monitoring (MIAM) dataset that captures realistic assembly and disassembly tasks, facilitating the evaluation of key meta-tasks such as action localization, object interaction, and engagement prediction. The dataset comprises multi-view RGB, depth, and Inertial Measurement Unit (IMU) data collected from 22 sessions, amounting to 290 minutes of untrimmed video, annotated in detail for task performance and operator behavior. Its distinctiveness lies in the integration of multiple data modalities and its emphasis on real-world, untrimmed industrial workflows-key for advancing research in human-robot collaboration and operator monitoring. Additionally, we propose a multimodal network that fuses RGB frames, IMU data, and skeleton sequences to predict engagement levels during industrial tasks. Our approach improves the accuracy of recognizing engagement states, providing a robust solution for monitoring operator performance in dynamic industrial environments. The dataset and code can be accessed from this https URL.
Abstract (translated)
在动态工业工作流程中检测和解读操作员的动作、参与度以及与物体的互动仍然是人机协作研究中的一个重要挑战，尤其是在复杂的真实世界环境中。传统的一模态方法往往难以捕捉这些无结构工业环境下的细微差别。为了解决这一缺口，我们提出了一个新型的多模态工业活动监测（MIAM）数据集，该数据集记录了现实装配和拆卸任务，便于评估包括动作定位、物体互动以及参与度预测等关键元任务的效果。此数据集中包含来自22个会话收集的多视角RGB、深度图像及惯性测量单元(IMU) 数据，共计未剪辑视频290分钟，并详细标注了任务表现和操作员行为。其独特之处在于整合了多种数据模态，并且着重于真实世界的无裁剪工业工作流程，这对推进人机协作和操作员监测研究至关重要。 此外，我们还提出了一种多模态网络，该网络融合RGB帧、IMU数据及骨架序列以预测工业任务中的人类参与度。我们的方法提高了识别参与状态的准确性，为在动态工业环境中监测操作员表现提供了一个稳健的解决方案。数据集和代码可以从以下链接获取：[此URL]（请将"[此URL]"替换为实际提供的访问地址）。
URL
https://arxiv.org/abs/2501.05936
