Abstract
The ability to determine when a person struggles during skill acquisition is crucial for both optimizing human learning and enabling the development of effective assistive systems. As skills develop, the type and frequency of struggles tend to change, and understanding this evolution is key to determining the user's current stage of learning. However, existing manipulation datasets have not focused on how struggle evolves over time. In this work, we collect a dataset for struggle determination, featuring 61.68 hours of video recordings, 2,793 videos, and 5,385 annotated temporal struggle segments collected from 76 participants. The dataset includes 18 tasks grouped into four diverse activities -- tying knots, origami, tangram puzzles, and shuffling cards, representing different task variations. In addition, participants repeated the same task five times to capture their evolution of skill. We define the struggle determination problem as a temporal action localization task, focusing on identifying and precisely localizing struggle segments with start and end times. Experimental results show that Temporal Action Localization models can successfully learn to detect struggle cues, even when evaluated on unseen tasks or activities. The models attain an overall average mAP of 34.56% when generalizing across tasks and 19.24% across activities, indicating that struggle is a transferable concept across various skill-based tasks while still posing challenges for further improvement in struggle detection. Our dataset is available at this https URL.
Abstract (translated)
确定一个人在技能获取过程中遇到困难的能力对于优化人类学习和开发有效的辅助系统至关重要。随着技能的发展，遇到的困难类型和频率往往会改变，理解这种演变是判断用户当前学习阶段的关键所在。然而，现有的操作数据集并没有关注随着时间推移困难是如何演化的。在这项工作中，我们收集了一个用于确定困难的数据集，该数据集包含76名参与者录制的视频资料，时长总计为61.68小时、2,793段视频和5,385个标注的时间段，涵盖了四种不同的活动——打结、折纸、七巧板拼图和洗牌，代表了不同任务的变化形式。此外，每个参与者重复执行相同的任务五次以记录他们的技能演变过程。 我们将确定困难的问题定义为一个时间动作定位任务，重点在于识别并精确定位开始时间和结束时间的困难片段。实验结果显示，时间动作定位模型能够成功地学习到检测困难信号的能力，即使在评估未见过的任务或活动中也是如此。当模型跨任务推广时，整体平均mAP值为34.56%，而跨活动推广时则为19.24%。这表明困难作为一个可转移的概念存在于各种基于技能的任务中，尽管它仍然对进一步改善困难检测构成了挑战。 我们的数据集可以在[这里](https://this.url/dataset)获取。
URL
https://arxiv.org/abs/2510.01362
