Abstract
Learning from Demonstration (LfD) is a useful paradigm for training policies that solve tasks involving complex motions. In practice, the successful application of LfD requires overcoming error accumulation during policy execution, i.e. the problem of drift due to errors compounding over time and the consequent out-of-distribution behaviours. Existing works seek to address this problem through scaling data collection, correcting policy errors with a human-in-the-loop, temporally ensembling policy predictions or through learning the parameters of a dynamical system model. In this work, we propose and validate an alternative approach to overcoming this issue. Inspired by reservoir computing, we develop a novel neural network layer that includes a fixed nonlinear dynamical system with tunable dynamical properties. We validate the efficacy of our neural network layer on the task of reproducing human handwriting motions using the LASA Human Handwriting Dataset. Through empirical experiments we demonstrate that incorporating our layer into existing neural network architectures addresses the issue of compounding errors in LfD. Furthermore, we perform a comparative evaluation against existing approaches including a temporal ensemble of policy predictions and an Echo State Networks (ESNs) implementation. We find that our approach yields greater policy precision and robustness on the handwriting task while also generalising to multiple dynamics regimes and maintaining competitive latency scores.
Abstract (translated)
学习演示（LfD）是一种有效的解决涉及复杂动作任务的策略训练范式。在实践中，成功应用LfD需要通过在策略执行过程中克服误差累积问题，即误差随时间累积导致的漂移问题，以及随之而来的离散行为。现有作品试图通过缩放数据收集、通过人机交互来纠正策略错误、通过学习动态系统模型的参数来解决这个问题。在这篇工作中，我们提出了并验证了一种克服这个问题的替代方法。受到水库计算的启发，我们开发了一个新神经网络层，包括一个固定非线性动力学系统，具有可调的动态特性。我们通过使用LASA人类手写字符数据集来验证我们神经网络层的有效性。通过实验我们证明了将我们的层应用于现有的神经网络架构可以解决LfD中累积误差的問題。此外，我们还对包括基于策略预测的时间聚类和Echo State Networks（ESNs）实现的现有方法进行了比较评估。我们发现，与其他方法相比，我们的方法在手写字符任务中具有更高的策略精度、鲁棒性和扩展性，同时保持竞争延迟分数。
URL
https://arxiv.org/abs/2409.18768
