Gesture
-
From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality
2026-02-03 03:34:12Yoonsang Kim, Divyansh Pradhan, Devshree Jadeja, Arie KaufmanAbstract
We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.
Abstract (translated)
我们介绍了一种名为Speech-to-Spatial的指代消歧框架，该框架将口头远程协助指令转换为基于空间定位的增强现实（AR）指导。与以往依赖额外线索（如手势、目光）或手动专家注释的系统不同，Speech-to-Spatial仅根据口语化的参考信息推断目标对象。受我们对言语引用模式初步研究的启发，我们将人们指定目标的方式进行了分类（直接属性、关系性、记忆式和链式），并将其与以物体为中心的关系图关联起来。对于每条语音输入，指代线索被解析并渲染为持久性的现场AR视觉指导，从而减少了远程指导过程中反复微调指引的需求（如“再往右一点”，“停。”）。 我们通过远程协助和意图消歧场景演示了系统的应用案例。我们的评估表明，与传统的仅依赖声音的基准系统相比，Speech-to-Spatial在提高任务效率、减轻认知负荷以及增强用户体验方面表现出明显优势，将抽象的语言指令转化为在实时共享视图中可视且可操作的指导。
URL
https://arxiv.org/abs/2602.03059
PDF
-
Bimanual High-Density EMG Control for In-Home Mobile Manipulation by a User with Quadriplegia
2026-02-02 20:28:18Jehan Yang, Eleanor Hodgson, Cindy Sun, Zackory Erickson, Doug WeberAbstract
Mobile manipulators in the home can enable people with cervical spinal cord injury (cSCI) to perform daily physical household tasks that they could not otherwise do themselves. However, paralysis in these users often limits access to traditional robot control interfaces such as joysticks or keyboards. In this work, we introduce and deploy the first system that enables a user with quadriplegia to control a mobile manipulator in their own home using bimanual high-density electromyography (HDEMG). We develop a pair of custom, fabric-integrated HDEMG forearm sleeves, worn on both arms, that capture residual neuromotor activity from clinically paralyzed degrees of freedom and support real-time gesture-based robot control. Second, by integrating vision, language, and motion planning modules, we introduce a shared autonomy framework that supports robust and user-driven teleoperation, with particular benefits for navigation-intensive tasks in home environments. Finally, to demonstrate the system in the wild, we present a twelve-day in-home user study evaluating real-time use of the wearable EMG interface for daily robot control. Together, these system components enable effective robot control for performing activities of daily living and other household tasks in a real home environment.
Abstract (translated)
家庭中的移动机械臂可以帮助患有颈椎脊髓损伤（cSCI）的人完成他们通常无法独自完成的日常家务任务。然而，这些用户的瘫痪往往限制了他们使用传统的机器人控制界面（如操纵杆或键盘）。在这项工作中，我们引入并部署了一种系统，使四肢瘫痪的用户能够在其家中通过双手高密度肌电图（HDEMG）来操控移动机械臂。我们开发了一对定制的、织入布料中的HDEMG前臂袖套，佩戴在双臂上，可以捕捉到临床认为瘫痪的自由度上的残留神经肌肉活动，并支持基于手势的实时机器人控制。 其次，通过整合视觉、语言和运动规划模块，我们引入了一个共享自主框架，以支持稳健且用户驱动的远程操作，在家庭环境中进行密集导航任务时尤其有益。最后，为了展示该系统在真实环境中的应用效果，我们开展了一项为期十二天的家庭使用者研究，评估了实时使用可穿戴EMG界面进行日常机器人控制的情况。 这些系统的组成部分共同使得有效操控机器人以完成日常生活活动及其他家务工作成为可能，在真实的家庭环境中得以实现。
URL
https://arxiv.org/abs/2602.02773
PDF
-
Time2Vec-Integrated Transformer for Robust Gesture Recognition from Low-Density sEMG
2026-02-02 09:28:27Blagoj Hristov, Hristijan Gjoreski, Vesna Ojleska Latkoska, Gorjan NadzinskiAbstract
Accurate and responsive myoelectric prosthesis control typically relies on complex, dense multi-sensor arrays, which limits consumer accessibility. This paper presents a novel, data-efficient deep learning framework designed to achieve precise and accurate control using minimal sensor hardware. Leveraging an external dataset of 8 subjects, our approach implements a hybrid Transformer optimized for sparse, two-channel surface electromyography (sEMG). Unlike standard architectures that use fixed positional encodings, we integrate Time2Vec learnable temporal embeddings to capture the stochastic temporal warping inherent in biological signals. Furthermore, we employ a normalized additive fusion strategy that aligns the latent distributions of spatial and temporal features, preventing the destructive interference common in standard implementations. A two-stage curriculum learning protocol is utilized to ensure robust feature extraction despite data scarcity. The proposed architecture achieves a state-of-the-art multi-subject F1-score of 95.7% $\pm$ 0.20% for a 10-class movement set, statistically outperforming both a standard Transformer with fixed encodings and a recurrent CNN-LSTM model. Architectural optimization reveals that a balanced allocation of model capacity between spatial and temporal dimensions yields the highest stability. Furthermore, while direct transfer to a new unseen subject led to poor accuracy due to domain shifts, a rapid calibration protocol utilizing only two trials per gesture recovered performance from 21.0% $\pm$ 2.98% to 96.9% $\pm$ 0.52%. By validating that high-fidelity temporal embeddings can compensate for low spatial resolution, this work challenges the necessity of high-density sensing. The proposed framework offers a robust, cost-effective blueprint for next-generation prosthetic interfaces capable of rapid personalization.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01855
PDF
-
SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Prediction?
2026-01-30 21:27:45Yueyi Yang, Haotian Liu, Fang Kang, Mengqi Zhang, Zheng Lian, Hao Tang, Haoyu ChenAbstract
We explore the use of large language models (LLMs) for next-utterance prediction in human dialogue. Despite recent advances in LLMs demonstrating their ability to engage in natural conversations with users, we show that even leading models surprisingly struggle to predict a human speaker's next utterance. Instead, humans can readily anticipate forthcoming utterances based on multimodal cues, such as gestures, gaze, and emotional tone, from the context. To systematically examine whether LLMs can reproduce this ability, we propose SayNext-Bench, a benchmark that evaluates LLMs and Multimodal LLMs (MLLMs) on anticipating context-conditioned responses from multimodal cues spanning a variety of real-world scenarios. To support this benchmark, we build SayNext-PC, a novel large-scale dataset containing dialogues with rich multimodal cues. Building on this, we further develop a dual-route prediction MLLM, SayNext-Chat, that incorporates cognitively inspired design to emulate predictive processing in conversation. Experimental results demonstrate that our model outperforms state-of-the-art MLLMs in terms of lexical overlap, semantic similarity, and emotion consistency. Our results prove the feasibility of next-utterance prediction with LLMs from multimodal cues and emphasize the (i) indispensable role of multimodal cues and (ii) actively predictive processing as the foundation of natural human interaction, which is missing in current MLLMs. We hope that this exploration offers a new research entry toward more human-like, context-sensitive AI interaction for human-centered AI. Our benchmark and model can be accessed at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00327
PDF
-
Leveraging Convolutional Sparse Autoencoders for Robust Movement Classification from Low-Density sEMG
2026-01-30 14:21:46Blagoj Hristov, Zoran Hadzi-Velkov, Katerina Hadzi-Velkova Saneva, Gorjan Nadzinski, Vesna Ojleska LatkoskaAbstract
Reliable control of myoelectric prostheses is often hindered by high inter-subject variability and the clinical impracticality of high-density sensor arrays. This study proposes a deep learning framework for accurate gesture recognition using only two surface electromyography (sEMG) channels. The method employs a Convolutional Sparse Autoencoder (CSAE) to extract temporal feature representations directly from raw signals, eliminating the need for heuristic feature engineering. On a 6-class gesture set, our model achieved a multi-subject F1-score of 94.3% $\pm$ 0.3%. To address subject-specific differences, we present a few-shot transfer learning protocol that improved performance on unseen subjects from a baseline of 35.1% $\pm$ 3.1% to 92.3% $\pm$ 0.9% with minimal calibration data. Furthermore, the system supports functional extensibility through an incremental learning strategy, allowing for expansion to a 10-class set with a 90.0% $\pm$ 0.2% F1-score without full model retraining. By combining high precision with minimal computational and sensor overhead, this framework provides a scalable and efficient approach for the next generation of affordable and adaptive prosthetic systems.
Abstract (translated)
URL
https://arxiv.org/abs/2601.23011
PDF
-
Plant-Inspired Robot Design Metaphors for Ambient HRI
2026-01-29 22:55:04Victor Nikhil Antony, Adithya R N, Sarah Derrick, Zhili Gong, Peter M. Donley, Chien-Ming HuangAbstract
Plants offer a paradoxical model for interaction: they are ambient, low-demand presences that nonetheless shape atmosphere, routines, and relationships through temporal rhythms and subtle expressions. In contrast, most human-robot interaction (HRI) has been grounded in anthropomorphic and zoomorphic paradigms, producing overt, high-demand forms of engagement. Using a Research through Design (RtD) methodology, we explore plants as metaphoric inspiration for HRI; we conducted iterative cycles of ideation, prototyping, and reflection to investigate what design primitives emerge from plant metaphors and morphologies, and how these primitives can be combined into expressive robotic forms. We present a suite of speculative, open-source prototypes that help probe plant-inspired presence, temporality, form, and gestures. We deepened our learnings from design and prototyping through prototype-centered workshops that explored people's perceptions and imaginaries of plant-inspired robots. This work contributes: (1) Set of plant-inspired robotic artifacts; (2) Designerly insights on how people perceive plant-inspired robots; and (3) Design consideration to inform how to use plant metaphors to reshape HRI.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22387
PDF
-
Tendon-based modelling, estimation and control for a simulated high-DoF anthropomorphic hand model
2026-01-28 15:10:02P\'eter Polcz, Katalin Sch\"affer, Mikl\'os KollerAbstract
Tendon-driven anthropomorphic robotic hands often lack direct joint angle sensing, as the integration of joint encoders can compromise mechanical compactness and dexterity. This paper presents a computational method for estimating joint positions from measured tendon displacements and tensions. An efficient kinematic modeling framework for anthropomorphic hands is first introduced based on the Denavit-Hartenberg convention. Using a simplified tendon model, a system of nonlinear equations relating tendon states to joint positions is derived and solved via a nonlinear optimization approach. The estimated joint angles are then employed for closed-loop control through a Jacobian-based proportional-integral (PI) controller augmented with a feedforward term, enabling gesture tracking without direct joint sensing. The effectiveness and limitations of the proposed estimation and control framework are demonstrated in the MuJoCo simulation environment using the Anatomically Correct Biomechatronic Hand, featuring five degrees of freedom for each long finger and six degrees of freedom for the thumb.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20682
PDF
-
Graph Modelling Analysis of Speech-Gesture Interaction for Aphasia Severity Estimation
2026-01-27 14:11:36Navya Martin Kollapally, Christa Akers, Renjith Nelson JosephAbstract
Aphasia is an acquired language disorder caused by injury to the regions of the brain that are responsible for language. Aphasia may impair the use and comprehension of written and spoken language. The Western Aphasia Battery-Revised (WAB-R) is an assessment tool administered by speech-language pathologists (SLPs) to evaluate the aphasia type and severity. Because the WAB-R measures isolated linguistic skills, there has been growing interest in the assessment of discourse production as a more holistic representation of everyday language abilities. Recent advancements in speech analysis focus on automated estimation of aphasia severity from spontaneous speech, relying mostly in isolated linguistic or acoustical features. In this work, we propose a graph neural network-based framework for estimating aphasia severity. We represented each participant's discourse as a directed multi-modal graph, where nodes represent lexical items and gestures and edges encode word-word, gesture-word, and word-gesture transitions. GraphSAGE is employed to learn participant-level embeddings, thus integrating information from immediate neighbors and overall graph structure. Our results suggest that aphasia severity is not encoded in isolated lexical distribution, but rather emerges from structured interactions between speech and gesture. The proposed architecture offers a reliable automated aphasia assessment, with possible uses in bedside screening and telehealth-based monitoring.
Abstract (translated)
失语症是一种由于大脑语言区域受损而获得的语言障碍。它可能会影响口语和书面语言的使用与理解。西方失语症评估量表修订版（WAB-R）是由言语-语言治疗师（SLPs）用来评估失语类型及严重程度的一种评估工具。因为WAB-R测量的是孤立的语言技能，所以越来越多的人开始对话语产出的评估感兴趣，以作为日常语言能力更全面的表现形式。最近在语音分析方面的进展主要集中在从自然语言中自动估计失语症的严重程度上，这主要依赖于孤立的语言或声学特征。在这项工作中，我们提出了一种基于图神经网络框架来估算失语症严重程度的方法。我们将每个参与者的对话表示为一个有向多模态图，在这种图中节点代表词汇和手势，边则编码词-词、手势-词以及词-手势的转换关系。使用GraphSAGE来学习参与者级别的嵌入，从而整合了来自直接邻居的信息及整个图形结构的信息。我们的结果显示，失语症严重程度并非体现在孤立的词汇分布上，而是从言语与手势之间的有机构成互动中浮现出来。所提出的架构提供了一种可靠的自动化失语评估方法，可能在床边筛查和基于远程医疗监测方面有着潜在的应用价值。
URL
https://arxiv.org/abs/2602.20163
PDF
-
MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreading
2026-01-27 09:19:18Matteo RossiAbstract
Lipreading, the technology of decoding spoken content from silent videos of lip movements, holds significant application value in fields such as public security. However, due to the subtle nature of articulatory gestures, existing lipreading methods often suffer from limited feature discriminability and poor generalization capabilities. To address these challenges, this paper delves into the purification of visual features from temporal, spatial, and channel dimensions. We propose a novel method named Multi-Attention Lipreading Network(MA-LipNet). The core of MA-LipNet lies in its sequential application of three dedicated attention modules. Firstly, a \textit{Channel Attention (CA)} module is employed to adaptively recalibrate channel-wise features, thereby mitigating interference from less informative channels. Subsequently, two spatio-temporal attention modules with distinct granularities-\textit{Joint Spatial-Temporal Attention (JSTA)} and \textit{Separate Spatial-Temporal Attention (SSTA)}-are leveraged to suppress the influence of irrelevant pixels and video frames. The JSTA module performs a coarse-grained filtering by computing a unified weight map across the spatio-temporal dimensions, while the SSTA module conducts a more fine-grained refinement by separately modeling temporal and spatial attentions. Extensive experiments conducted on the CMLR and GRID datasets demonstrate that MA-LipNet significantly reduces the Character Error Rate (CER) and Word Error Rate (WER), validating its effectiveness and superiority over several state-of-the-art methods. Our work highlights the importance of multi-dimensional feature refinement for robust visual speech recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20881
PDF
-
GhostUI: Unveiling Hidden Interactions in Mobile UI
2026-01-27 06:40:29Minkyu Kweon, Seokhyeon Park, Soohyun Lee, You Been Lee, Jeongmin Rhee, Jinwook SeoAbstract
Modern mobile applications rely on hidden interactions--gestures without visual cues like long presses and swipes--to provide functionality without cluttering interfaces. While experienced users may discover these interactions through prior use or onboarding tutorials, their implicit nature makes them difficult for most users to uncover. Similarly, mobile agents--systems designed to automate tasks on mobile user interfaces, powered by vision language models (VLMs)--struggle to detect veiled interactions or determine actions for completing tasks. To address this challenge, we present GhostUI, a new dataset designed to enable the detection of hidden interactions in mobile applications. GhostUI provides before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions, allowing VLMs to better recognize concealed gestures and anticipate post-interaction states. Quantitative evaluations with VLMs show that models fine-tuned on GhostUI outperform baseline VLMs, particularly in predicting hidden interactions and inferring post-interaction screens, underscoring GhostUI's potential as a foundation for advancing mobile task automation.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19258
PDF
-
3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control
2026-01-26 12:57:36Xuanmeng Sha, Liyun Zhang, Tomohiro Mashita, Naoya Chiba, Yuki UranishiAbstract
Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18451
PDF
-
SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis
2026-01-26 09:35:10Xuan Wang, Siyuan Su, Quantong Fu, Yongxiang Hu, Yangfan ZhouAbstract
With the widespread adoption of Graphical User Interface (GUI) agents for automating GUI interaction tasks, substantial research focused on improving GUI perception to ground task instructions into concrete action steps. However, the step execution capability of these agents has gradually emerged as a new bottleneck for task completion. In particular, existing GUI agents often adopt overly simplified strategies for handling swipe interactions, preventing them from accurately replicating human-like behavior. To address this limitation, we decompose human swipe gestures into multiple quantifiable dimensions and propose an automated pipeline SwipeGen to synthesize human-like swipe interactions through GUI exploration. Based on this pipeline, we construct and release the first benchmark for evaluating the swipe execution capability of GUI agents. Furthermore, leveraging the synthesized data, we propose GUISwiper, a GUI agent with enhanced interaction execution capabilities. Experimental results demonstrate that GUISwiper achieves a swipe execution accuracy of 69.07%, representing a 214% improvement over existing VLM baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18305
PDF
-
NeuroManip: Prosthetic Hand Manipulation System Based on EMG and Eye Tracking Powered by the Neuromorphic Processor AltAi
2026-01-25 20:44:03Roman Akinshin, Elizaveta Lopatina, Kirill Bogatikov, Nikolai Kiz, Anna V. Makarova, Mikhail Lebedev, Miguel Altamirano Cabrera, Dzmitry Tsetserukou, Valerii KanglerAbstract
This paper presents a novel neuromorphic control architecture for upper-limb prostheses that combines surface electromyography (sEMG) with gaze-guided computer vision. The system uses a spiking neural network deployed on the neuromorphic processor AltAi to classify EMG patterns in real time while an eye-tracking headset and scene camera identify the object within the user's focus. In our prototype, the same EMG recognition model that was originally developed for a conventional GPU is deployed as a spiking network on AltAi, achieving comparable accuracy while operating in a sub-watt power regime, which enables a lightweight, wearable implementation. For six distinct functional gestures recorded from upper-limb amputees, the system achieves robust recognition performance comparable to state-of-the-art myoelectric interfaces. When the vision pipeline restricts the decision space to three context-appropriate gestures for the currently viewed object, recognition accuracy increases to roughly 95% while excluding unsafe, object-inappropriate grasps. These results indicate that the proposed neuromorphic, context-aware controller can provide energy-efficient and reliable prosthesis control and has the potential to improve safety and usability in everyday activities for people with upper-limb amputation.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17991
PDF
-
Real-Time Synchronized Interaction Framework for Emotion-Aware Humanoid Robots
2026-01-24 03:53:09Yanrong Chen, Xihan BianAbstract
As humanoid robots increasingly introduced into social scene, achieving emotionally synchronized multimodal interaction remains a significant challenges. To facilitate the further adoption and integration of humanoid robots into service roles, we present a real-time framework for NAO robots that synchronizes speech prosody with full-body gestures through three key innovations: (1) A dual-channel emotion engine where large language model (LLM) simultaneously generates context-aware text responses and biomechanically feasible motion descriptors, constrained by a structured joint movement library; (2) Duration-aware dynamic time warping for precise temporal alignment of speech output and kinematic motion keyframes; (3) Closed-loop feasibility verification ensuring gestures adhere to NAO's physical joint limits through real-time adaptation. Evaluations show 21% higher emotional alignment compared to rule-based systems, achieved by coordinating vocal pitch (arousal-driven) with upper-limb kinematics while maintaining lower-body stability. By enabling seamless sensorimotor coordination, this framework advances the deployment of context-aware social robots in dynamic applications such as personalized healthcare, interactive education, and responsive customer service platforms.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17287
PDF
-
Glove2UAV: A Wearable IMU-Based Glove for Intuitive Control of UAV
2026-01-22 09:03:57Amir Habel, Ivan Snegirev, Elizaveta Semenyakina, Miguel Altamirano Cabrera, Jeffrin Sam, Fawad Mehboob, Roohan Ahmed Khan, Muhammad Ahsan Mustafa, Dzmitry TsetserukouAbstract
This paper presents Glove2UAV, a wearable IMU-glove interface for intuitive UAV control through hand and finger gestures, augmented with vibrotactile warnings for exceeding predefined speed thresholds. To promote safer and more predictable interaction in dynamic flight, Glove2UAV is designed as a lightweight and easily deployable wearable interface intended for real-time operation. Glove2UAV streams inertial measurements in real time and estimates palm and finger orientations using a compact processing pipeline that combines median-based outlier suppression with Madgwick-based orientation estimation. The resulting motion estimations are mapped to a small set of control primitives for directional flight (forward/backward and lateral motion) and, when supported by the platform, to object-interaction commands. Vibrotactile feedback is triggered when flight speed exceeds predefined threshold values, providing an additional alert channel during operation. We validate real-time feasibility by synchronizing glove signals with UAV telemetry in both simulation and real-world flights. The results show fast gesture-based command execution, stable coupling between gesture dynamics and platform motion, correct operation of the core command set in our trials, and timely delivery of vibratile warning cues.
Abstract (translated)
本文介绍了Glove2UAV，这是一种可穿戴的IMU（惯性测量单元）手套界面，通过手部和手指手势进行直观的无人机控制，并辅以振动反馈提醒超过预定义速度阈值的情况。为了促进动态飞行中更安全、更具预测性的交互，Glove2UAV被设计成轻便且易于部署的手持式接口，旨在实现实时操作。 Glove2UAV能够实现实时惯性测量数据流传输，并利用一个结合了基于中位数的异常值抑制和Madgwick姿态估计的方法的小型处理管道来估算手掌和手指的姿态。由此产生的运动估计被映射到一组有限的控制基本动作，用于方向飞行（前后和横向移动），并在平台支持的情况下映射为对象交互命令。 当飞行速度超过预定义阈值时，振动反馈会被触发，这在操作过程中提供了一种额外的警报渠道。通过在模拟环境和真实世界飞行中同步手套信号与无人机遥测数据，我们验证了其实时可行性。实验结果表明：基于手势的命令执行速度快；手势动态变化与平台运动之间耦合稳定；我们的试验中核心命令集运行正确；振动警告提示及时送达。
URL
https://arxiv.org/abs/2601.15775
PDF
-
Arabic Sign Language Recognition using Multimodal Approach
2026-01-20 09:21:43Ghadeer Alanazi, Abir BenabidAbstract
Arabic Sign Language (ArSL) is an essential communication method for individuals in the Deaf and Hard-of-Hearing community. However, existing recognition systems face significant challenges due to their reliance on single sensor approaches like Leap Motion or RGB cameras. These systems struggle with limitations such as inadequate tracking of complex hand orientations and imprecise recognition of 3D hand movements. This research paper aims to investigate the potential of a multimodal approach that combines Leap Motion and RGB camera data to explore the feasibility of recognition of ArSL. The system architecture includes two parallel subnetworks: a custom dense neural network for Leap Motion data, incorporating dropout and L2 regularization, and an image subnetwork based on a fine-tuned VGG16 model enhanced with data augmentation techniques. Feature representations from both modalities are concatenated in a fusion model and passed through fully connected layers, with final classification performed via SoftMax activation to analyze spatial and temporal features of hand gestures. The system was evaluated on a custom dataset comprising 18 ArSL words, of which 13 were correctly recognized, yielding an overall accuracy of 78%. These results offer preliminary insights into the viability of multimodal fusion for sign language recognition and highlight areas for further optimization and dataset expansion.
Abstract (translated)
URL
https://arxiv.org/abs/2601.17041
PDF
-
Dynamic Hand Gesture Recognition for Robot Manipulator Tasks
2026-01-19 10:17:20Dharmendra Sharma, Peeyush Thakur, Sandeep Gupta, Narendra Kumar Dhar, Laxmidhar BeheraAbstract
This paper proposes a novel approach to recognizing dynamic hand gestures facilitating seamless interaction between humans and robots. Here, each robot manipulator task is assigned a specific gesture. There may be several such tasks, hence, several gestures. These gestures may be prone to several dynamic variations. All such variations for different gestures shown to the robot are accurately recognized in real-time using the proposed unsupervised model based on the Gaussian Mixture model. The accuracy during training and real-time testing prove the efficacy of this methodology.
Abstract (translated)
本文提出了一种新颖的手势识别方法，旨在促进人类与机器人之间无缝交互。该方法为每个机器人操作任务分配一个特定手势。可能有许多这样的任务，因此也会有多个手势。这些手势可能会出现多种动态变化。使用基于高斯混合模型的无监督模型，可以准确地实时识别展示给机器人的不同手势的所有此类变化。在训练和实时测试期间的准确性证明了该方法的有效性。
URL
https://arxiv.org/abs/2601.12918
PDF
-
Rapport du Projet de Recherche TRAIMA
2026-01-19 08:55:50Julie Ran\c{c}on (UP, FoReLLIS, Poitiers), Jean-Fran\c{c}ois Cerisier (Techn\'e, Poitiers), Emilie Remond (Techn\'e, Poitiers), Aur\'elien Nguyen (Techn\'e, Poitiers), Andrew Peterson (Techn\'e, Poitiers), Ladjel Bellatreche (ISAE-ENSMA, IDD, A\&S)Abstract
The TRAIMA project (TRaitement Automatique des Interactions Multimodales en Apprentissage), conducted between March 2019 and June 2020, investigates the potential of automatic processing of multimodal interactions in educational settings. The project addresses a central methodological challenge in educational and interactional research: the analysis of verbal, paraverbal, and non-verbal data is currently carried out manually, making it extremely time-consuming and difficult to scale. TRAIMA explores how machine learning approaches could contribute to the categorisation and classification of such interactions. The project focuses specifically on explanatory and collaborative sequences occurring in classroom interactions, particularly in French as a Foreign Language (FLE) and French as a First Language (FLM) contexts. These sequences are analysed as inherently multimodal phenomena, combining spoken language with prosody, gestures, posture, gaze, and spatial positioning. A key theoretical contribution of the project is the precise linguistic and interactional definition of explanatory discourse as a tripartite sequence (opening, explanatory core, closure), drawing on discourse analysis and interactional linguistics. A substantial part of the research is devoted to the methodological foundations of transcription, which constitute a critical bottleneck for any form of automation. The report provides a detailed state of the art of existing transcription conventions (ICOR, Mondada, GARS, VALIBEL, Ferr{é}), highlighting their respective strengths and limitations when applied to multimodal classroom data. Through comparative analyses of manually transcribed sequences, the project demonstrates the inevitable variability and interpretative dimension of transcription practices, depending on theoretical positioning and analytical goals. Empirical work is based on several corpora, notably the INTER-EXPLIC corpus (approximately 30 hours of classroom interaction) and the EXPLIC-LEXIC corpus, which serve both as testing grounds for manual annotation and as reference datasets for future automation. Particular attention is paid to teacher gestures (kin{é}sic and proxemic resources), prosodic features, and their functional role in meaning construction and learner comprehension. The project also highlights the strategic role of the Techn{é}LAB platform, which provides advanced multimodal data capture (multi-camera video, synchronized audio, eye-tracking, digital interaction traces) and constitutes both a research infrastructure and a test environment for the development of automated tools. In conclusion, TRAIMA does not aim to deliver a fully operational automated system, but rather to establish a rigorous methodological framework for the automatic processing of multimodal pedagogical interactions. The project identifies transcription conventions, annotation categories, and analytical units that are compatible with machine learning approaches, while emphasizing the need for theoretical explicitness and researcher reflexivity. TRAIMA thus lays the groundwork for future interdisciplinary research at the intersection of didactics, discourse analysis, multimodality, and artificial intelligence in education.
Abstract (translated)
TRAIMA项目（2019年3月至2020年6月期间进行）旨在研究教育环境中自动处理多模态互动的潜力。该项目解决了一个在教育和互动研究中的核心方法论挑战：目前，对于言语、副语言以及非言语数据的分析都是手动完成的，这使得工作既耗时又难以扩展。TRAIMA探索了机器学习方法如何有助于此类交互的数据分类和归类。 项目特别关注课堂中解释性和协作性的序列，特别是在作为外语（FLE）和母语（FLM）背景下的法国语言教学环境中。这些序列被分析为多模态现象，结合口头言语、韵律、手势、姿态、目光接触以及空间定位等元素进行研究。项目的一个重要理论贡献在于精确地用三部分结构定义解释性话语：引入部分、核心阐释和结束部分，并借鉴了语篇分析与互动语言学的研究成果。 项目的重要一部分致力于转录方法论的基础，这是任何自动化形式的关键瓶颈之一。报告详述了现有转录惯例（如ICOR、Mondada、GARS、VALIBEL和Ferré）的现状，并突出显示这些惯例在应用于多模态教室数据时各自的优缺点与限制。 通过对比分析手动转录序列，项目展示了转录实践不可避免的变异性和解释维度，这取决于理论定位和分析目标。实证研究基于几个语料库，包括INTER-EXPLIC（大约30小时的课堂互动）和EXPLIC-LEXIC语料库，这些不仅用于测试人工注释方法，还作为未来自动化的参考数据集。 项目特别关注教师的手势（身体动作与空间距离资源）、韵律特征及其在意义构建和学习者理解中的功能角色。此外，项目强调了TechnéLAB平台的战略作用，该平台提供先进的多模态数据采集工具（包括多摄像头视频、同步音频、眼动追踪以及数字互动记录），并同时作为研究基础设施和自动化工具开发的测试环境。 最终，TRAIMA项目的目的是建立一个严格的自动处理多模态教育交互的方法论框架，而不是试图提供一套完全可操作的自动化系统。项目确定了与机器学习方法兼容的转录惯例、注释类别以及分析单位，并强调理论清晰度和研究者反思的重要性。因此，TRAIMA为跨学科研究奠定了基础，在这一领域中，教学法、语篇分析、多模态性及教育领域的人工智能相互交叉。
URL
https://arxiv.org/abs/2601.12844
PDF
-
VR$^2$: A Co-Located Dual-Headset Platform for Touch-Enabled Human-Robot Interaction Research
2026-01-18 13:11:41Chao Wang, Anna Belardinelli, Michael GiengerAbstract
Touch-rich human-robot interaction (HRI) is difficult to study: building and programming physical robots is costly and slow, while VR-based robot prototypes often remove physical contact or break the tight coupling between an agent's body and the user's felt touch. We present VR2VR, a co-located dual VR-headset platform for HRI research in which a participant and a hidden operator share the same physical space while experiencing different virtual embodiments. The participant sees an expressive virtual robot that interacts face-to-face in a shared virtual environment. In real time, the robot's upper-body gestures, head and gaze behaviors, and facial expressions are mapped from the operator's tracked motion and face signals. Because the operator is physically co-present and calibrated into the same coordinate frame, the operator can also physically touch the participant, enabling the participant to perceive robot touch aligned with the robot's hands; finger and hand motion are mapped to the robot using inverse kinematics to support precise contact. Beyond faithful motion retargeting for limb teleoperation, our VR2VR system supports experimental control by retargeting or selectively enabling nonverbal channels (e.g., head only vs. head+eyes vs. head+eyes+facial expressions) while keeping physical interaction constant. We detail the system design, calibration workflow, and safety considerations, and demonstrate the platform through a touch-based Wizard-of-Oz HRI study, illustrating how VR2VR lowers barriers for rapidly prototyping and rigorously evaluating embodied, touch-centric robot behaviors.
Abstract (translated)
触觉丰富的机器人交互（HRI）研究颇具挑战性：制造和编程实体机器人成本高昂且耗时，而基于虚拟现实的机器人原型往往会移除物理接触或破坏代理身体与用户感知到的触摸之间的紧密耦合。我们提出了一种名为VR2VR的双VR头显平台，用于共存空间中的HRI研究，在这种平台上，参与者和隐藏的操作员共享同一物理空间但拥有不同的虚拟化身。参与者可以看到一个具有表现力的虚拟机器人在共享的虚拟环境中进行面对面交互。通过实时映射操作员被追踪的动作及面部信号，机器人的上半身手势、头部与视线行为以及面部表情得以再现。由于操作者在同一坐标系中身体共存且已校准，因此他们可以物理地接触参与者，从而使参与者能够感知到与机器人手部动作相一致的触觉；手指和手臂运动通过逆向动力学映射至机器人的手部以支持精确接触。 除了为肢体远程操作提供忠实的动作重定向外，我们的VR2VR系统还通过重新定位或选择性启用非语言通道（例如仅头部、头部+眼睛或头部+眼睛+面部表情）来进行实验控制，同时保持物理交互不变。我们详细介绍了系统的架构设计、校准工作流程以及安全考虑，并通过基于触觉的“魔术师之助”HRI研究展示了该平台的能力，说明了VR2VR如何降低快速原型制作和严格评估以触摸为中心的机器人行为的门槛。
URL
https://arxiv.org/abs/2601.12395
PDF
-
Model selection and real-time skill assessment for suturing in robotic surgery
2026-01-17 11:28:59Zhaoyang Jacopo Hu, Alex Ranne, Alaa Eldin Abdelaal, Kiran Bhattacharyya, Etienne Burdet, Allison M. Okamura, Ferdinando Rodriguez y BaenaAbstract
Automated feedback systems have the potential to provide objective skill assessment for training and evaluation in robot-assisted surgery. In this study, we examine methods to achieve real-time prediction of surgical skill level in real-time based on Objective Structured Assessment of Technical Skills (OSATS) scores. Using data acquired from the da Vinci Surgical System, we carry out three main analyses, focusing on model design, their real-time performance, and their skill-level-based cross-validation training. For the model design, we evaluate the effectiveness of multimodal deep learning models for predicting surgical skill levels using synchronized kinematic and vision data. Our models include separate unimodal baselines and fusion architectures that integrate features from both modalities and are evaluated using mean Spearman's correlation coefficients, demonstrating that the fusion model consistently outperforms unimodal models for real-time predictions. For the real-time performance, we observe the prediction's trend over time and highlight correlation with the surgeon's gestures. For the skill-level-based cross-validation, we separately trained models on surgeons with different skill levels, which showed that high-skill demonstrations allow for better performance than those trained on low-skilled ones and generalize well to similarly skilled participants. Our findings show that multimodal learning allows more stable fine-grained evaluation of surgical performance and highlights the value of expert-level training data for model generalization.
Abstract (translated)
自动化反馈系统有潜力为机器人辅助手术中的培训和评估提供客观的技能评定。在这项研究中，我们探讨了基于客观结构化技术评分（OSATS）得分实现实时预测外科医生技能水平的方法。利用达芬奇外科系统的数据，我们进行了三项主要分析：模型设计、其实时性能以及基于技能水平的交叉验证训练。 在模型设计方面，我们评估了多模态深度学习模型的有效性，这些模型使用同步运动学和视觉数据来预测手术技能水平。我们的模型包括单一模式基准线及融合架构，后者整合了两种模式的特征，并通过平均Spearman相关系数进行评价。结果显示，融合模型在实时预测中始终优于单模式模型。 对于实时性能分析，我们观察了随时间变化的预测趋势，并强调与外科医生手势的相关性。在基于技能水平的交叉验证中，我们在不同技能水平的外科医生上分别训练了模型，结果表明，由高技能演示培训的模型比那些基于低技能数据训练的模型表现更好且能很好地泛化至类似技能水平的参与者。 我们的研究发现表明，多模态学习能够进行更为稳定细致的手术性能评估，并强调专家级训练数据对模型泛化的价值。
URL
https://arxiv.org/abs/2601.12012
PDF
