Gesture
-
Intuitive Human-Robot Interaction: Development and Evaluation of a Gesture-Based User Interface for Object Selection
2026-04-07 16:52:59Bijan Kavousian, Oliver Petrovic, Werner HerfsAbstract
Gestures are a natural form of communication between humans and can also be leveraged for human-robot interaction. This work presents a gesture-based user interface for object selection using pointing and click gestures. An experiment with 20 participants evaluates accuracy and selection time, demonstrating the potential for efficient collaboration.
Abstract (translated)
手势是人类之间自然的交流形式，也可用于人机交互。本研究提出了一种基于手势的用户界面，通过指点与点击手势实现物体选择。一项针对20名参与者的实验评估了其准确性与选择时间，展示了高效协作的潜力。
URL
https://arxiv.org/abs/2604.06073
PDF
-
Automating Manual Tasks through Intuitive Robot Programming and Cognitive Robotics
2026-04-07 15:08:20Bijan Kavousian, Petar Tesic, Oliver Petrovic, Christian BrecherAbstract
This paper presents a novel concept for intuitive end-user programming of robots, inspired by natural interaction between humans. Natural language and supportive gestures are translated into robot programs using large language models (LLMs) and computer vision (CV). Through equally natural system feedback in the form of clarification questions and visual representations, the generated program can be reviewed and adjusted, thereby ensuring safety, transparency, and user acceptance.
Abstract (translated)
本文提出了一种受人类自然交互启发的、面向终端用户的机器人直观编程新概念。通过大语言模型（LLM）与计算机视觉（CV）技术，自然语言与辅助手势可被转换为机器人程序。系统通过自然的澄清性提问与视觉呈现提供反馈，使用户能够审查并调整所生成的程序，从而确保安全性、透明性与用户接受度。
URL
https://arxiv.org/abs/2604.05978
PDF
-
INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognition
2026-04-07 08:56:53Nikolaos D. Tantaroudas, Andrew J. McCracken, Ilias Karachalios, Evangelos PapatheouAbstract
Video conferencing has become central to professional collaboration, yet most platforms offer limited support for deaf, hard-of-hearing, and multilingual users. The World Health Organisation estimates that over 430 million people worldwide require rehabilitation for disabling hearing loss, a figure projected to exceed 700 million by 2050. Conventional accessibility measures remain constrained by high costs, limited availability, and logistical barriers, while Extended Reality (XR) technologies open new possibilities for immersive and inclusive communication. This paper presents INTERACT (Inclusive Networking for Translation and Embodied Real-Time Augmented Communication Tool), an AI-driven XR platform that integrates real-time speech-to-text conversion, International Sign Language (ISL) rendering through 3D avatars, multilingual translation, and emotion recognition within an immersive virtual environment. Built on the CORTEX2 framework and deployed on Meta Quest 3 headsets, INTERACT combines Whisper for speech recognition, NLLB for multilingual translation, RoBERTa for emotion classification, and Google MediaPipe for gesture extraction. Pilot evaluations were conducted in two phases, first with technical experts from academia and industry, and subsequently with members of the deaf community. The trials reported 92% user satisfaction, transcription accuracy above 85%, and 90% emotion-detection precision, with a mean overall experience rating of 4.6 out of 5.0 and 90% of participants willing to take part in further testing. The results highlight strong potential for advancing accessibility across educational, cultural, and professional settings. An extended version of this work, including full pilot data and implementation details, has been published as an Open Research Europe article [Tantaroudas et al., 2026a].
Abstract (translated)
URL
https://arxiv.org/abs/2604.05605
PDF
-
AI-Driven Modular Services for Accessible Multilingual Education in Immersive Extended Reality Settings: Integrating Speech Processing, Translation, and Sign Language Rendering
2026-04-07 08:35:53N. D. Tantaroudas, A. J. McCracken, I. Karachalios, E. PapatheouAbstract
This work introduces a modular platform that brings together six AI services, automatic speech recognition via OpenAI Whisper, multilingual translation through Meta NLLB, speech synthesis using AWS Polly, emotion classification with RoBERTa, dialogue summarisation via flan t5 base samsum, and International Sign (IS) rendering through Google MediaPipe. A corpus of IS gesture recordings was processed to derive hand landmark coordinates, which were subsequently mapped onto three dimensional avatar animations inside a virtual reality (VR) environment. Validation comprised technical benchmarking of each AI component, including comparative assessments of speech synthesis providers and multilingual translation models (NLLB 200 and EuroLLM 1.7B variants). Technical evaluations confirmed the suitability of the platform for real time XR deployment. Speech synthesis benchmarking established that AWS Polly delivers the lowest latency at a competitive price point. The EuroLLM 1.7B Instruct variant attained a higher BLEU score, surpassing NLLB. These findings establish the viability of orchestrating cross modal AI services within XR settings for accessible, multilingual language instruction. The modular design permits independent scaling and adaptation to varied educational contexts, providing a foundation for equitable learning solutions aligned with European Union digital accessibility goals.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05591
PDF
-
A Synthetic Eye Movement Dataset for Script Reading Detection: Real Trajectory Replay on a 3D Simulator
2026-04-07 06:15:48Kidus Zewde, Yuchen Zhou, Dennis Ng, Neo Tiangratanakul, Tommy Duong, Ankit Raj, Yuxin Zhang, Xingyu Shen, Simiao RenAbstract
Large vision-language models have achieved remarkable capabilities by training on massive internet-scale data, yet a fundamental asymmetry persists: while LLMs can leverage self-supervised pretraining on abundant text and image data, the same is not true for many behavioral modalities. Video-based behavioral data -- gestures, eye movements, social signals -- remains scarce, expensive to annotate, and privacy-sensitive. A promising alternative is simulation: replace real data collection with controlled synthetic generation to produce automatically labeled data at scale. We introduce infrastructure for this paradigm applied to eye movement, a behavioral signal with applications across vision-language modeling, virtual reality, robotics, accessibility systems, and cognitive science. We present a pipeline for generating synthetic labeled eye movement video by extracting real human iris trajectories from reference videos and replaying them on a 3D eye movement simulator via headless browser automation. Applying this to the task of script-reading detection during video interviews, we release final_dataset_v1: 144 sessions (72 reading, 72 conversation) totaling 12 hours of synthetic eye movement video at 25fps. Evaluation shows that generated trajectories preserve the temporal dynamics of the source data (KS D < 0.14 across all metrics). A matched frame-by-frame comparison reveals that the 3D simulator exhibits bounded sensitivity at reading-scale movements, attributable to the absence of coupled head movement -- a finding that informs future simulator design. The pipeline, dataset, and evaluation tools are released to support downstream behavioral classifier development at the intersection of behavioral modeling and vision-language systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05475
PDF
-
Sharpness-Aware Surrogate Training for On-Sensor Spiking Neural Networks
Abstract
Spiking neural networks (SNNs) are a natural computational model for on-sensor and near-sensor vision, where event driven processors must operate under strict power budgets with hard binary spikes. However, models trained with surrogate gradients often degrade sharply when the smooth surrogate nonlinearity is replaced by a hard threshold at deployment; a surrogate-to-hard transfer gap that directly limits on-sensor accuracy. We study Sharpness-Aware Surrogate Training (SAST), which applies Sharpness-Aware Minimization (SAM) to a surrogate-forward SNN so that the training objective is smooth and the gradient is exact, and position it as one gap-reduction strategy under the tested settings rather than the only viable mechanism. Under explicit contraction assumptions we provide state-stability, input-Lipschitz, and smoothness bounds, together with a corresponding nonconvex convergence result. On two event-camera benchmarks, swap-only hard-spike accuracy improves from 65.7\% to 94.7\% on N-MNIST and from 31.8\% to 63.3\% on DVS Gesture. Under a hardware-aware inference simulation (INT8/INT4 weight quantization, fixed-point membrane potentials, discrete leak factors), SAST remains strong: on N-MNIST, hard-spike accuracy improves from 47.6\% to 96.9\% (INT8) and from 43.2\% to 81.0\% (INT4), while on DVS Gesture it improves from 25.3\% to 47.6\% (INT8) and from 26.0\% to 43.8\% (INT4). SynOps also decrease under the same hardware-aware setting, including 1734k$\rightarrow$1315k (N-MNIST, INT8) and 86221k$\rightarrow$4323k (DVS Gesture, INT8). These results suggest that SAST is a promising component in a broader toolbox for on-sensor spiking inference under the tested settings.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09696
PDF
-
SentiAvatar: Towards Expressive and Interactive Digital Humans
2026-04-03 09:26:28Chuhao Jin, Rui Zhang, Qingzhe Gao, Haoyu Shi, Dayu Wu, Yichen Jiang, Yihan Wu, Ruihua SongAbstract
We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at this https URL.
Abstract (translated)
我们提出了SentiAvatar，一个用于构建富有表现力的交互式3D数字人框架，并利用它创建了虚拟角色SuSu，该角色能够实时说话、做手势并表达情感。实现这样的系统仍具有挑战性，因为它需要共同解决三个关键问题：缺乏大规模、高质量的多模态数据，鲁棒的语义到运动映射，以及细粒度的帧级运动-韵律同步。为解决这些问题，首先，我们构建了SuSuInterActs（21,000个片段，37小时），这是一个通过光学运动捕捉采集的对话语料库，围绕单一角色录制，并同步了语音、全身运动和面部表情。其次，我们在200,000多个运动序列上预训练了一个运动基础模型，使其具备远超对话场景的丰富动作先验知识。接着，我们提出了一种音频感知的规划-再填充架构，将句子级语义规划与帧级韵律驱动的插值解耦，从而使生成的动作既在语义上恰当，又在节奏上与语音对齐。实验表明，SentiAvatar在SuSuInterActs（R@1 43.64%，接近最佳基线的2倍）和BEATv2（FGD 4.941，BC 8.078）上均达到最先进水平，仅需0.3秒即可生成6秒的输出，并支持无限多轮流式生成。源代码、模型和数据集可通过此https URL获取。
URL
https://arxiv.org/abs/2604.02908
PDF
-
How to measure the optimality of word or gesture order with respect to the principle of swap distance minimization
Abstract
The structure of all the permutations of a sequence can be represented as a permutohedron, a graph where vertices are permutations and two vertices are linked if a swap of adjacent elements in the permutation of one of the vertices produces the permutation of the other vertex. It has been hypothesized that word orders in languages minimize the swap distance in the permutohedron: given a source order, word orders that are closer in the permutohedron should be less costly and thus more likely. Here we explain how to measure the degree of optimality of word order variation with respect to swap distance minimization. We illustrate the power of our novel mathematical framework by showing that crosslinguistic gestures are at least $77\%$ optimal. It is unlikely that the multiple times where crosslinguistic gestures hit optimality are due to chance. We establish the theoretical foundations for research on the optimality of word or gesture order with respect to swap distance minimization in communication systems. Finally, we introduce the quadratic assignment problem (QAP) into language research as an umbrella for multiple optimization problems and, accordingly, postulate a general principle of optimal assignment that unifies various linguistic principles including swap distance minimization.
Abstract (translated)
一个序列的所有排列结构可表示为排列多面体（permutohedron），这是一种图结构：顶点代表排列，若一个顶点的排列中相邻元素交换后得到另一顶点的排列，则两顶点相连。有假说认为，语言中的词序会最小化排列多面体中的交换距离：给定一个源词序，在排列多面体中交换距离更近的词序成本更低，因而更可能出现。本文阐释了如何衡量词序变异在交换距离最小化方面的最优程度。通过证明跨语言手势至少达到77%的最优性，我们展示了这一新数学框架的效力。跨语言手势多次达到最优性不太可能出于偶然。我们为研究通信系统中词序或手势序在交换距离最小化方面的最优性奠定了理论基础。最后，我们将二次分配问题（QAP）引入语言研究，作为统摄多种优化问题的框架，并据此提出一个统一多种语言学原则（包括交换距离最小化）的一般最优分配原则。
URL
https://arxiv.org/abs/2604.01938
PDF
-
MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation
2026-03-31 04:49:53Takumi Taki, Masato Kobayashi, Yuki UranishiAbstract
Autonomous mobile robots operating in human-shared indoor environments often require paths that reflect human spatial intentions, such as avoiding interference with pedestrian flow or maintaining comfortable clearance. However, conventional path planners primarily optimize geometric costs and provide limited support for explicit route specification by human operators. This paper presents MRReP, a Mixed Reality-based interface that enables users to draw a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures. The drawn HRP is integrated into the robot navigation stack through a custom Hand-drawn Reference Path Planner, which converts the user-specified point sequence into a global path for autonomous navigation. We evaluated MRReP in a within-subject experiment against a conventional 2D baseline interface. The results demonstrated that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment. These findings suggest that direct path specification in MR is an effective approach for incorporating human spatial intention into mobile robot navigation. Additional material is available at this https URL
Abstract (translated)
在人类共享的室内环境中运行的自主移动机器人，通常需要其路径能够反映人类的空间意图，例如避免干扰人流或保持舒适的间距。然而，传统的路径规划器主要优化几何成本，并为人类操作员提供有限的显式路径指定支持。本文提出了MRReP，一个基于混合现实的界面，允许用户通过手势直接在物理地面上绘制手绘参考路径（HRP）。所绘制的手绘参考路径通过一个定制的手绘参考路径规划器集成到机器人导航栈中，该规划器将用户指定的点序列转换为自主导航的全局路径。我们在一个被试内实验中，将MRReP与传统的2D基线界面进行了对比评估。结果表明，MRReP提高了路径指定的准确性、可用性，并降低了感知工作负荷，同时实现了在物理环境中更稳定的路径指定。这些发现表明，在混合现实中直接指定路径是将人类空间意图融入移动机器人导航的有效方法。更多材料可通过此https URL获取。
URL
https://arxiv.org/abs/2604.00059
PDF
-
Evaluation of Generative Models for Emotional 3D Animation Generation in VR
2026-03-30 23:16:13Kiran Chhatre, Renan Guarese, Andrii Matviienko, Christopher PetersAbstract
Social interactions incorporate nonverbal signals to convey emotions alongside speech, including facial expressions and body gestures. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of model effectiveness. To address this, we evaluate emotional 3D animation generative models within a Virtual Reality (VR) environment, emphasizing user-centric metrics emotional arousal realism, naturalness, enjoyment, diversity, and interaction quality in a real-time human-agent interaction scenario. Through a user study (N=48), we examine perceived emotional quality for three state of the art speech-driven 3D animation methods across two emotions happiness (high arousal) and neutral (mid arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated the realism and naturalness of happy animations significantly higher than those of neutral animations, highlighting the limitations of current generative models in handling subtle emotional states. Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.
Abstract (translated)
社交互动通过言语与非语言信号（如面部表情和肢体手势）共同传递情感。生成模型在创建与语音同步的全身非语言动画方面已展现出前景；然而，基于二维场景的统计指标评估未能充分捕捉用户感知的情感，制约了我们对模型有效性的理解。为解决这一问题，我们在虚拟现实（VR）环境中评估情感三维动画生成模型，重点关注实时人机交互场景中以用户为中心的指标：情感唤醒真实感、自然度、愉悦感、多样性及交互质量。通过一项用户研究（N=48），我们考察了三种最先进的语音驱动三维动画方法在两种情感（高兴[高唤醒]与中性[中唤醒]）下的感知情感质量。此外，我们将这些生成模型与通过基于重建方法获取的真实人类表情进行对比，以评估其优势与局限，以及它们在多大程度上复现了真实人类的面部与身体表达。研究结果表明，显式建模情感的方法相较于仅关注语音同步的方法具有更高的识别准确率。用户对高兴动画的真实感与自然度评分显著高于中性动画，凸显了当前生成模型在处理微妙情感状态时的局限性。在面部表情质量上，生成模型表现低于基于重建的方法；所有方法在动画愉悦感与交互质量上均获得相对低分，强调了将用户为中心评估纳入生成模型开发的重要性。最后，参与者对所有生成模型的动画多样性均给予积极认可。
URL
https://arxiv.org/abs/2512.16081
PDF
-
TinyGaze: Lightweight Gaze-Gesture Recognition on Commodity Mobile Devices
2026-03-30 14:30:37Yaxiong Lei, Hyochan Cho, Fergus Buchanan, Shijing He, Xinya Gong, Yuheng Wang, Juan YeAbstract
Gaze gestures can provide hands free input on mobile devices, but practical use requires (i) gestures users can learn and recall and (ii) recognition models that are efficient enough for on-device deployment. We present an end-to-end pipeline using commodity ARKit head/eye transforms and a scaffolded guidance-to-recall protocol grounded in learning theory. In a pilot feasibility study (N=4 participants; 240 trials; controlled single-session setting), we benchmark a compact time-series model (TinyHAR) against deeper baselines (DeepConvLSTM, SA-HAR) on 5-way gesture recognition and 4-way user identification. TinyHAR achieves strong performance in this pilot benchmark (Macro F1 = 0.960 for gesture recognition; Macro F1 = 0.997 for user identification) while using only 46k parameters. A modality analysis further indicates that head pose dynamics are highly informative for mobile gaze gestures, highlighting embodied head--eye coordination as a key design consideration. Although the small sample size and controlled setting limit generalizability, these results indicate a potential direction for further investigation into on-device gaze gesture recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09658
PDF
-
Users and Wizards in Conversations: How WoZ Interface Choices Define Human-Robot Interactions
2026-03-30 12:09:58Ekaterina Torubarova, Jura Miniota, Andre PereiraAbstract
In this paper, we investigated how the choice of a Wizard-of-Oz (WoZ) interface affects communication with a robot from both the user's and the wizard's perspective. In a conversational setting, we used three WoZ interfaces with varying levels of dialogue input and output restrictions: a) a restricted perception GUI that showed fixed-view video and ASR transcripts and let the wizard trigger pre-scripted utterances and gestures; b) an unrestricted perception GUI that added real-time audio from the participant and the robot c) a VR telepresence interface that streamed immersive stereo video and audio to the wizard and forwarded the wizard's spontaneous speech, gaze and facial expressions to the robot. We found that the interaction mediated by the VR interface was preferred by users in terms of robot features and perceived social presence. For the wizards, the VR condition turned out to be the most demanding but elicited a higher social connection with the users. VR interface also induced the most connected interaction in terms of inter-speaker gaps and overlaps, while Restricted GUI induced the least connected flow and the largest silences. Given these results, we argue for more WoZ studies using telepresence interfaces. These studies better reflect the robots of tomorrow and offer a promising path to automation based on naturalistic contextualized verbal and non-verbal behavioral data.
Abstract (translated)
本文研究了绿野仙踪（WoZ）接口的选择如何从用户和操作者两个角度影响与机器人的交互。在对话场景中，我们使用了三种对话输入输出限制程度不同的WoZ接口：a）受限感知图形界面，显示固定视角视频和自动语音识别文本，允许操作者触发预设的言语和手势；b）无限制感知图形界面，在a基础上增加了参与者和机器人的实时音频；c）VR临场感接口，向操作者传输沉浸式立体视频和音频，并将操作者的自发言语、注视和面部表情转发给机器人。研究发现，经VR接口中介的交互在机器人特征和社会临场感方面更受用户青睐。对操作者而言，VR条件要求最高，却引发了与用户更强的社会联结。就说话者间隙与重叠而言，VR接口促成了最具联结感的交互，而受限图形界面则导致联结感最弱、静默时间最长。基于这些结果，我们主张更多采用临场感接口开展WoZ研究。这类研究能更好地反映未来机器人的形态，并为基于自然情境下言语与非言语行为数据的自动化路径提供前景。
URL
https://arxiv.org/abs/2603.28338
PDF
-
Point of View: How Perspective Affects Perceived Robot Sociability
2026-03-30 10:54:59Subham Agrawal, Aftab Akthar, Nils Dengler, Maren BennewitzAbstract
Ensuring that robot navigation is safe and socially acceptable is crucial for comfortable human-robot interaction in shared environments. However, existing validation methods often rely on a bird's-eye (allocentric) perspective, which fails to capture the subjective first-person experience of pedestrians encountering robots in the real world. In this paper, we address the perceptual gap between allocentric validation and egocentric experience by investigating how different perspectives affect the perceived sociability and disturbance of robot trajectories. Our approach uses an immersive VR environment to evaluate identical robot trajectories across allocentric, egocentric-proximal, and egocentric-distal viewpoints in a user study. We perform this analysis for trajectories generated from two different navigation policies to understand if the observed differences are unique to a single type of trajectory or more generalizable. We further examine whether augmenting a trajectory with a head-nod gesture can bridge the perceptual gap and improve human comfort. Our experiments suggest that trajectories rated as sociable from an allocentric view may be perceived as significantly more disturbing when experienced from a first-person perspective in close proximity. Our results also demonstrate that while passing distance affects perceived disturbance, communicative social signaling, such as a head-nod, can effectively enhance the perceived sociability of the robot's behavior.
Abstract (translated)
确保机器人在共享环境中的导航安全且符合社交规范，对于实现舒适的人机交互至关重要。然而，现有验证方法通常依赖鸟瞰（全视角）视角，无法捕捉行人在现实世界中遭遇机器人时的第一人称主观体验。本文通过研究不同视角如何影响对机器人轨迹的社交性与干扰性感知，旨在弥合全视角验证与自我视角体验之间的感知差距。我们的方法利用沉浸式虚拟现实环境，在用户研究中从全视角、自我视角-近距和自我视角-远距三个视角评估相同的机器人轨迹。我们针对两种不同导航策略生成的轨迹进行分析，以理解观察到的差异是否仅针对单一轨迹类型，还是具有更广泛的普遍性。我们还进一步检验了为轨迹添加点头手势是否能够弥合感知差距并提升人类舒适度。实验表明，在全视角下被评为具有社交性的轨迹，当以第一人称近距离体验时可能被视为显著更具干扰性。结果同时显示，虽然通行距离会影响感知干扰程度，但点头等交际性社交信号能有效提升机器人行为的感知社交性。
URL
https://arxiv.org/abs/2603.28272
PDF
-
FlashSign: Pose-Free Guidance for Efficient Sign Language Video Generation
2026-03-30 00:06:26Liuzhou Zhang, Zeyu Zhang, Biao Wu, Luyao Tang, Zirui Song, Hongyang He, Renda Han, Guangzhen Yao, Huacan Wang, Ronghao Chen, Xiuying Chen, Guan Huang, Zheng ZhuAbstract
Sign language plays a crucial role in bridging communication gaps between the deaf and hard-of-hearing communities. However, existing sign language video generation models often rely on complex intermediate representations, which limits their flexibility and efficiency. In this work, we propose a novel pose-free framework for real-time sign language video generation. Our method eliminates the need for intermediate pose representations by directly mapping natural language text to sign language videos using a diffusion-based approach. We introduce two key innovations: (1) a pose-free generative model based on the a state-of-the-art diffusion backbone, which learns implicit text-to-gesture alignments without pose estimation, and (2) a Trainable Sliding Tile Attention (T-STA) mechanism that accelerates inference by exploiting spatio-temporal locality patterns. Unlike previous training-free sparsity approaches, T-STA integrates trainable sparsity into both training and inference, ensuring consistency and eliminating the train-test gap. This approach significantly reduces computational overhead while maintaining high generation quality, making real-time deployment feasible. Our method increases video generation speed by 3.07x without compromising video quality. Our contributions open new avenues for real-time, high-quality, pose-free sign language synthesis, with potential applications in inclusive communication tools for diverse communities. Code: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27915
PDF
-
LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
2026-03-28 23:58:29Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue WangAbstract
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.
Abstract (translated)
学习人-物操纵因其动作的精细性和高接触性而面临重大挑战。传统基于物理的动画需要大量建模和手动设置，更重要的是，它既不能很好地泛化到不同形态的物体，也无法有效扩展到真实环境。为应对这些局限，我们提出了LOME——一个以自我为中心的世界模型，能够根据输入图像、文本提示以及每帧的人体动作（包括身体姿态和手势）生成逼真的人-物交互视频。LOME通过在训练中联合估计人体空间动作和环境上下文，将强大而精确的动作引导注入物体操纵中。在对预训练视频生成模型进行多样化第一人称人-物交互视频微调后，LOME不仅展现出高动作跟随精度和对未知场景的强泛化能力，还能呈现手-物交互的真实物理后果，例如执行“倾倒”动作后液体从瓶中流入杯子的效果。大量实验表明，我们的视频框架在时间一致性和运动控制方面，均显著优于最先进的图像/视频动作条件化方法以及图像/文本到视频（I/T2V）生成模型。LOME为照片般真实的AR/VR体验和可扩展的机器人训练铺平了道路，且不受模拟环境限制或依赖显式3D/4D建模。
URL
https://arxiv.org/abs/2603.27449
PDF
-
HolisticSemGes: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching
2026-03-27 16:11:44Lanmiao Liu, Esam Ghaleb, Asl{\i} \"Ozy\"urek, Zerrin YumakAbstract
While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two datasets, BEAT2 and SHOW.
Abstract (translated)
尽管共语音手势生成领域已取得显著进展，但生成整体性、语义合理的手势仍是一项挑战。现有方法依赖外部语义检索，因其依赖预定义语言规则而限制了泛化能力。基于流匹配的方法虽产生令人期待的结果，但网络仅使用语义一致的样本进行优化，缺乏对负样本的接触，导致其学习到的是节奏性手势而非稀疏运动（如图示性手势和隐喻性手势）。此外，多数方法因孤立建模身体部位而无法保持跨模态一致性。本文提出一种基于对比流匹配的共语音手势生成模型，利用不匹配的音频-文本条件作为负样本，训练速度场遵循正确的运动轨迹同时排斥语义不一致的轨迹。该模型通过余弦与对比目标将文本、音频和整体运动嵌入复合潜在空间，确保跨模态连贯性。大量实验与用户研究表明，所提方法在BEAT2和SHOW两个数据集上均优于现有最先进方法。
URL
https://arxiv.org/abs/2603.26553
PDF
-
T-800: An 800 Hz Data Glove for Precise Hand Gesture Tracking
2026-03-27 13:27:06Haoyang Luo, Zihang Zhao, Leiyao Cui, Saiyao Zhang, Liu Yang, Zhi Han, Xiyuan Tang, Yixin ZhuAbstract
Human dexterity relies on rapid, sub-second motor adjustments, yet capturing these high-frequency dynamics remains an enduring challenge in biomechanics and robotics. Existing motion capture paradigms are compromised by a trade-off between temporal resolution and visual occlusion, failing to record the fine-grained hand motion of fast, contact-rich manipulation. Here we introduce T-800, a high-bandwidth data glove system that achieves synchronized, full-hand motion tracking at 800 Hz. By integrating a novel broadcast-based synchronization mechanism with a mechanical stress isolation architecture, our system maintains sub-frame temporal alignment across 18 distributed inertial measurement units (IMUs) during extended, vigorous movements. We demonstrate that T-800 recovers fine-grained manipulation details previously lost to temporal undersampling. Our analysis reveals that human dexterity exhibits significantly high-frequency motion energy (>100 Hz) that was fundamentally inaccessible due to the Nyquist sampling limit imposed by previous hardware constraints. To validate the system's utility for robotic manipulation, we implement a kinematic retargeting algorithm that maps T-800's high-fidelity human gestures onto dexterous robotic hand models. This demonstrates that the high-frequency motion data can be accurately translated while respecting the kinematic constraints of robotic hands, providing the rich behavioral data necessary for training robust control policies in the future.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26403
PDF
-
SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training
2026-03-27 13:24:41Le Ma, Thiago Freitas dos Santos, Nadia Magnenat-Thalmann, Katarzyna WacAbstract
In surgical training for medical students, proficiency development relies on expert-led skill assessment, which is costly, time-limited, difficult to scale, and its expertise remains confined to institutions with available specialists. Automated AI-based assessment offers a viable alternative, but progress is constrained by the lack of datasets containing realistic trainee errors and the multi-view variability needed to train robust computer vision approaches. To address this gap, we present Surgical-Hands (SHands), a large-scale multi-view video dataset for surgical hand-gesture and error recognition for medical training. \textsc{SHands} captures linear incision and suturing using five RGB cameras from complementary viewpoints, performed by 52 participants (20 experts and 32 trainees), each completing three standardized trials per procedure. The videos are annotated at the frame level with 15 gesture primitives and include a validated taxonomy of 8 trainee error types, enabling both gesture recognition and error detection. We further define standardized evaluation protocols for single-view, multi-view, and cross-view generalization, and benchmark state-of-the-art deep learning models on the dataset. SHands is publicly released to support the development of robust and scalable AI systems for surgical training grounded in clinically curated domain knowledge.
Abstract (translated)
在医学生手术培训中，技能熟练度的发展依赖于专家主导的技能评估，但这种方式成本高昂、时间有限、难以规模化，且专业知识仅限于拥有专科医师的医疗机构。基于人工智能的自动化评估提供了一种可行的替代方案，然而其进展受限于缺乏包含真实学员错误的数据集，以及训练鲁棒计算机视觉方法所需的多视角变化。为填补这一空白，我们提出了Surgical-Hands（SHands）——一个用于手术手势与错误识别的大规模多视角视频数据集，旨在支持医学培训。SHands通过五个互补视角的RGB摄像头采集了线性切口与缝合操作，由52名参与者（20名专家与32名学员）完成，每人每项操作进行三次标准化试验。视频在帧级别标注了15种手势基元，并包含经过验证的8类学员错误分类体系，可同时支持手势识别与错误检测。我们还定义了单视角、多视角及跨视角泛化的标准化评估协议，并在该数据集上对前沿深度学习模型进行了基准测试。SHands已公开发布，旨在支持基于临床精选领域知识的、稳健且可扩展的手术培训AI系统发展。
URL
https://arxiv.org/abs/2603.26400
PDF
-
HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models
2026-03-27 12:42:26MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul BaekAbstract
Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).
Abstract (translated)
深入理解人手的精细关节运动，在机器人辅助手术、芯片制造以及基于AR/VR的人机交互等高风险场景中至关重要。尽管当前视觉语言模型（VLMs）在通用视觉语言基准测试中已接近人类表现，但它们仍难以胜任精细的空间推理任务，尤其在解读复杂且多关节的手部姿态时。我们推出了HandVQA——一个大规模诊断性基准，旨在通过视觉问答评估VLMs对手部精细解剖结构的理解。该基准基于高质量3D手部数据集（FreiHAND、InterHand2.6M、FPHA），包含超过160万道受控的多项选择题，用于探测手部关节间的空间关系，如角度、距离和相对位置。我们使用轻量级LoRA微调方法，在基础模型和微调后两种设置下，对多个前沿视觉语言模型（LLaVA、DeepSeek、Qwen-VL）进行了评估。研究发现当前模型存在系统性局限：包括虚构手指部件、几何解释错误以及泛化能力不足。HandVQA不仅揭示了这些关键推理缺陷，还提供了经验证的改进路径。我们证明，从该基准学到的3D空间知识能够实现零样本迁移，显著提升模型在新任务上的准确率：手势识别提升10.33%，手物交互识别提升2.63%。
URL
https://arxiv.org/abs/2603.26362
PDF
-
Dynamic LIBRAS Gesture Recognition via CNN over Spatiotemporal Matrix Representation
Abstract
This paper proposes a method for dynamic hand gesture recognition based on the composition of two models: the MediaPipe Hand Landmarker, responsible for extracting 21 skeletal keypoints of the hand, and a convolutional neural network (CNN) trained to classify gestures from a spatiotemporal matrix representation of dimensions 90 by 21 of those keypoints. The method is applied to the recognition of LIBRAS (Brazilian Sign Language) gestures for device control in a home automation system, covering 11 classes of static and dynamic gestures. For real-time inference, a sliding window with temporal frame triplication is used, enabling continuous recognition without recurrent networks. Tests achieved 95\% accuracy under low-light conditions and 92\% under normal lighting. The results indicate that the approach is effective, although systematic experiments with greater user diversity are needed for a more thorough evaluation of generalization.
Abstract (translated)
本文提出一种基于两种模型组合的动态手势识别方法：MediaPipe手部关键点检测器负责提取21个手部骨骼关键点，卷积神经网络（CNN）则对这些关键点形成的90×21维时空矩阵进行手势分类。该方法应用于巴西手语（LIBRAS）手势识别，以实现智能家居设备控制，涵盖11类静态与动态手势。为实现实时推理，采用带时间帧复制的滑动窗口机制，无需循环网络即可实现连续识别。测试表明，该方法在低光照条件下准确率达95%，正常光照下为92%。结果表明该方案有效，但需通过更多样化的用户实验来进一步评估其泛化能力。
URL
https://arxiv.org/abs/2603.25863
PDF
