Gesture
-
iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis
2026-02-25 00:38:19Sofoklis Kakouros, Fang Kang, Haoyu ChenAbstract
This work presents iMiGUE-Speech, an extension of the iMiGUE dataset that provides a spontaneous affective corpus for studying emotional and affective states. The new release focuses on speech and enriches the original dataset with additional metadata, including speech transcripts, speaker-role separation between interviewer and interviewee, and word-level forced alignments. Unlike existing emotional speech datasets that rely on acted or laboratory-elicited emotions, iMiGUE-Speech captures spontaneous affect arising naturally from real match outcomes. To demonstrate the utility of the dataset and establish initial benchmarks, we introduce two evaluation tasks for comparative assessment: speech emotion recognition and transcript-based sentiment analysis. These tasks leverage state-of-the-art pre-trained representations to assess the dataset's ability to capture spontaneous affective states from both acoustic and linguistic modalities. iMiGUE-Speech can also be synchronously paired with micro-gesture annotations from the original iMiGUE dataset, forming a uniquely multimodal resource for studying speech-gesture affective dynamics. The extended dataset is available at this https URL.
Abstract (translated)
这项工作介绍了iMiGUE-Speech，这是对iMiGUE数据集的扩展，提供了用于研究情绪和情感状态的自发性情感语料库。新版本专注于语音，并通过添加额外的元数据来丰富原始数据集，包括语音转录、访谈者与受访者的角色分离以及词级别强制对齐信息。 不同于现有的基于表演或实验室诱发的情绪语音数据集，iMiGUE-Speech 捕捉到了自然来源于真实比赛结果中的自发性情感。为了展示该数据集的实用性并建立初步基准，我们引入了两个用于比较评估的任务：语音情绪识别和基于转录的情感分析。这些任务利用最先进的预训练表示来评估数据集捕捉来自声学和语言模态的自发情感状态的能力。 iMiGUE-Speech还可以与原始iMiGUE数据集中微手势注释同步配对，形成一个独特的多模式资源，用于研究语音-手势情感动态。扩展后的数据集可以在[此链接](https://this https URL/)获取。
URL
https://arxiv.org/abs/2602.21464
PDF
-
SARAH: Spatially Aware Real-time Agentic Humans
2026-02-20 18:59:35Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander RichardAbstract
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see this https URL for details.
Abstract (translated)
随着虚拟现实（VR）、远程呈现和数字人类应用的发展，实体代理的角色变得越来越重要。这些代理的动作需要超越与语音同步的手势，它们应该能够面向用户、响应用户的移动，并保持自然的眼神交流。目前的方法缺乏这种空间感知能力。我们填补了这一空白，推出了首个实时的、完全因果关系的空间感知对话动作生成方法，能够在流媒体VR头显上部署。 我们的方法基于给定用户的位置和双向音频信息来生成全身运动，使手势与语音同步的同时根据用户的移动调整代理的方向。我们的架构结合了一个因果Transformer-based变分自编码器（VAE）以及交错的隐变量令牌以支持流式推理，还有一个受控于用户轨迹和音频的数据匹配模型。 为了适应不同的眼神交流偏好，我们引入了一个眼神评分机制，并使用无分类指导来分离学习与控制：模型从数据中捕捉自然的空间对应关系，而用户则可以在推断时调整目光接触的强度。在Embody 3D数据集上，我们的方法实现了超过每秒300帧的速度（比非因果基准快三倍）并捕获了自然对话中的微妙空间动态变化。 我们通过一个实时VR系统验证了这一方案的有效性，将具有空间感知能力的对话代理带到了现实世界的部署中。详情请参阅此[链接](https://example.com)。
URL
https://arxiv.org/abs/2602.18432
PDF
-
HiAER-Spike Software-Hardware Reconfigurable Platform for Event-Driven Neuromorphic Computing at Scale
2026-02-20 08:55:47Gwenevere Frank, Gopabandhu Hota, Keli Wang, Christopher Deng, Krish Arora, Diana Vins, Abhinav Uppal, Omowuyi Olajide, Kenneth Yoshimoto, Qingbo Wang, Mari Yamaoka, Johannes Leugering, Stephen Deiss, Leif Gibb, Gert CauwenberghsAbstract
In this work, we present HiAER-Spike, a modular, reconfigurable, event-driven neuromorphic computing platform designed to execute large spiking neural networks with up to 160 million neurons and 40 billion synapses - roughly twice the neurons of a mouse brain at faster than real time. This system, assembled at the UC San Diego Supercomputer Center, comprises a co-designed hard- and software stack that is optimized for run-time massively parallel processing and hierarchical address-event routing (HiAER) of spikes while promoting memory-efficient network storage and execution. The architecture efficiently handles both sparse connectivity and sparse activity for robust and low-latency event-driven inference for both edge and cloud computing. A Python programming interface to HiAER-Spike, agnostic to hardware-level detail, shields the user from complexity in the configuration and execution of general spiking neural networks with minimal constraints in topology. The system is made easily available over a web portal for use by the wider community. In the following, we provide an overview of the hard- and software stack, explain the underlying design principles, demonstrate some of the system's capabilities and solicit feedback from the broader neuromorphic community. Examples are shown demonstrating HiAER-Spike's capabilities for event-driven vision on benchmark CIFAR-10, DVS event-based gesture, MNIST, and Pong tasks.
Abstract (translated)
在这项工作中，我们介绍了HiAER-Spike，这是一个模块化、可重构的事件驱动神经形态计算平台，旨在执行高达1.6亿个神经元和400亿个突触的大规模脉冲神经网络——这大约是老鼠大脑神经元数量的两倍，并且运行速度超过实时。该系统在加州大学圣地亚哥分校超级计算机中心组装而成，它包含了一个专门为大规模并行处理而优化、采用层次化地址事件路由（HiAER）的软硬件堆栈，在提高内存效率的同时支持网络存储和执行。这种架构能够高效地应对稀疏连接和稀疏活动，适用于边缘计算和云计算中的稳健且低延迟的事件驱动推理。我们还提供了一个Python编程接口给HiAER-Spike，该接口不依赖于硬件细节，并能简化用户在配置和执行一般脉冲神经网络时的操作复杂性，仅需遵守有限的拓扑约束。通过一个网页门户，这个系统被广泛地社区成员所轻松访问并使用。 接下来，我们概述了软硬件堆栈，解释了背后的设计原则，并展示了该系统的部分能力，同时邀请更广泛的神经形态计算社区提供反馈意见。文中还提供了实例展示HiAER-Spike在事件驱动视觉（基准CIFAR-10）、DVS基于事件的手势识别、MNIST任务和Pong游戏等领域的应用能力。
URL
https://arxiv.org/abs/2602.18072
PDF
-
On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective
Abstract
This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szirányi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion matrix, learning curves, and dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in vision-based gesture-recognition research, especially for applications - such as UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.
Abstract (translated)
本文对刘和Szirányi提出的姿态识别方法进行了方法论分析，特别关注其评估协议的有效性。我们展示了所报告的近乎完美的准确率指标是由帧级别的随机训练-测试拆分导致的结果，这种拆分不可避免地将同一主体的数据混入了两个数据集之中，从而造成了严重的信息泄露问题。通过审查已发布的混淆矩阵、学习曲线以及数据集构建情况，我们证明该评估无法衡量对未见过个体的泛化能力。我们的研究结果强调，在基于视觉的姿态识别研究中进行独立于主体的数据拆分的重要性，尤其是在如无人机与人交互这样的应用领域，这些应用需要可靠地识别人群中新出现的手势。
URL
https://arxiv.org/abs/2602.17854
PDF
-
FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations
2026-02-19 17:31:08Konstantinos Foteinos, Georgios Angelidis, Aggelos Psiris, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. PapadopoulosAbstract
The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.17573
PDF
-
Advanced Acceptance Score: A Holistic Measure for Biometric Quantification
2026-02-17 12:33:45Aman Verma, Seshan Srirangarajan, Sumantra Dutta RoyAbstract
Quantifying biometric characteristics within hand gestures involve derivation of fitness scores from a gesture and identity aware feature space. However, evaluating the quality of these scores remains an open question. Existing biometric capacity estimation literature relies upon error rates. But these rates do not indicate goodness of scores. Thus, in this manuscript we present an exhaustive set of evaluation measures. We firstly identify ranking order and relevance of output scores as the primary basis for evaluation. In particular, we consider both rank deviation as well as rewards for: (i) higher scores of high ranked gestures and (ii) lower scores of low ranked gestures. We also compensate for correspondence between trends of output and ground truth scores. Finally, we account for disentanglement between identity features of gestures as a discounting factor. Integrating these elements with adequate weighting, we formulate advanced acceptance score as a holistic evaluation measure. To assess effectivity of the proposed we perform in-depth experimentation over three datasets with five state-of-the-art (SOTA) models. Results show that the optimal score selected with our measure is more appropriate than existing other measures. Also, our proposed measure depicts correlation with existing measures. This further validates its reliability. We have made our \href{this https URL}{code} public.
Abstract (translated)
URL
https://arxiv.org/abs/2602.15535
PDF
-
ProAct: A Dual-System Framework for Proactive Embodied Social Agents
2026-02-15 08:27:34Zeyi Zhang, Zixi Kang, Ruijie Zhao, Yusen Feng, Biao Jiang, Libin LiuAbstract
Embodied social agents have recently advanced in generating synchronized speech and gestures. However, most interactive systems remain fundamentally reactive, responding only to current sensory inputs within a short temporal window. Proactive social behavior, in contrast, requires deliberation over accumulated context and intent inference, which conflicts with the strict latency budget of real-time interaction. We present \emph{ProAct}, a dual-system framework that reconciles this time-scale conflict by decoupling a low-latency \emph{Behavioral System} for streaming multimodal interaction from a slower \emph{Cognitive System} which performs long-horizon social reasoning and produces high-level proactive intentions. To translate deliberative intentions into continuous non-verbal behaviors without disrupting fluency, we introduce a streaming flow-matching model conditioned on intentions via ControlNet. This mechanism supports asynchronous intention injection, enabling seamless transitions between reactive and proactive gestures within a single motion stream. We deploy ProAct on a physical humanoid robot and evaluate both motion quality and interactive effectiveness. In real-world interaction user studies, participants and observers consistently prefer ProAct over reactive variants in perceived proactivity, social presence, and overall engagement, demonstrating the benefits of dual-system proactive control for embodied social interaction.
Abstract (translated)
URL
https://arxiv.org/abs/2602.14048
PDF
-
Resource-Efficient Gesture Recognition through Convexified Attention
2026-02-13 15:37:02Daniel Schwartz, Dario Salvucci, Yusuf Osmanlioglu, Richard Vallett, Genevieve Dion, Ali ShokoufandehAbstract
Wearable e-textile interfaces require gesture recognition capabilities but face severe constraints in power consumption, computational capacity, and form factor that make traditional deep learning impractical. While lightweight architectures like MobileNet improve efficiency, they still demand thousands of parameters, limiting deployment on textile-integrated platforms. We introduce a convexified attention mechanism for wearable applications that dynamically weights features while preserving convexity through nonexpansive simplex projection and convex loss functions. Unlike conventional attention mechanisms using non-convex softmax operations, our approach employs Euclidean projection onto the probability simplex combined with multi-class hinge loss, ensuring global convergence guarantees. Implemented on a textile-based capacitive sensor with four connection points, our approach achieves 100.00\% accuracy on tap gestures and 100.00\% on swipe gestures -- consistent across 10-fold cross-validation and held-out test evaluation -- while requiring only 120--360 parameters, a 97\% reduction compared to conventional approaches. With sub-millisecond inference times (290--296$\mu$s) and minimal storage requirements ($<$7KB), our method enables gesture interfaces directly within e-textiles without external processing. Our evaluation, conducted in controlled laboratory conditions with a single-user dataset, demonstrates feasibility for basic gesture interactions. Real-world deployment would require validation across multiple users, environmental conditions, and more complex gesture vocabularies. These results demonstrate how convex optimization can enable efficient on-device machine learning for textile interfaces.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13030
PDF
-
Your Robot Will Feel You Now: Empathy in Robots and Embodied Agents
2026-02-12 22:07:44Angelica Lim, \"O. Nilay Yal\c{c}inAbstract
The fields of human-robot interaction (HRI) and embodied conversational agents (ECAs) have long studied how empathy could be implemented in machines. One of the major drivers has been the goal of giving multimodal social and emotional intelligence to these artificially intelligent agents, which interact with people through facial expressions, body, gesture, and speech. What empathic behaviors and models have these fields implemented by mimicking human and animal behavior? In what ways have they explored creating machine-specific analogies? This chapter aims to review the knowledge from these studies, towards applying the lessons learned to today's ubiquitous, language-based agents such as ChatGPT.
Abstract (translated)
人机交互（HRI）与具身智能体（ECAs）领域长期以来一直研究如何在机器中实现共情。其主要目标之一是为这些通过面部表情、身体动作、手势和语音与人交互的人工智能体赋予多模态社交与情感智能。这些领域通过模仿人类与动物行为，实现了哪些共情行为与模型？它们又以何种方式探索了创造机器特有的类比？本章旨在回顾这些研究的成果，从而将经验应用于当今无处不在的语言模型（如ChatGPT）。
URL
https://arxiv.org/abs/2603.20200
PDF
-
MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery
2026-02-12 20:56:15Keshara Weerasinghe (MD), Seyed Hamid Reza Roodabeh (MD), Andrew Hawkins (MD), Zhaomeng Zhang, Zachary Schrader, Homa AlemzadehAbstract
Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.
Abstract (translated)
URL
https://arxiv.org/abs/2602.12407
PDF
-
Energy-Aware Spike Budgeting for Continual Learning in Spiking Neural Networks for Neuromorphic Vision
2026-02-12 18:15:32Anika Tabassum Meem, Muntasir Hossain Nadid, Md Zesun Ahmed MiaAbstract
Neuromorphic vision systems based on spiking neural networks (SNNs) offer ultra-low-power perception for event-based and frame-based cameras, yet catastrophic forgetting remains a critical barrier to deployment in continually evolving environments. Existing continual learning methods, developed primarily for artificial neural networks, seldom jointly optimize accuracy and energy efficiency, with particularly limited exploration on event-based datasets. We propose an energy-aware spike budgeting framework for continual SNN learning that integrates experience replay, learnable leaky integrate-and-fire neuron parameters, and an adaptive spike scheduler to enforce dataset-specific energy constraints during training. Our approach exhibits modality-dependent behavior: on frame-based datasets (MNIST, CIFAR-10), spike budgeting acts as a sparsity-inducing regularizer, improving accuracy while reducing spike rates by up to 47\%; on event-based datasets (DVS-Gesture, N-MNIST, CIFAR-10-DVS), controlled budget relaxation enables accuracy gains up to 17.45 percentage points with minimal computational overhead. Across five benchmarks spanning both modalities, our method demonstrates consistent performance improvements while minimizing dynamic power consumption, advancing the practical viability of continual learning in neuromorphic vision systems.
Abstract (translated)
基于脉冲神经网络（SNN）的类脑视觉系统为事件驱动和帧基摄像头提供了超低功耗感知能力，但灾难性遗忘仍然是在不断变化环境中部署的关键障碍。现有的持续学习方法主要是针对人工神经网络开发的，很少同时优化准确性和能源效率，并且对事件数据集的研究尤其有限。我们提出了一种能耗感知的脉冲预算框架，用于连续SNN学习，该框架结合了经验回放、可学习的漏电积分发放神经元参数以及自适应脉冲调度器，在训练过程中执行特定于数据集的能量约束。我们的方法表现出模态依赖性行为：在帧基数据集（MNIST, CIFAR-10）上，脉冲预算起到了促进稀疏性的正则化作用，提高了准确性同时将脉冲率减少了最多47%；而在事件驱动的数据集（DVS-Gesture, N-MNIST, CIFAR-10-DVS）上，控制性预算放松能够使准确度提升高达17.45个百分点，并且计算开销极小。在横跨两种模态的五个基准测试中，我们的方法展示了一致性的性能改进并同时最小化了动态功耗，从而推进了类脑视觉系统中持续学习的实际可行性。
URL
https://arxiv.org/abs/2602.12236
PDF
-
Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures
2026-02-10 09:51:07Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, Xingang PanAbstract
Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.
Abstract (translated)
自体中心互动世界模型对于增强现实和具身人工智能至关重要，在这些领域中，视觉生成必须能够快速响应用户输入，并且保持几何一致性和长期稳定性。本文研究了仅基于单一场景图像以及自由空间手势的情况下进行自体中心交互生成的方法，旨在合成这样的视频：在其中手进入场景、与物体互动并根据头部运动产生合理的世界动态变化。这种设置引入了许多基本挑战，包括自由空间手势和接触密集型训练数据之间的分布差异、单目视角下手部动作和相机移动的模糊性以及任意长度视频生成的需求。 我们提出了Hand2World框架，这是一个统一的自回归架构，通过基于投影3D手模型的手部条件处理解决了这些问题。这种处理方式使得可视性和遮挡可以从场景上下文中推断出来而不是直接编码到控制信号中。为了稳定自体视角的变化，我们在每像素Plücker射线嵌入的基础上注入了显式的相机几何信息，从而将相机移动与手部动作区分开来，并防止背景漂移现象的发生。 我们还开发了一条全自动单目注释流水线并从双向扩散模型中提取出了因果生成器，这使得任意长度的合成成为了可能。我们在三个自体中心交互基准上进行了实验，结果显示在感知质量、3D一致性以及支持相机控制和长期互动生成方面取得了显著改进。 简单来说，Hand2World框架通过处理手势与场景之间的关系，并结合先进的模型简化技术，能够生成更加真实且稳定的交互式视频内容，在增强现实等领域具有广泛的应用前景。
URL
https://arxiv.org/abs/2602.09600
PDF
-
Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks
2026-02-09 17:13:34Kiyosu Maeda, William P. McCarthy, Ching-Yi Tsai, Jeffrey Mu, Haoliang Wang, Robert D. Hawkins, Judith E. Fan, Parastoo AbtahiAbstract
A quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner's hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.
Abstract (translated)
人类智能的一个典型特征是随着时间的推移，能够创建临时约定以高效地实现共同目标。我们研究了通过反复合作，沟通策略如何随着人们在共享过程抽象上进行协调而演变。为此，我们开展了一项在线单模态研究（n = 98），使用自然语言来探索抽象层次结构。在后续的实验室研究（n = 40）中，我们考察了多模态通信（包括言语和手势）在物理协作过程中如何变化。实验中，成对参与者通过增强现实技术隔离彼此的手部动作和声音；一名参与者观察一个3D虚拟塔楼，并向另一名正在构建实体塔楼的同伴发出指令。参与者通过建立语言和手势抽象，并利用跨模态冗余来强调先前互动中的关键变化而变得更快且更准确。 基于这些发现，我们将概率模型在约定形成方面的应用扩展到了多模态环境，捕捉到模式偏好转变的趋势。我们的研究结果与模型为设计适用于物理世界的感知常规的智能代理提供了基础构建模块。
URL
https://arxiv.org/abs/2602.08914
PDF
-
Gesture Matters: Pedestrian Gesture Recognition for AVs Through Skeleton Pose Evaluation
2026-02-09 10:28:21Alif Rizqullah Mahdi, Mahdi Rezaei, Natasha MeratAbstract
Gestures are a key component of non-verbal communication in traffic, often helping pedestrian-to-driver interactions when formal traffic rules may be insufficient. This problem becomes more apparent when autonomous vehicles (AVs) struggle to interpret such gestures. In this study, we present a gesture classification framework using 2D pose estimation applied to real-world video sequences from the WIVW dataset. We categorise gestures into four primary classes (Stop, Go, Thank & Greet, and No Gesture) and extract 76 static and dynamic features from normalised keypoints. Our analysis demonstrates that hand position and movement velocity are especially discriminative in distinguishing between gesture classes, achieving a classification accuracy score of 87%. These findings not only improve the perceptual capabilities of AV systems but also contribute to the broader understanding of pedestrian behaviour in traffic contexts.
Abstract (translated)
手势是非言语交通交流中的关键组成部分，常常在正式交通规则不足时帮助行人与司机之间的互动。当自动驾驶汽车（AV）难以解读这些手势时，这一问题变得更加明显。在这项研究中，我们提出了一种基于二维姿态估计的框架，用于从WIVW数据集中提取的真实世界视频序列的手势分类。我们将手势分为四大类（停止、前进、感谢与问候、无手势），并从归一化的关键点中抽取76个静态和动态特征。我们的分析表明，手的位置和移动速度在区分不同手势类别时尤其具有鉴别力，达到了87%的分类准确率。这些发现不仅提升了自动驾驶系统的感知能力，还为理解交通环境中行人的行为提供了更广泛的认识。
URL
https://arxiv.org/abs/2602.08479
PDF
-
Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making
2026-02-07 19:25:02Ross Greer, Laura Fleig, Maitrayee Keskar, Erika Maquiling, Giovanni Tapia Lopez, Angel Martinez-Sanchez, Parthib Roy, Jake Rattigan, Mira Sur, Alejandra Vidrio, Thomas Marcotte, Mohan TrivediAbstract
The looking-in-looking-out (LILO) framework has enabled intelligent vehicle applications that understand both the outside scene and the driver state to improve safety outcomes, with examples in smart airbag deployment, takeover time prediction in autonomous control transitions, and driver attention monitoring. In this research, we propose an augmentation to this framework, making a case for the audio modality as an additional source of information to understand the driver, and in the evolving autonomy landscape, also the passengers and those outside the vehicle. We expand LILO by incorporating audio signals, forming the looking-and-listening inside-and-outside (L-LIO) framework to enhance driver state assessment and environment understanding through multimodal sensor fusion. We evaluate three example cases where audio enhances vehicle safety: supervised learning on driver speech audio to classify potential impairment states (e.g., intoxication), collection and analysis of passenger natural language instructions (e.g., "turn after that red building") to motivate how spoken language can interface with planning systems through audio-aligned instruction data, and limitations of vision-only systems where audio may disambiguate the guidance and gestures of external agents. Datasets include custom-collected in-vehicle and external audio samples in real-world environments. Pilot findings show that audio yields safety-relevant insights, particularly in nuanced or context-rich scenarios where sound is critical to safe decision-making or visual signals alone are insufficient. Challenges include ambient noise interference, privacy considerations, and robustness across human subjects, motivating further work on reliability in dynamic real-world contexts. L-LIO augments driver and scene understanding through multimodal fusion of audio and visual sensing, offering new paths for safety intervention.
Abstract (translated)
"观察与倾听框架（LILO）使得智能车辆应用能够在了解外部场景和驾驶员状态的基础上提高安全性能，例如智能气囊部署、自动驾驶控制转换时的接管时间预测以及驾驶员注意力监控。在此研究中，我们提议对此框架进行增强，并强调音频模态作为理解驾驶员及其他人员的重要信息来源，在不断发展的自主驾驶环境中尤其如此。我们将LILO扩展为结合声音信号的形式，形成“倾听与观察内部和外部”（L-LIO）框架，通过多模态传感器融合提高对驾驶员状态的评估及环境的理解能力。 我们在三个案例中评估了音频如何增强车辆安全性：使用监督学习分析司机语音音频来分类潜在的身体状况（如酒后驾驶），收集并分析乘客自然语言指令（例如，“在那个红房子之后转弯”）以展示口语如何通过与视觉信号对齐的指令数据与规划系统交互，以及仅依赖视觉系统的局限性，在这种情况下声音可以消除外部代理行为指导和手势中的歧义。这些案例的数据集包括从真实世界环境中定制收集的车内及车外音频样本。 初步研究结果表明，音频提供了重要的安全信息，尤其是在细微或背景丰富的场景中，其中声音对于做出安全决策至关重要或是视觉信号本身是不够的时候。然而，挑战还包括环境噪音干扰、隐私考虑以及在不同人类受试者间实现稳健性的难题，这需要我们在动态的真实世界环境中进一步增强系统的可靠性。 L-LIO框架通过多模态融合的听觉和视觉感应来提高驾驶员状态和场景的理解能力，并为安全干预提供了新的路径。"
URL
https://arxiv.org/abs/2602.07668
PDF
-
Bridging Speech, Emotion, and Motion: a VLM-based Multimodal Edge-deployable Framework for Humanoid Robots
2026-02-07 08:32:54Songhua Yang, Xuetao Li, Xuanye Fei, Mengde Li, Miao LiAbstract
Effective human-robot interaction requires emotionally rich multimodal expressions, yet most humanoid robots lack coordinated speech, facial expressions, and gestures. Meanwhile, real-world deployment demands on-device solutions that can operate autonomously without continuous cloud connectivity. To bridging \underline{\textit{S}}peech, \underline{\textit{E}}motion, and \underline{\textit{M}}otion, we present \textit{SeM$^2$}, a Vision Language Model-based framework that orchestrates emotionally coherent multimodal interactions through three key components: a multimodal perception module capturing user contextual cues, a Chain-of-Thought reasoning for response planning, and a novel Semantic-Sequence Aligning Mechanism (SSAM) that ensures precise temporal coordination between verbal content and physical expressions. We implement both cloud-based and \underline{\textit{e}}dge-deployed versions (\textit{SeM$^2_e$}), with the latter knowledge distilled to operate efficiently on edge hardware while maintaining 95\% of the relative performance. Comprehensive evaluations demonstrate that our approach significantly outperforms unimodal baselines in naturalness, emotional clarity, and modal coherence, advancing socially expressive humanoid robotics for diverse real-world environments.
Abstract (translated)
有效的类人机器人交互需要丰富的情感多模态表达，然而大多数的类人机器人在语音、面部表情和手势方面的协调能力不足。同时，实际部署要求设备能够在没有持续云连接的情况下自主运行。为了填补**S**peech（言语）、**E**motion（情感）和**M**otion（动作）之间的空白，我们提出了基于视觉语言模型框架的SeM$^2$，该框架通过三个关键组件协调情感一致性的多模态交互：一个多模态感知模块捕捉用户上下文线索、一个Chain-of-Thought推理用于响应规划以及一种新颖的语义序列对齐机制（SSAM），确保口头内容和物理表达在时间上精确同步。我们实现了基于云和**e**dge部署版本（SeM$^2_e$），后者通过知识蒸馏可以在边缘硬件上高效运行，同时保持95%以上的相对性能。全面的评估表明，我们的方法在自然性、情感清晰度以及模态一致性方面显著优于单模基线，在各种实际环境中推进了社会表达型类人机器人的发展。
URL
https://arxiv.org/abs/2602.07434
PDF
-
A novel Integrated Motion Tracking Device for Objective Laparoscopic Training Assessment: Development and Validation
2026-02-06 09:53:48Siwar Bouzid (COBRA), Abdelbadia Chaker (COBRA), Marc Arsicault (COBRA), Sami Bennour (COBRA, PPrime), Med Amine Laribi (COBRA, PPrime)Abstract
This paper presents a novel, compact four-degree-of-freedom motion-tracking device (IMTD) designed for training and evaluation in laparoscopic surgery. The device's kinematics, mechanical design, instrumentation, and prototypes are developed and presented to meet the specific requirements of laparoscopic training context, including movement around a fixed center of motion and seamless integration into standard box trainers. The system IMTD's tracking accuracy and reliability are compared to a motion capture system (MoCap), assessing its ability to capture both angular and translational motions of surgical instruments. The study then focuses on key performance parameters including precision, fluidity, speed, and overall motion efficiency. The results highlight the system's effectiveness in tracking surgical gestures, providing valuable insights into its potential as a tool for training and performance evaluation in minimally invasive surgery. Additionally, IMTD's low cost and integrated design allow for easy integration and implementation in training rooms, offering a practical and accessible solution for general use. By offering objective, real-time feedback, the system can significantly contribute to improving surgical skills and shortening the learning curve for novice students, while also providing a foundation for future development of gesture scoring algorithms and standardized training protocols.
Abstract (translated)
URL
https://arxiv.org/abs/2602.15885
PDF
-
MicroBi-ConvLSTM: An Ultra-Lightweight Efficient Model for Human Activity Recognition on Resource Constrained Devices
2026-02-06 09:26:29Mridankan MandalAbstract
Human Activity Recognition (HAR) on resource constrained wearables requires models that balance accuracy against strict memory and computational budgets. State of the art lightweight architectures such as TinierHAR (34K parameters) and TinyHAR (55K parameters) achieve strong accuracy, but exceed memory budgets of microcontrollers with limited SRAM once operating system overhead is considered. We present MicroBi-ConvLSTM, an ultra-lightweight convolutional-recurrent architecture achieving 11.4K parameters on average through two stage convolutional feature extraction with 4x temporal pooling and a single bidirectional LSTM layer. This represents 2.9x parameter reduction versus TinierHAR and 11.9x versus DeepConvLSTM while preserving linear O(N) complexity. Evaluation across eight diverse HAR benchmarks shows that MicroBi-ConvLSTM maintains competitive performance within the ultra-lightweight regime: 93.41% macro F1 on UCI-HAR, 94.46% on SKODA assembly gestures, and 88.98% on Daphnet gait freeze detection. Systematic ablation reveals task dependent component contributions where bidirectionality benefits episodic event detection, but provides marginal gains on periodic locomotion. INT8 post training quantization incurs only 0.21% average F1-score degradation, yielding a 23.0 KB average deployment footprint suitable for memory constrained edge devices.
Abstract (translated)
在资源有限的可穿戴设备上进行人体活动识别（HAR）需要能够平衡精度和严格内存及计算预算的模型。尽管像TinierHAR（含34K参数）和TinyHAR（含55K参数）这样的轻量级架构实现了强大的准确性，但一旦考虑了操作系统的开销后，这些模型还是超出了资源有限的微控制器的SRAM存储限制。 我们提出了一种名为MicroBi-ConvLSTM的超轻量化卷积递归架构，在两阶段卷积特征提取的基础上（其中时间池化比为4倍），配合一个单向双向LSTM层，平均参数量仅为11.4K。这相对于TinierHAR减少了2.9倍参数，并且相比DeepConvLSTM减少达11.9倍的参数数量，同时保持线性O(N)复杂度。 在八个不同的人体活动识别基准测试中，MicroBi-ConvLSTM展示了其在超轻量级模型下的竞争力：UCI-HAR基准上达到93.41%的宏F1分数，在SKODA装配手势检测上为94.46%，而Daphnet步态冻结检测则达到了88.98%。系统性消融实验揭示了任务依赖性的组件贡献，其中双向性在事件检测中表现出了明显的优势，但在周期性运动检测中的边际收益有限。 INT8后训练量化仅导致平均F1分数下降0.21%，但可以将模型的部署大小降至约23.0KB，这对于内存受限的边缘设备来说是理想的。
URL
https://arxiv.org/abs/2602.06523
PDF
-
PuppetAI: A Customizable Platform for Designing Tactile-Rich Affective Robot Interaction
2026-02-04 17:37:31Jiaye Li, Tongshun Chen, Siyi Ma, Elizabeth Churchill, Ke WuAbstract
We introduce PuppetAI, a modular soft robot interaction platform. This platform offers a scalable cable-driven actuation system and a customizable, puppet-inspired robot gesture framework, supporting a multitude of interaction gesture robot design formats. The platform comprises a four-layer decoupled software architecture that includes perceptual processing, affective modeling, motion scheduling, and low-level actuation. We also implemented an affective expression loop that connects human input to the robot platform by producing real-time emotional gestural responses to human vocal input. For our own designs, we have worked with nuanced gestures enacted by "soft robots" with enhanced dexterity and "pleasant-to-touch" plush exteriors. By reducing operational complexity and production costs while enhancing customizability, our work creates an adaptable and accessible foundation for future tactile-based expressive robot research. Our goal is to provide a platform that allows researchers to independently construct or refine highly specific gestures and movements performed by social robots.
Abstract (translated)
我们介绍PuppetAI，这是一种模块化软机器人交互平台。该平台提供了一种可扩展的电缆驱动执行系统和一个受提线木偶启发、可根据需求定制化的机器人手势框架，支持多种互动手势机器人的设计格式。平台采用了四层解耦软件架构，包括感知处理、情感建模、动作调度以及低级执行等模块。 我们还实现了一个情感表达循环，通过生成实时的情感手势反应来将人类的声音输入与机器人平台相连结，从而使得该平台能够根据人类的语音输入产生即时反馈。在我们的设计中，我们利用了由柔软材料制成且具有增强灵活性和触感舒适的“软机器人”，执行细腻的手势动作。 通过降低操作复杂性和生产成本并提高定制化能力，我们的工作为未来基于触觉的情感表达机器人的研究建立了一个灵活易用的基础平台。我们的目标是提供一个平台，允许研究人员独立构建或改进由社交机器人执行的高度特定手势和运动。
URL
https://arxiv.org/abs/2602.04787
PDF
-
Time Is All It Takes: Spike-Retiming Attacks on Event-Driven Spiking Neural Networks
2026-02-03 09:06:53Yi Yu, Qixin Zhang, Shuhan Ye, Xun Lin, Qianshan Wei, Kun Wang, Wenhan Yang, Dacheng Tao, Xudong JiangAbstract
Spiking neural networks (SNNs) compute with discrete spikes and exploit temporal structure, yet most adversarial attacks change intensities or event counts instead of timing. We study a timing-only adversary that retimes existing spikes while preserving spike counts and amplitudes in event-driven SNNs, thus remaining rate-preserving. We formalize a capacity-1 spike-retiming threat model with a unified trio of budgets: per-spike jitter $\mathcal{B}_{\infty}$, total delay $\mathcal{B}_{1}$, and tamper count $\mathcal{B}_{0}$. Feasible adversarial examples must satisfy timeline consistency and non-overlap, which makes the search space discrete and constrained. To optimize such retimings at scale, we use projected-in-the-loop (PIL) optimization: shift-probability logits yield a differentiable soft retiming for backpropagation, and a strict projection in the forward pass produces a feasible discrete schedule that satisfies capacity-1, non-overlap, and the chosen budget at every step. The objective maximizes task loss on the projected input and adds a capacity regularizer together with budget-aware penalties, which stabilizes gradients and aligns optimization with evaluation. Across event-driven benchmarks (CIFAR10-DVS, DVS-Gesture, N-MNIST) and diverse SNN architectures, we evaluate under binary and integer event grids and a range of retiming budgets, and also test models trained with timing-aware adversarial training designed to counter timing-only attacks. For example, on DVS-Gesture the attack attains high success (over $90\%$) while touching fewer than $2\%$ of spikes under $\mathcal{B}_{0}$. Taken together, our results show that spike retiming is a practical and stealthy attack surface that current defenses struggle to counter, providing a clear reference for temporal robustness in event-driven SNNs. Code is available at this https URL.
Abstract (translated)
脉冲神经网络（SNNs）利用离散的脉冲和时间结构进行计算，然而大多数对抗性攻击会改变强度或事件计数而不是时间。本文研究了一种仅基于时间调整现有脉冲而不保留其时间和幅度的时间型对手，在事件驱动的SNN中，这种对手保持了速率不变。我们形式化了一个容量为1的脉冲重定时威胁模型，并使用统一的预算组合：每个脉冲抖动$\mathcal{B}_{\infty}$、总延迟$\mathcal{B}_{1}$和篡改计数$\mathcal{B}_{0}$。可行的对抗性示例必须满足时间线一致性与非重叠条件，这使得搜索空间既离散又受约束。为优化大规模脉冲重定时，我们采用了投影内循环（PIL）优化方法：通过转移概率对数获得可微软重定时以进行反向传播，并在正向传递中使用严格的投影生成满足容量1、非重叠以及选定预算条件的可行离散时间表。目标是最大化在投影输入上的任务损失，同时添加了容量正则化器和预算感知惩罚项，这有助于稳定梯度并使优化与评估保持一致。跨事件驱动基准（CIFAR10-DVS、DVS-Gesture、N-MNIST）及多种SNN架构，在二进制和整数时间网格以及广泛的重定时预算下进行评测，并测试了针对时间型攻击训练的模型。例如，在DVS-Gesture上，该攻击在$\mathcal{B}_{0}$条件下以少于2%的脉冲接触量实现了超过90%的成功率。总体而言，我们的结果显示脉冲重定时是一种实用且隐蔽的攻击面，现有的防御手段难以应对，并为事件驱动SNN的时间鲁棒性提供了一个清晰参考。代码可在该链接处获得。
URL
https://arxiv.org/abs/2602.03284
PDF
