Gesture
-
An AI-IoT Based Smart Wheelchair with Gesture-Controlled Mobility, Deep Learning-Based Obstacle Detection, Multi-Sensor Health Monitoring, and Emergency Alert System
2026-01-17 09:37:11Md. Asiful Islam, Abdul Hasib, Tousif Mahmud Emon, Khandaker Tabin Hasan, A. S. M. Ahsanul Sarkar AkibAbstract
The growing number of differently-abled and elderly individuals demands affordable, intelligent wheelchairs that combine safe navigation with health monitoring. Traditional wheelchairs lack dynamic features, and many smart alternatives remain costly, single-modality, and limited in health integration. Motivated by the pressing demand for advanced, personalized, and affordable assistive technologies, we propose a comprehensive AI-IoT based smart wheelchair system that incorporates glove-based gesture control for hands-free navigation, real-time object detection using YOLOv8 with auditory feedback for obstacle avoidance, and ultrasonic for immediate collision avoidance. Vital signs (heart rate, SpO$_2$, ECG, temperature) are continuously monitored, uploaded to ThingSpeak, and trigger email alerts for critical conditions. Built on a modular and low-cost architecture, the gesture control achieved a 95.5\% success rate, ultrasonic obstacle detection reached 94\% accuracy, and YOLOv8-based object detection delivered 91.5\% Precision, 90.2\% Recall, and a 90.8\% F1-score. This integrated, multi-modal approach offers a practical, scalable, and affordable solution, significantly enhancing user autonomy, safety, and independence by bridging the gap between innovative research and real-world deployment.
Abstract (translated)
越来越多的残障人士和老年人需要一种既经济又智能的轮椅，这种轮椅能够结合安全导航与健康监测。传统轮椅缺乏动态特性，而许多智能化替代品依然昂贵，并且大多局限于单一功能并且在健康整合方面有限。 鉴于对先进的、个性化的以及可负担得起的辅助技术的迫切需求，我们提出了一种基于AI-IoT（人工智能和物联网）的综合智能轮椅系统。该系统包括用于手控自由导航的手套控制装置，使用YOLOv8进行实时物体检测，并结合听觉反馈来避开障碍物；同时采用超声波传感器以实现即时碰撞避免功能。此外，该系统还能够连续监测心率、血氧饱和度（SpO2）、心电图（ECG）和体温等生命体征信息，并将这些数据上传至ThingSpeak平台，在出现紧急情况时触发电子邮件警报。 我们的设计基于模块化且成本低廉的架构：手部控制装置达到了95.5%的成功率，超声波障碍物检测准确率达到94%，而基于YOLOv8的目标检测则分别实现了91.5% 的精确度、90.2% 的召回率和90.8% 的F1得分。 这种集成的多模式方法提供了一种实用且可扩展的解决方案，以合理的价格显著提升了用户的自主性、安全性和独立性，从而在创新研究与实际应用之间架起了桥梁。
URL
https://arxiv.org/abs/2601.11983
PDF
-
Telling Human and Machine Handwriting Apart
2026-01-16 18:45:16Luis A. Leiva, Moises Diaz, Nuwan T. Attygalle, Miguel A. Ferrer, Rejean PlamondonAbstract
Handwriting movements can be leveraged as a unique form of behavioral biometrics, to verify whether a real user is operating a device or application. This task can be framed as a reverse Turing test in which a computer has to detect if an input instance has been generated by a human or artificially. To tackle this task, we study ten public datasets of handwritten symbols (isolated characters, digits, gestures, pointing traces, and signatures) that are artificially reproduced using seven different synthesizers, including, among others, the Kinematic Theory (Sigma h model), generative adversarial networks, Transformers, and Diffusion models. We train a shallow recurrent neural network that achieves excellent performance (98.3 percent Area Under the ROC Curve (AUC) score and 1.4 percent equal error rate on average across all synthesizers and datasets) using nonfeaturized trajectory data as input. In few-shot settings, we show that our classifier achieves such an excellent performance when trained on just 10 percent of the data, as evaluated on the remaining 90% of the data as a test set. We further challenge our classifier in out-of-domain settings, and observe very competitive results as well. Our work has implications for computerized systems that need to verify human presence, and adds an additional layer of security to keep attackers at bay.
Abstract (translated)
手写动作可以被用作一种独特的行为生物识别方式，以验证实际用户是否在操作设备或应用程序。这一任务可以被视为一个反向图灵测试，在这种测试中，计算机需要检测输入实例是由真人生成的还是人工合成的。为了解决这个问题，我们研究了十个公开的手写符号数据集（包括孤立字符、数字、手势、指点轨迹和签名），这些数据集使用七种不同的生成器进行了人工再现，其中包括动学理论（Sigma h 模型）、生成对抗网络、Transformer 和扩散模型等。 我们训练了一个浅层循环神经网络，在不进行特征化处理的情况下以轨迹数据为输入，该网络在所有合成器和数据集中实现了卓越的性能（98.3% 的接收者操作特性曲线（ROC）面积得分和1.4% 的平均错误接受率）。此外，我们在少量样本设置下展示了当训练集仅为总数据量的10%，而剩余90%的数据作为测试集时，我们的分类器也能实现如此卓越的表现。我们进一步在跨域场景中挑战了这一分类器，并观察到了非常具有竞争力的结果。 这项工作对需要验证人类存在的计算机系统有着重要的影响，并为防止攻击者侵入添加了一层额外的安全保障。
URL
https://arxiv.org/abs/2601.11700
PDF
-
TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge
2026-01-15 18:05:25Matteo Fasulo, Giusy Spacone, Thorir Mar Ingolfsson, Yawei Li, Luca Benini, Andrea CossettiniAbstract
Objective: Surface electromyography (EMG) is a non-invasive sensing modality widely used in biomechanics, rehabilitation, prosthetic control, and human-machine interfaces. Despite decades of use, achieving robust generalization across subjects, recording systems, and acquisition protocols remains challenging. While foundation models (FMs) are gaining traction for EMG, existing approaches remain limited to single downstream tasks and lack deployability on embedded platforms. This work addresses these limitations. Methods: We present TinyMyo, a lightweight FM based on a Transformer encoder architecture. The model is pre-trained in a self-supervised manner using masked reconstruction on publicly available datasets. With only 3.6M parameters, TinyMyo is designed to support multiple downstream tasks through minimal task-specific head adaptations. Results: We demonstrate generalization across hand gesture classification, hand kinematic regression, speech production and speech recognition, with performance comparable to or surpassing the state of the art (SoA), and model size below 5M parameters. We achieve SoA results compared to previous FM-based works on the NinaPro DB5 (89.4%), UCI-EMG (97.56%), and EPN-612 (96.74%) datasets. We demonstrate the first-time deployment of an EMG FM on an ultra-low power microcontroller (GAP9), with an inference time of 0.785 s, energy of 44.91 mJ and power envelope of 57.18 mW. Conclusion: TinyMyo demonstrates that compact, self-supervised EMG FM can guarantee strong generalization across multiple downstream tasks while remaining compatible with low-power edge devices. Significance: TinyMyo is the first EMG FM for ultra-low power edge devices, enabling scalable and energy-efficient sensing for motor intent decoding, neuromuscular assessment, and biosignal driven human-machine interaction.
Abstract (translated)
目标：表面肌电图（EMG）是一种广泛应用于生物力学、康复、假肢控制和人机界面的非侵入式传感模式。尽管经过几十年的应用，实现跨受试者、记录系统及采集协议的良好泛化仍然具有挑战性。虽然基础模型（FMs）在EMG应用中正日益受到重视，但现有方法依然局限于单一下游任务，并且无法部署于嵌入式平台上。这项工作旨在解决这些限制。 方法：我们提出了TinyMyo，一种基于Transformer编码器架构的轻量级基础模型。该模型通过公开可用的数据集上自监督的方式进行预训练，采用掩码重构方式。仅包含3.6M参数的小型模型被设计为能够支持多种下游任务，并且只需要对特定任务头部进行微小调整即可。 结果：我们展示了TinyMyo在手部手势分类、手部运动学回归、语音生成和识别等领域的泛化能力，其性能与现有最佳方法相当或更优，同时模型规模低于5M参数。在NinaPro DB5（89.4%）、UCI-EMG（97.56%）以及EPN-612（96.74%）数据集上相比之前的基于基础模型的工作，我们的结果达到了现有最佳水平。此外，我们首次将一个EMG基础模型部署到了超低功耗微控制器（GAP9）上，在推理时间仅为0.785秒、能耗为44.91 mJ和功率上限为57.18 mW的情况下实现了这一目标。 结论：TinyMyo表明，紧凑的自监督EMG基础模型能够在多种下游任务中保证强大的泛化能力，并且兼容低功耗边缘设备。意义：TinyMyo是首个应用于超低功耗边缘设备的EMG基础模型，这使得运动意图解码、神经肌肉评估及生物信号驱动的人机交互能够实现可扩展和节能的目标感测技术。
URL
https://arxiv.org/abs/2512.15729
PDF
-
When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions
2026-01-13 15:21:08Mingyu Zhu, Jiangong Chen, Bin LiAbstract
Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.
Abstract (translated)
扩展现实（XR），包括虚拟现实（VR）、增强现实（AR）和混合现实（MR），为各种应用场景提供了沉浸式和互动体验，从基于 VR 的教育到基于 AR 的辅助功能以及基于 MR 的培训。然而，由于两个关键挑战，广泛采用 XR 仍然受到限制：1) 创作三维内容的高成本和复杂性，特别是对于大规模环境或复杂的交互；2) 手持控制器或脚本化手势等非直观互动方法的学习曲线陡峭。 生成式人工智能（GenAI）通过支持直觉化的语言驱动互动以及自动内容生成提供了一种有前景的解决方案。利用视觉-语言模型和扩散生成技术，GenAI 可以解读模糊指令、理解物理场景，并生成或操作三维内容，显著降低 XR 采用的门槛。本文探讨了通过三个具体用例将 XR 和 GenAI 集成的方法，展示了它们如何解决可扩展性和自然互动的关键障碍，并确定了必须解决的技术挑战，以便实现更广泛的采用。
URL
https://arxiv.org/abs/2601.15308
PDF
-
Teaching Robots Like Dogs: Learning Agile Navigation from Luring, Gesture, and Speech
2026-01-13 10:47:12Taerim Yoon, Dongho Kang, Jin Cheng, Fatemeh Zargarbashi, Yijiang Huang, Minsung Ahn, Stelian Coros, Sungjoon ChoiAbstract
In this work, we aim to enable legged robots to learn how to interpret human social cues and produce appropriate behaviors through physical human guidance. However, learning through physical engagement can place a heavy burden on users when the process requires large amounts of human-provided data. To address this, we propose a human-in-the-loop framework that enables robots to acquire navigational behaviors in a data-efficient manner and to be controlled via multimodal natural human inputs, specifically gestural and verbal commands. We reconstruct interaction scenes using a physics-based simulation and aggregate data to mitigate distributional shifts arising from limited demonstration data. Our progressive goal cueing strategy adaptively feeds appropriate commands and navigation goals during training, leading to more accurate navigation and stronger alignment between human input and robot behavior. We evaluate our framework across six real-world agile navigation scenarios, including jumping over or avoiding obstacles. Our experimental results show that our proposed method succeeds in almost all trials across these scenarios, achieving a 97.15% task success rate with less than 1 hour of demonstration data in total.
Abstract (translated)
在这项工作中，我们旨在使腿部机器人通过物理人类指导学会解读人类社交线索并产生适当的行为。然而，通过身体互动进行学习可能会给用户带来沉重的负担，特别是当过程需要大量的人类提供的数据时。为了解决这个问题，我们提出了一种人机交互框架，使得机器人能够在数据高效的方式下获取导航行为，并能够接受多模态自然人类输入（具体来说是手势和口头命令）。我们使用基于物理的模拟重建互动场景，并汇总数据以缓解由于演示数据有限而产生的分布变化。我们的渐进式目标提示策略在训练过程中适应性地提供适当的指令和导航目标，从而导致更准确的导航以及人机交互与机器人行为之间更强的一致性。 我们在六个现实世界中的敏捷导航场景中评估了该框架，包括跳跃或避开障碍物的情况。实验结果显示，在这些场景下，我们的方法几乎都在所有试验中取得成功，并且在总演示数据不到1小时的情况下达到了97.15%的任务成功率。
URL
https://arxiv.org/abs/2601.08422
PDF
-
VGG Induced Deep Hand Sign Language Detection
2026-01-13 06:39:29Subham Sharma, Sharmila SubudhiAbstract
Hand gesture recognition is an important aspect of human-computer interaction. It forms the basis of sign language for the visually impaired people. This work proposes a novel hand gesture recognizing system for the differently-abled persons. The model uses a convolutional neural network, known as VGG-16 net, for building a trained model on a widely used image dataset by employing Python and Keras libraries. Furthermore, the result is validated by the NUS dataset, consisting of 10 classes of hand gestures, fed to the model as the validation set. Afterwards, a testing dataset of 10 classes is built by employing Google's open source Application Programming Interface (API) that captures different gestures of human hand and the efficacy is then measured by carrying out experiments. The experimental results show that by combining a transfer learning mechanism together with the image data augmentation, the VGG-16 net produced around 98% accuracy.
Abstract (translated)
手部手势识别是人机交互的重要组成部分，它是视障人士使用的手语的基础。本研究提出了一种针对残障人士的手势识别系统。该模型采用卷积神经网络（称为VGG-16网络），利用Python和Keras库在广泛使用的一组图像数据上建立训练模型。此外，结果通过新加坡国立大学（NUS）数据集进行验证，该数据集中包含10类手部手势，并将其作为模型的验证集输入。随后，构建了一个包含10个类别测试数据的数据集，利用Google开源的应用编程接口（API），捕捉人类不同手势动作的有效性并执行实验以测量其效能。实验结果显示，通过结合迁移学习机制和图像数据增强技术，VGG-16网络达到了约98%的准确率。
URL
https://arxiv.org/abs/2601.08262
PDF
-
THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control
2026-01-12 17:50:02Alex Huang, Akshay KarthikAbstract
The teleoperation of robotic hands is limited by the high costs of depth cameras and sensor gloves, commonly used to estimate hand relative joint positions (XYZ). We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers. We also introduce a modified DexHand, a low-cost robotic hand from TheRobotStudio, to demonstrate THETA's real-time application. Data collection involved 40 distinct hand gestures using three 640x480p webcams arranged at 120-degree intervals, generating over 48,000 RGB images. Joint angles were manually determined by measuring midpoints of the MCP, PIP, and DIP finger joints. Captured RGB frames were processed using a DeepLabV3 segmentation model with a ResNet-50 backbone for multi-scale hand segmentation. The segmented images were then HSV-filtered and fed into THETA's architecture, consisting of a MobileNetV2-based CNN classifier optimized for hierarchical spatial feature extraction and a 9-channel input tensor encoding multi-perspective hand representations. The classification model maps segmented hand views into discrete joint angles, achieving 97.18% accuracy, 98.72% recall, F1 Score of 0.9274, and a precision of 0.8906. In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification. Joint-angle predictions are relayed via serial to an Arduino, enabling the DexHand to replicate hand movements. Future research will increase dataset diversity, integrate wrist tracking, and apply computer vision techniques such as OpenAI-Vision. THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.
Abstract (translated)
机器人手的遥操作受到深度相机和传感器手套高昂成本的限制，这些设备通常用于估算相对关节位置（XYZ）。我们提出了一种新颖且成本效益高的方法，使用三台网络摄像头进行三角测量跟踪，以近似人类手指的相对关节角度（θ）。此外，还引入了经过修改的DexHand，这是一种来自TheRobotStudio的低成本机器人手，用以展示THETA在实时应用中的效果。数据收集涉及40种不同的手势，通过三个间隔120度排列的640x480p网络摄像头捕捉到超过48,000张RGB图像。关节角度由测量掌指（MCP）、近侧指间（PIP）和远侧指间（DIP）关节中点的手动确定得出。 收集的RGB帧使用带有ResNet-50骨干的DeepLabV3分割模型进行多尺度手部分割处理。随后，对分段图像应用HSV滤波，并将其输入到THETA架构中，该架构由基于MobileNetV2的CNN分类器和编码多种视角的手部表示的9通道张量组成，用于优化层级空间特征提取。分类模型将分割后的手部视图映射为离散关节角度，在准确性、召回率、F1分数以及精度方面分别达到了97.18%、98.72%、0.9274和0.8906。 在实时推断过程中，THETA同时捕捉图像帧，分割手部区域，并过滤这些区域以生成用于分类的9通道张量。关节角度预测通过串行通信发送到Arduino，使DexHand能够复制人类的手部动作。未来的研究将进一步丰富数据集多样性、整合手腕跟踪以及应用计算机视觉技术如OpenAI-Vision。 THETA有望确保成本效益高且用户友好的远程操作，在医疗、语言和制造应用中具有潜在价值。
URL
https://arxiv.org/abs/2601.07768
PDF
-
Compensation Effect Amplification Control : A movement-based approach for coordinated position and velocity control of the elbow of upper-limb prostheses
Abstract
Despite advances in upper-limb (UL) prosthetic design, achieving intuitive control of intermediate joints - such as the wrist and elbow - remains challenging, particularly for continuous and velocity-modulated movements. We introduce a novel movement-based control paradigm entitled Compensation Effect Amplification Control (CEAC) that leverages users' trunk flexion and extension as input for controlling prosthetic elbow velocity. Considering that the trunk can be both a functional and compensatory joint when performing upper-limb actions, CEAC amplifies the natural coupling between trunk and prosthesis while introducing a controlled delay that allows users to modulate both the position and velocity of the prosthetic joint. We evaluated CEAC in a generic drawing task performed by twelve able-bodied participants using a supernumerary prosthesis with an active elbow. Additionally a multiple-target-reaching task was performed by a subset of ten participants. Results demonstrate task performances comparable to those obtained with natural arm movements, even when gesture velocity or drawing size were varied, while maintaining ergonomic trunk postures. Analysis revealed that CEAC effectively restores joint coordinated action, distributes movement effort between trunk and elbow, enabling intuitive trajectory control without requiring extreme compensatory movements. Overall, CEAC offers a promising control strategy for intermediate joints of UL prostheses, particularly in tasks requiring continuous and precise coordination.
Abstract (translated)
尽管上肢（UL）假体设计取得了进展，但实现中间关节（如手腕和肘部）的直观控制仍然颇具挑战性，尤其是在连续运动和速度调节方面。我们引入了一种新的基于动作的控制范式——补偿效应放大控制（CEAC），该范式利用用户的躯干屈曲和伸展作为控制假肢肘部速度的输入。考虑到在执行上肢动作时，躯干可以是一个功能关节或代偿关节，CEAC 放大了躯干与假肢之间的自然耦合，并引入了一个可控延迟，使用户能够调节假肢关节的位置和速度。我们使用一个具有活动肘部的超数假肢，让十二名健全参与者在通用绘图任务中评估了 CEAC 的效果；此外，十个参与者的子集还执行了多目标到达任务。 结果表明，即使是在变化手势速度或绘制大小的情况下，CEAC 仍能实现与自然手臂运动相当的任务表现，并且保持了符合人体工程学的躯干姿势。分析显示，CEAC 有效地恢复了关节协调动作，在躯干和肘部之间分配了运动努力，使得用户可以直观地控制轨迹而无需进行极端代偿性动作。 总的来说，CEAC 为 UL 假肢中间关节提供了一种有前景的控制策略，尤其是在需要连续且精确协调的任务中。
URL
https://arxiv.org/abs/2601.05074
PDF
-
FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusion
2026-01-07 14:18:59Enes Duran, Nikos Athanasiou, Muhammed Kocabas, Michael J. Black, Omid TaheriAbstract
Hands are central to interacting with our surroundings and conveying gestures, making their inclusion essential for full-body motion synthesis. Despite this, existing human motion synthesis methods fall short: some ignore hand motions entirely, while others generate full-body motions only for narrowly scoped tasks under highly constrained settings. A key obstacle is the lack of large-scale datasets that jointly capture diverse full-body motion with detailed hand articulation. While some datasets capture both, they are limited in scale and diversity. Conversely, large-scale datasets typically focus either on body motion without hands or on hand motions without the body. To overcome this, we curate and unify existing hand motion datasets with large-scale body motion data to generate full-body sequences that capture both hand and body. We then propose the first diffusion-based unconditional full-body motion prior, FUSION, which jointly models body and hand motion. Despite using a pose-based motion representation, FUSION surpasses state-of-the-art skeletal control models on the Keypoint Tracking task in the HumanML3D dataset and achieves superior motion naturalness. Beyond standard benchmarks, we demonstrate that FUSION can go beyond typical uses of motion priors through two applications: (1) generating detailed full-body motion including fingers during interaction given the motion of an object, and (2) generating Self-Interaction motions using an LLM to transform natural language cues into actionable motion constraints. For these applications, we develop an optimization pipeline that refines the latent space of our diffusion model to generate task-specific motions. Experiments on these tasks highlight precise control over hand motion while maintaining plausible full-body coordination. The code will be public.
Abstract (translated)
手部在与周围环境互动以及传达手势方面起着关键作用，因此在全身动作合成中包括手部运动是必不可少的。然而，现有的人体动作合成方法存在不足：一些方法完全忽略了手部动作，而另一些则仅能在高度受限的情况下生成窄范围任务的全身体动。一个主要障碍在于缺乏大规模的数据集，这些数据集能够同时捕捉多样化的全身动作和详细的手部关节运动。虽然有些数据集能捕获两者，但它们在规模和多样性方面存在限制。相比之下，大型数据集通常要么专注于没有手的身体运动，要么专注于没有身体的手部运动。 为了解决这个问题，我们整合并统一现有的手部运动数据集与大规模身体运动数据集，以生成同时捕捉手部和身体动作的全身序列。随后，我们提出了首个基于扩散模型的无条件全身体动先验FUSION（Full-body UNconditional motion SynthesIzation），该模型能够联合建模身体和手部的动作。尽管使用的是基于姿势的动作表示方式，但FUSION在HumanML3D数据集的关键点跟踪任务上超越了最新的骨骼控制模型，并实现了更自然的运动效果。 除了标准基准测试外，我们通过两个应用展示了FUSION可以超越传统动作用途：（1）根据物体的移动生成详细的全身动作，包括手指互动；（2）利用LLM将自然语言提示转化为可操作的动作约束来生成自我交互动作。为了这些应用程序，我们开发了一个优化管道，该管道能够细化我们的扩散模型的潜在空间以生成特定任务的运动。在这些任务上的实验突显了对手部动作的精确控制同时保持合理的全身协调性。代码将公开提供。
URL
https://arxiv.org/abs/2601.03959
PDF
-
EASLT: Emotion-Aware Sign Language Translation
Abstract
Sign Language Translation (SLT) is a complex cross-modal task requiring the integration of Manual Signals (MS) and Non-Manual Signals (NMS). While recent gloss-free SLT methods have made strides in translating manual gestures, they frequently overlook the semantic criticality of facial expressions, resulting in ambiguity when distinct concepts share identical manual articulations. To address this, we present **EASLT** (**E**motion-**A**ware **S**ign **L**anguage **T**ranslation), a framework that treats facial affect not as auxiliary information, but as a robust semantic anchor. Unlike methods that relegate facial expressions to a secondary role, EASLT incorporates a dedicated emotional encoder to capture continuous affective dynamics. These representations are integrated via a novel *Emotion-Aware Fusion* (EAF) module, which adaptively recalibrates spatio-temporal sign features based on affective context to resolve semantic ambiguities. Extensive evaluations on the PHOENIX14T and CSL-Daily benchmarks demonstrate that EASLT establishes advanced performance among gloss-free methods, achieving BLEU-4 scores of 26.15 and 22.80, and BLEURT scores of 61.0 and 57.8, respectively. Ablation studies confirm that explicitly modeling emotion effectively decouples affective semantics from manual dynamics, significantly enhancing translation fidelity. Code is available at this https URL.
Abstract (translated)
手语翻译（SLT）是一项复杂的跨模态任务，需要整合手势信号（MS）和非手动信号（NMS）。尽管最近的无助词符SLT方法在翻译手动动作方面取得了进展，但它们常常忽视面部表情所包含的重要语义信息，在不同概念的手势相同的情况下导致理解上的模糊性。为解决这一问题，我们提出了**EASLT** (**E**motion-**A**ware **S**ign **L**anguage **T**ranslation)，这是一个将面部情感视为重要语义锚点而非辅助信息的框架。与那些将面部表情置于次要地位的方法不同，EASLT引入了一个专用的情感编码器来捕捉连续的情感动态。通过一种新颖的*情感感知融合*（EAF）模块，这些表示被整合在一起，该模块根据情感背景自适应地重新校准空间-时间手势特征以解决语义模糊问题。 在PHOENIX14T和CSL-Daily基准测试上进行的广泛评估表明，EASLT在无助词符的方法中表现出色，在这两个数据集上分别实现了BLEU-4得分为26.15和22.80，以及BLEURT得分分别为61.0和57.8。消融实验证实了显式建模情感能够有效分离情感语义与手动动态，并显著提升翻译的准确性。 代码可在以下链接获取：[提供链接处]（请将[提供链接处]替换为实际提供的代码链接）。
URL
https://arxiv.org/abs/2601.03549
PDF
-
Millimeter-Wave Gesture Recognition in ISAC: Does Reducing Sensing Airtime Hamper Accuracy?
2026-01-06 12:42:18Jakob Struye, Nabeel Nisar Bhat, Siddhartha Kumar, Mohammad Hossein Moghaddam, Jeroen FamaeyAbstract
Most Integrated Sensing and Communications (ISAC) systems require dividing airtime across their two modes. However, the specific impact of this decision on sensing performance remains unclear and underexplored. In this paper, we therefore investigate the impact on a gesture recognition system using a Millimeter-Wave (mmWave) ISAC system. With our dataset of power per beam pair gathered with two mmWave devices performing constant beam sweeps while test subjects performed distinct gestures, we train a gesture classifier using Convolutional Neural Networks. We then subsample these measurements, emulating reduced sensing airtime, showing that a sensing airtime of 25 % only reduces classification accuracy by 0.15 percentage points from full-time sensing. Alongside this high-quality sensing at low airtime, mmWave systems are known to provide extremely high data throughputs, making mmWave ISAC a prime enabler for applications such as truly wireless Extended Reality.
Abstract (translated)
大多数集成传感和通信（ISAC）系统需要在其两种模式之间分配空中时间。然而，这一决策对感知性能的具体影响尚不清楚且未被充分研究。因此，在本文中，我们调查了手势识别系统在使用毫米波（mmWave）ISAC系统时所受到的影响。通过收集两台进行连续波束扫描的mmWave设备的数据集，并记录测试对象执行不同手势期间每个波束对的能量值，我们利用卷积神经网络训练了一个手势分类器。然后，通过对这些测量数据进行子采样，模拟了减少感知空中时间的情况，结果显示，当感知空中时间为25%时，仅使全时段感知下的分类准确率下降0.15个百分点。此外，在这种低空中时间内仍能实现高质量的感知，mmWave系统还因其极高的数据吞吐量而著称，这使得毫米波ISAC成为支持如真正无线扩展现实等应用的理想选择。
URL
https://arxiv.org/abs/2601.10733
PDF
-
Advancing Assistive Robotics: Multi-Modal Navigation and Biophysical Monitoring for Next-Generation Wheelchairs
2026-01-06 06:58:26Md. Anowar Hossain, Mohd. Ehsanul HoqueAbstract
Assistive electric-powered wheelchairs (EPWs) have become essential mobility aids for people with disabilities such as amyotrophic lateral sclerosis (ALS), post-stroke hemiplegia, and dementia-related mobility impairment. This work presents a novel multi-modal EPW control system designed to prioritize patient needs while allowing seamless switching between control modes. Four complementary interfaces, namely joystick, speech, hand gesture, and electrooculography (EOG), are integrated with a continuous vital sign monitoring framework measuring heart rate variability, oxygen saturation (SpO2), and skin temperature. This combination enables greater patient independence while allowing caregivers to maintain real-time supervision and early intervention capability. Two-point calibration of the biophysical sensors against clinical reference devices resulted in root mean square errors of at most 2 bpm for heart rate, 0.5 degree Celsius for skin temperature, and 1 percent for SpO2. Experimental evaluation involved twenty participants with mobility impairments executing a total of 500 indoor navigation commands. The achieved command recognition accuracies were 99 percent for joystick control, 97 percent plus or minus 2 percent for speech, and 95 percent plus or minus 3 percent for hand gesture, with an average closed-loop latency of 20 plus or minus 0.5 milliseconds. Caregivers receive real-time alerts through an Android application following encrypted cloud transmission of physiological data. By integrating multi-modal mobility control with cloud-enabled health monitoring and reporting latency and energy budgets, the proposed prototype addresses key challenges in assistive robotics, contributes toward compliance with ISO 7176-31 and IEC 80601-2-78 safety standards, and establishes a foundation for future adaptive machine learning enhancements.
Abstract (translated)
助行电动轮椅（EPW）已成为肌萎缩侧索硬化症（ALS）、中风后偏瘫和痴呆相关移动障碍等残疾人士的重要移动辅助工具。本文介绍了一种新型的多模式EPW控制系统，该系统旨在优先考虑患者需求并允许在控制模式之间无缝切换。四种互补接口——即操纵杆、语音、手势和眼电图（EOG）被整合进一个连续的生命体征监测框架中，此框架测量心率变异性、血氧饱和度（SpO2）以及皮肤温度。这种组合提高了患者的独立性，同时让护理人员能够进行实时监督并及时采取干预措施。通过与临床参考设备进行两点校准后，生物物理传感器的心率最大误差为每分钟2次搏动，皮肤温度的误差为0.5摄氏度，血氧饱和度的最大误差为1%。 实验评估涉及20名具有移动障碍的参与者执行总共500个室内导航命令。实现的命令识别准确性为：操纵杆控制99%，语音控制97%±2%，手势控制95%±3%，平均闭环延迟时间为20±0.5毫秒。护理人员通过一个安卓应用程序接收实时警报，生理数据则经过加密后上传至云端。 该原型将多模式移动控制与基于云的健康监测相结合，并考虑了报告延迟和能耗预算，从而解决了辅助机器人领域的关键挑战，有助于符合ISO 7176-31和IEC 80601-2-78安全标准的要求。此外，它还为未来的自适应机器学习增强奠定了基础。
URL
https://arxiv.org/abs/2601.02766
PDF
-
SmoothSync: Dual-Stream Diffusion Transformers for Jitter-Robust Beat-Synchronized Gesture Generation from Quantized Audio
2026-01-04 09:53:07Yujiao Jiang, Qingmin Liao, Zongqing LuAbstract
Co-speech gesture generation is a critical area of research aimed at synthesizing speech-synchronized human-like gestures. Existing methods often suffer from issues such as rhythmic inconsistency, motion jitter, foot sliding and limited multi-sampling diversity. In this paper, we present SmoothSync, a novel framework that leverages quantized audio tokens in a novel dual-stream Diffusion Transformer (DiT) architecture to synthesis holistic gestures and enhance sampling variation. Specifically, we (1) fuse audio-motion features via complementary transformer streams to achieve superior synchronization, (2) introduce a jitter-suppression loss to improve temporal smoothness, (3) implement probabilistic audio quantization to generate distinct gesture sequences from identical inputs. To reliably evaluate beat synchronization under jitter, we introduce Smooth-BC, a robust variant of the beat consistency metric less sensitive to motion noise. Comprehensive experiments on the BEAT2 and SHOW datasets demonstrate SmoothSync's superiority, outperforming state-of-the-art methods by -30.6% FGD, 10.3% Smooth-BC, and 8.4% Diversity on BEAT2, while reducing jitter and foot sliding by -62.9% and -17.1% respectively. The code will be released to facilitate future research.
Abstract (translated)
本文介绍了一种名为SmoothSync的新型框架，该框架旨在解决同步生成人类样貌手势过程中存在的诸如节奏不一致、动作抖动和脚步滑移等问题。SmoothSync利用新颖的双流Diffusion Transformer（DiT）架构中的量化音频令牌来合成整体手势，并增强采样的多样性。 具体而言，我们采用了以下方法： 1. 通过互补的Transformer流融合音频-运动特征以实现更优同步。 2. 引入抖动抑制损失函数，提升时间上的平滑度。 3. 实现概率性音频量化技术，从相同输入生成不同的手势序列。 为了可靠地评估在抖动情况下的节拍同步，我们引入了Smooth-BC，这是一种对运动噪声不敏感的节拍一致性指标的稳健变体。在BEAT2和SHOW数据集上的全面实验表明，与当前最佳方法相比，SmoothSync的表现更优，在BEAT2数据集中，它分别减少了30.6% FGD（固定手势差异）、提高了10.3% Smooth-BC以及增加了8.4%的多样性，并且在减少抖动和脚步滑移方面分别改善了62.9%和17.1%。未来的研究人员可以获取该代码以进一步开展相关研究工作。
URL
https://arxiv.org/abs/2601.04236
PDF
-
LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving
2025-12-31 08:27:10Qian Cheng, Weitao Zhou, Cheng Jing, Nanshan Deng, Junze Wen, Zhaoyang Liu, Kun Jiang, Diange YangAbstract
Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time this http URL work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.
Abstract (translated)
现实中的自动驾驶必须遵循超出法律规定的交通规则的复杂人类社会规范。许多这样的语义约束，例如给紧急车辆让路、遵守交通警察的手势或为校车停车等行为，对于人类来说是直观的，但对于机器来说却很难明确编码。尽管大型视觉语言模型（VLM）能够解释这些语义，但其推理成本使其无法在实时环境中使用。这项工作提出了LSRE框架，即潜在语义规则编码框架，它将稀疏采样的VLM判断转化为递归世界模型的隐空间中的决策边界。通过将用语言定义的安全语义编码到轻量级的隐分类器中，LSRE能够在不进行每帧VLM查询的情况下以10 Hz的速度实时评估语义风险。在CARLA环境下的六种语义故障场景实验表明，LSRE达到了与大型VLM基线相当的语义风险检测准确性，并提供了显著更早的安全预警同时保持低计算延迟。此外，LSRE还推广到很少见但语义相似的测试案例中，表明语言指导的潜在分类为自动驾驶中的语义安全监控提供了一种有效且可部署的方法。
URL
https://arxiv.org/abs/2512.24712
PDF
-
Kinematic-Based Assessment of Surgical Actions in Microanastomosis
2025-12-30 02:18:49Yan Meng, Daniel Donoho, Marcelle Altshuler, Omar ArnaoutAbstract
Proficiency in microanastomosis is a critical surgical skill in neurosurgery, where the ability to precisely manipulate fine instruments is crucial to successful outcomes. These procedures require sustained attention, coordinated hand movements, and highly refined motor skills, underscoring the need for objective and systematic methods to evaluate and enhance microsurgical training. Conventional assessment approaches typically rely on expert raters supervising the procedures or reviewing surgical videos, which is an inherently subjective process prone to inter-rater variability, inconsistency, and significant time investment. These limitations highlight the necessity for automated and scalable solutions. To address this challenge, we introduce a novel AI-driven framework for automated action segmentation and performance assessment in microanastomosis procedures, designed to operate efficiently on edge computing platforms. The proposed system comprises three main components: (1) an object tip tracking and localization module based on YOLO and DeepSORT; (2) an action segmentation module leveraging self-similarity matrix for action boundary detection and unsupervised clustering; and (3) a supervised classification module designed to evaluate surgical gesture proficiency. Experimental validation on a dataset of 58 expert-rated microanastomosis videos demonstrates the effectiveness of our approach, achieving a frame-level action segmentation accuracy of 92.4% and an overall skill classification accuracy of 85.5% in replicating expert evaluations. These findings demonstrate the potential of the proposed method to provide objective, real-time feedback in microsurgical education, thereby enabling more standardized, data-driven training protocols and advancing competency assessment in high-stakes surgical environments.
Abstract (translated)
在神经外科中，显微吻合术的熟练掌握是一项关键的手术技能。该技术需要精确操作精细仪器的能力以确保成功的治疗结果。这些程序要求持续集中注意力、协调的手部动作以及高度精炼的运动技巧，这凸显了客观和系统方法评估及提升显微手术培训的需求。 传统评估方法通常依赖于专家监督手术过程或审查手术视频，这是一个主观性很强的过程，容易导致评分者之间的一致性问题、不稳定性，并需要大量的时间投入。这些局限性突出了自动化的、可扩展的解决方案的必要性。 为了解决这一挑战，我们引入了一种新颖的人工智能驱动框架，用于显微吻合术程序中的自动化动作分割和性能评估，旨在有效运行于边缘计算平台上。该提议系统由三个主要组成部分构成：（1）基于YOLO和DeepSORT的对象尖端跟踪与定位模块；（2）利用自相似矩阵进行动作边界检测和无监督聚类的动作分割模块；以及（3）用于评估手术手势熟练程度的监督分类模块。 通过使用包含58个专家评分显微吻合术视频的数据集进行了实验验证，证明了我们方法的有效性。在该数据集中，我们的系统达到了92.4%的帧级动作分割准确率和85.5%的整体技能分类准确率，在再现专家评估方面取得了显著成果。 这些发现表明所提出的方法具有提供显微外科教育中客观、实时反馈的潜力，从而能够促进更为标准化、基于数据驱动的培训协议，并在高风险手术环境中推进能力评估。
URL
https://arxiv.org/abs/2512.23942
PDF
-
Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control
2025-12-29 17:59:24Zhe Li, Cheng Chi, Yangyang Wei, Boan Zhu, Tao Huang, Zhenguo Sun, Yibo Peng, Pengwei Wang, Zhongyuan Wang, Fangzhou Liu, Chang Xu, Shanghang ZhangAbstract
Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.
Abstract (translated)
人类天生会随着声音移动，但目前的人形机器人缺乏表达性和即兴表演能力，只能执行预定义的动作或少量命令。从音频生成动作并将其重新定位到机器人上通常依赖于显式的运动重构，这会导致级联误差、高延迟以及声学与驱动之间的映射不连续问题。 我们提出了RoboPerform，这是一个首个统一的音频至行走框架，可以直接根据音乐生成舞蹈，并且可以根据语音生成伴随手势。此框架基于“动作=内容+风格”的核心原则运作，将音频视为隐式的风格信号，从而消除了显式运动重构的需求。RoboPerform整合了一个ResMoE教师策略以适应各种运动模式和一个基于扩散的学生策略来注入音频风格信息。这种无需重新定位的设计确保了低延迟和高保真度。 实验验证显示，RoboPerform在物理真实性和声学同步方面取得了令人满意的结果，成功地将机器人转变成能够响应声音的互动表演者。
URL
https://arxiv.org/abs/2512.23650
PDF
-
Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition
2025-12-29 08:22:46Arman Martirosyan, Shahane Tigranyan, Maria Razzhivina, Artak Aslanyan, Nazgul Salikhova, Ilya Makarov, Andrey Savchenko, Aram AvetisyanAbstract
Micro-gesture recognition and behavior-based emotion prediction are both highly challenging tasks that require modeling subtle, fine-grained human behaviors, primarily leveraging video and skeletal pose data. In this work, we present two multimodal frameworks designed to tackle both problems on the iMiGUE dataset. For micro-gesture classification, we explore the complementary strengths of RGB and 3D pose-based representations to capture nuanced spatio-temporal patterns. To comprehensively represent gestures, video, and skeletal embeddings are extracted using MViTv2-S and 2s-AGCN, respectively. Then, they are integrated through a Cross-Modal Token Fusion module to combine spatial and pose information. For emotion recognition, our framework extends to behavior-based emotion prediction, a binary classification task identifying emotional states based on visual cues. We leverage facial and contextual embeddings extracted using SwinFace and MViTv2-S models and fuse them through an InterFusion module designed to capture emotional expressions and body gestures. Experiments conducted on the iMiGUE dataset, within the scope of the MiGA 2025 Challenge, demonstrate the robust performance and accuracy of our method in the behavior-based emotion prediction task, where our approach secured 2nd place.
Abstract (translated)
微手势识别和基于行为的情绪预测都是极具挑战性的任务，需要对细微的人体动作进行建模。这些任务主要依赖于视频和骨骼姿态数据。在这项工作中，我们提出了两个多模式框架，旨在解决iMiGUE数据集上的这两个问题。 对于微手势分类，我们探索了RGB表示与基于3D姿势的表示互补的优势，以捕捉细腻的空间-时间模式。为了全面表示手势、视频和骨骼嵌入，分别使用MViTv2-S模型提取视频特征，并利用2s-AGCN模型提取骨骼姿态数据。然后，通过跨模态标记融合模块将空间信息与姿态信息相结合。 在情绪识别方面，我们的框架扩展到了基于行为的情绪预测任务上，这是一个二分类问题，旨在根据视觉线索识别情感状态。我们使用SwinFace和MViTv2-S模型提取面部和上下文嵌入，并利用一个设计用于捕捉面部表情和肢体动作的InterFusion模块进行融合。 在MiGA 2025挑战赛中针对iMiGUE数据集进行的实验表明，我们的方法在基于行为的情绪预测任务上展现了强大的性能和准确性。我们在该比赛的任务中取得了第二名的成绩。
URL
https://arxiv.org/abs/2512.23291
PDF
-
StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
2025-12-26 15:41:24Zhiyao Sun, Ziqiao Peng, Yifeng Ma, Yi Chen, Zhengguang Zhou, Zixiang Zhou, Guozhen Zhang, Youliang Zhang, Yuan Zhou, Qinglin Lu, Yong-Jin LiuAbstract
Real-time, streaming interactive avatars represent a critical yet challenging goal in digital human research. Although diffusion-based human avatar generation methods achieve remarkable success, their non-causal architecture and high computational costs make them unsuitable for streaming. Moreover, existing interactive approaches are typically limited to head-and-shoulder region, limiting their ability to produce gestures and body motions. To address these challenges, we propose a two-stage autoregressive adaptation and acceleration framework that applies autoregressive distillation and adversarial refinement to adapt a high-fidelity human video diffusion model for real-time, interactive streaming. To ensure long-term stability and consistency, we introduce three key components: a Reference Sink, a Reference-Anchored Positional Re-encoding (RAPR) strategy, and a Consistency-Aware Discriminator. Building on this framework, we develop a one-shot, interactive, human avatar model capable of generating both natural talking and listening behaviors with coherent gestures. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing approaches in generation quality, real-time efficiency, and interaction naturalness. Project page: this https URL .
Abstract (translated)
实时流媒体互动化身代表了数字人类研究中的一个关键但具有挑战性的目标。尽管基于扩散的人类化身生成方法取得了显著的成功，但由于其非因果架构和高昂的计算成本，这些方法不适合用于流媒体应用。此外，现有的交互式方法通常局限于头部和肩部区域，限制了它们生成手势和身体动作的能力。 为了解决这些挑战，我们提出了一种两阶段自回归适应和加速框架，该框架通过自回归蒸馏和对抗性优化来调整高保真的人类视频扩散模型，使其适用于实时交互流媒体。为了确保长期稳定性和一致性，我们引入了三个关键组件：参考源（Reference Sink）、基于参考的位置重编码策略（RAPR）以及一种感知一致性的判别器。 在此框架的基础上，我们开发了一种一次性、互动式的人类化身模型，能够生成自然的说话和倾听行为，并且具备连贯的手势。广泛的实验表明，我们的方法在生成质量、实时效率及交互自然度方面均达到了业界领先水平，超过了现有的方法。项目页面：[请参见原文链接]。 请注意，最后一个句子中的“[请参见原文链接]”需要您查看原始文档以获取实际的链接地址。
URL
https://arxiv.org/abs/2512.22065
PDF
-
SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching
2025-12-26 12:32:39Xiangwen Zhang, Xiaowei Dai, Runnan Chen, Xiaoming Chen, Zeke Zexi HuAbstract
Creating physically realistic content in VR often requires complex modeling tools or predefined 3D models, textures, and animations, which present significant barriers for non-expert users. In this paper, we propose SketchPlay, a novel VR interaction framework that transforms humans' air-drawn sketches and gestures into dynamic, physically realistic scenes, making content creation intuitive and playful like drawing. Specifically, sketches capture the structure and spatial arrangement of objects and scenes, while gestures convey physical cues such as velocity, direction, and force that define movement and behavior. By combining these complementary forms of input, SketchPlay captures both the structure and dynamics of user-created content, enabling the generation of a wide range of complex physical phenomena, such as rigid body motion, elastic deformation, and cloth dynamics. Experimental results demonstrate that, compared to traditional text-driven methods, SketchPlay offers significant advantages in expressiveness, and user experience. By providing an intuitive and engaging creation process, SketchPlay lowers the entry barrier for non-expert users and shows strong potential for applications in education, art, and immersive storytelling.
Abstract (translated)
在虚拟现实中创建物理上逼真的内容通常需要复杂的建模工具或预定义的3D模型、纹理和动画，这些都为非专业用户设置了相当大的障碍。在这篇论文中，我们提出了SketchPlay，这是一种新颖的VR交互框架，它将人类在空气中的手绘草图和手势转化为动态且物理上逼真的场景，使内容创作直观而有趣，就像画画一样。具体而言，草图捕捉了物体和场景的结构和空间布局，而手势传达了诸如速度、方向和力等物理线索，这些定义了运动和行为。通过结合这两种互补形式的输入，SketchPlay 能够同时捕获用户创建内容的结构和动态性，从而生成广泛的复杂物理现象，包括刚体运动、弹性变形和布料动力学。 实验结果表明，与传统的基于文本的方法相比，SketchPlay 在表现力和用户体验方面具有显著优势。通过提供直观且引人入胜的内容创作过程，SketchPlay 降低了非专业用户的入门门槛，并显示出在教育、艺术和沉浸式叙事等领域的强大应用潜力。
URL
https://arxiv.org/abs/2512.22016
PDF
-
Xiaomi MiMo-VL-Miloco Technical Report
2025-12-19 10:43:37Jiaze Li, Jingyang Chen, Yuxun Qu, Jianzhong Ju, Zhenbo Luo, Jian Luan, Shijie Xu, Zhenru Lin, Junyou Zhu, Boshen Xu, Wenhui Tan, Pei FuAbstract
We open-source \textbf{MiMo-VL-Miloco-7B} and its quantized variant \textbf{MiMo-VL-Miloco-7B-GGUF}, a pair of home-centric vision-language models that achieve strong performance on both home-scenario understanding and general multimodal reasoning. Built on the MiMo-VL-7B backbone, MiMo-VL-Miloco-7B is specialized for smart-home environments, attaining leading F1 scores on gesture recognition and common home-scenario understanding, while also delivering consistent gains across video benchmarks such as Video-MME, Video-MMMU, and Charades-STA, as well as language understanding benchmarks including MMMU-Pro and MMLU-Pro. In our experiments, MiMo-VL-Miloco-7B outperforms strong closed-source and open-source baselines on home-scenario understanding and several multimodal reasoning benchmarks. To balance specialization and generality, we design a two-stage training pipeline that combines supervised fine-tuning with reinforcement learning based on Group Relative Policy Optimization, leveraging efficient multi-domain data. We further incorporate chain-of-thought supervision and token-budget-aware reasoning, enabling the model to learn knowledge in a data-efficient manner while also performing reasoning efficiently. Our analysis shows that targeted home-scenario training not only enhances activity and gesture understanding, but also improves text-only reasoning with only modest trade-offs on document-centric tasks. Model checkpoints, quantized GGUF weights, and our home-scenario evaluation toolkit are publicly available at \href{this https URL}{this https URL} to support research and deployment in real-world smart-home applications.
Abstract (translated)
我们开源了**MiMo-VL-Miloco-7B**及其量化版本**MiMo-VL-Miloco-7B-GGUF**，这是一对专注于家庭环境的视觉语言模型，在家用场景理解和通用多模态推理方面均表现出色。基于MiMo-VL-7B骨干网络，MiMo-VL-Miloco-7B专门针对智能家庭环境进行了优化，在手势识别和常见家庭情景理解上取得了领先的F1分数，并且在Video-MME、Video-MMMU、Charades-STA等视频基准测试以及MMMU-Pro和MMLU-Pro等语言理解基准测试中也表现出一致的优势。实验表明，MiMo-VL-Miloco-7B在家用场景理解和多项多模态推理基准测试上均超越了强大的闭源和开源基线模型。 为了平衡专业性和通用性，我们设计了一个两阶段的训练流程，结合监督微调与基于Group Relative Policy Optimization（GRPO）的强化学习方法，并利用高效的跨域数据。此外，我们进一步引入了链式思维监督及令牌预算感知推理机制，使得模型能够在高效的数据利用率下学习知识的同时，还能够进行有效的推理过程。 我们的分析表明，针对家庭情景的定向训练不仅增强了活动和手势的理解能力，而且还适度改善了仅限于文本的任务上的推理性能。模型检查点、量化GGUF权重以及我们家用场景评估工具包已公开发布在[此链接](this https URL)上，以支持研究与实际智能家庭应用中的部署工作。
URL
https://arxiv.org/abs/2512.17436
PDF
