Abstract
We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.
Abstract (translated)
我们提出了一种开创性的手语驱动的视觉-语言-动作（VLA）框架，旨在实现直观且包容的人机交互。与传统的依赖于词素注释作为中间监督的方法不同，所提出的系统采用无词素范式，并直接将视觉手势映射为语义指令。这种设计减少了标注成本，避免了由词素表示引入的信息损失，从而实现了更自然和可扩展的多模态互动。在本研究中，我们专注于一个实时的手字母级手指拼写界面，该界面提供了一个可靠且低延迟的机器人控制通信通道。 与大规模连续手语识别相比，字母级别的交互在安全关键性的实体环境中提供了更高的可靠性、可解释性和部署可行性。所提出的流水线通过几何标准化、时间平滑和词汇精炼将连续手势流转换为连贯的语言命令，确保了稳定且一致的互动体验。此外，该框架设计支持未来基于变压器的无词素手语模型的集成，从而能够实现单词级和句子级的语义理解。 实验结果证明，在各种交互场景中，所提出的系统能有效地将手势指令转化为精确的机器人动作，突显了该框架在推进可访问性、可扩展性和多模态实体智能方面的潜力。
URL
https://arxiv.org/abs/2602.22514
