Abstract
The global rise in the number of people with physical disabilities, in part due to improvements in post-trauma survivorship and longevity, has amplified the demand for advanced assistive technologies to improve mobility and independence. Autonomous assistive robots, such as smart wheelchairs, require robust capabilities in spatial segmentation and semantic recognition to navigate complex built environments effectively. Place segmentation involves delineating spatial regions like rooms or functional areas, while semantic recognition assigns semantic labels to these regions, enabling accurate localization to user-specific needs. Existing approaches often utilize deep learning; however, these close-vocabulary detection systems struggle to interpret intuitive and casual human instructions. Additionally, most existing methods ignore the uncertainty of the scene recognition problem, leading to low success rates, particularly in ambiguous and complex environments. To address these challenges, we propose an open-vocabulary scene semantic segmentation and detection pipeline leveraging Vision Language Models (VLMs) and Large Language Models (LLMs). Our approach follows a 'Segment Detect Select' framework for open-vocabulary scene classification, enabling adaptive and intuitive navigation for assistive robots in built environments.
Abstract (translated)
全球因创伤幸存率和寿命的提高，导致身体残疾人数增加，这加大了对能够改善移动能力和独立性的先进辅助技术的需求。自主辅助机器人（如智能轮椅）需要具备强大的空间分割与语义识别能力才能有效地在复杂的建筑环境中导航。空间分割涉及划分诸如房间或功能区域等空间区域，而语义识别则为这些区域分配语义标签，从而能够根据用户的特定需求进行精准定位。现有方法通常利用深度学习技术；然而，这些封闭词汇表检测系统难以理解直观且随意的人类指令。此外，大多数现有的方法忽视了场景识别问题中的不确定性，导致成功率较低，尤其是在模糊和复杂的环境中。 为了解决这些问题，我们提出了一种基于视觉语言模型（VLMs）和大规模语言模型（LLMs）的开放词汇表场景语义分割与检测流水线。我们的方法遵循“Segment Detect Select”框架进行开放词汇表场景分类，使辅助机器人能够在建筑环境中实现适应性和直观导航。
URL
https://arxiv.org/abs/2503.23105
