Abstract
We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.
Abstract (translated)
我们介绍了DreamToNav，这是一种新颖的自主机器人框架，它使用生成式视频模型来实现直观、人机协作控制。与依赖固定的航点导航不同，用户可以提供自然语言指令（例如“小心地跟随那个人”），系统将这些指令转换为可执行的动作。我们的流程首先利用Qwen 2.5-VL-7B-Instruct将模糊的用户指令细化为精确的视觉描述。然后使用最先进的视频基础模型NVIDIA Cosmos 2.5根据这些描述合成一段机器人执行任务时的物理上一致的视频序列。从这段合成视频中，我们通过视觉姿态估计、机器人检测和轨迹恢复提取出有效的运动路径。将视频生成视为规划引擎后，DreamToNav可以让机器人在执行之前“想象”复杂的行为，为不同平台提供统一的避障及目标导向导航框架，并且无需针对特定任务进行工程设计。 我们在室内导航任务中对这款轮式移动机器人和四足机器人的DreamToNav进行了评估。DreamToNav的成功率为76.7%，最终目标误差通常在0.05-0.10米之间，轨迹跟踪误差低于0.15米。这些结果表明，从生成视频预测中提取的轨迹可以在不同运动平台上的物理机器人上可靠地执行。
URL
https://arxiv.org/abs/2603.06190
