Abstract
Autonomous driving requires rich contextual comprehension and precise predictive reasoning to navigate dynamic and complex environments safely. Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge. VLMs provide interpretability and robust action prediction through their ability to understand multi-modal context, while DWMs excel in generating detailed and plausible future driving scenarios essential for proactive planning. Integrating VLMs with DWMs is an intuitive, promising, yet understudied strategy to exploit the complementary strengths of accurate behavioral prediction and realistic scene generation. Nevertheless, this integration presents notable challenges, particularly in effectively connecting action-level decisions with high-fidelity pixel-level predictions and maintaining computational efficiency. In this paper, we propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer to form a unified imagination-and-planning loop. The driving agent predicts initial driving trajectories based on multi-modal inputs, guiding the scene imaginer to generate corresponding future scenarios. These imagined scenarios are subsequently utilized to iteratively refine the driving agent's planning decisions. To address efficiency and predictive accuracy challenges inherent in this integration, we introduce an early stopping mechanism and a trajectory selection strategy. Extensive experimental validation on the nuScenes and NAVSIM datasets demonstrates the robustness and superiority of ImagiDrive over previous alternatives under both open-loop and closed-loop conditions.
Abstract (translated)
自动驾驶需要丰富的语境理解和精确的预测推理，以安全地导航动态和复杂的环境。视觉语言模型（VLMs）和驾驶世界模型（DWMs）分别在解决这一挑战的不同方面中独立涌现为强有力的解决方案。VLMs通过其理解多模态上下文的能力提供解释性和稳健的动作预测，而DWMs则擅长生成详细且可信的未来驾驶场景，这对于主动规划至关重要。将VLM与DWM结合是一种直观、有前景但研究不足的战略，旨在利用准确的行为预测和逼真场景生成的互补优势。然而，这种整合带来了显著挑战，特别是在有效地连接行动级决策与高保真的像素级预测以及保持计算效率方面。 在本文中，我们提出了ImagiDrive——一种新颖的端到端自动驾驶框架，它将基于VLM的驾驶代理与基于DWM的场景想象器相结合，形成统一的想象和规划循环。该驾驶代理根据多模态输入预测初始行驶轨迹，并指导场景想象器生成相应的未来情景。这些被设想的情景随后用于迭代优化驾驶代理的规划决策。 为了应对这种整合中固有的效率和预测准确性挑战，我们引入了一个早期停止机制和一种轨迹选择策略。在nuScenes和NAVSIM数据集上的广泛实验验证表明，在开环和闭环条件下，ImagiDrive比以前的方法具有更强健性和优越性。
URL
https://arxiv.org/abs/2508.11428
