Abstract
We explore Bird's-Eye View (BEV) generation, converting a BEV map into its corresponding multi-view street images. Valued for its unified spatial representation aiding multi-sensor fusion, BEV is pivotal for various autonomous driving applications. Creating accurate street-view images from BEV maps is essential for portraying complex traffic scenarios and enhancing driving algorithms. Concurrently, diffusion-based conditional image generation models have demonstrated remarkable outcomes, adept at producing diverse, high-quality, and condition-aligned results. Nonetheless, the training of these models demands substantial data and computational resources. Hence, exploring methods to fine-tune these advanced models, like Stable Diffusion, for specific conditional generation tasks emerges as a promising avenue. In this paper, we introduce a practical framework for generating images from a BEV layout. Our approach comprises two main components: the Neural View Transformation and the Street Image Generation. The Neural View Transformation phase converts the BEV map into aligned multi-view semantic segmentation maps by learning the shape correspondence between the BEV and perspective views. Subsequently, the Street Image Generation phase utilizes these segmentations as a condition to guide a fine-tuned latent diffusion model. This finetuning process ensures both view and style consistency. Our model leverages the generative capacity of large pretrained diffusion models within traffic contexts, effectively yielding diverse and condition-coherent street view images.
Abstract (translated)
我们探讨了Bird's-Eye View (BEV) 生成，将BEV图转换为相应的多视角街头图像。由于其统一的空间表示帮助多传感器融合，BEV对于各种自动驾驶应用至关重要。从BEV图创建准确的街头图像至关重要，以描绘复杂的交通场景并提高驾驶算法。同时，基于扩散的条件下图像生成模型已经取得了显著的成果，擅长产生多样、高品质和条件相关的结果。然而，这些模型的训练需要大量数据和计算资源。因此，探索为特定条件生成任务对先进模型进行微调的方法，如Stable Diffusion，被视为一个有前景的途径。 在本文中，我们介绍了一种从BEV布局生成图像的实用框架。我们的方法包括两个主要组件：神经视图变换和街头图像生成。神经视图变换阶段通过学习BEV和透视视图之间的形状匹配将BEV图转换为对齐的多视角语义分割图。然后，街头图像生成阶段利用这些分割图作为条件来指导微调的潜在扩散模型。这一微调过程确保了视力和风格的一致性。我们的模型利用交通背景下大型预训练扩散模型的生成能力，有效地产生了多样且条件相关的街头图像。
URL
https://arxiv.org/abs/2409.01014
