Scene_Generation
-
Cube: A Roblox View of 3D Intelligence
2025-03-19 17:52:17Foundation AI Team, Kiran Bhat, Nishchaie Khanna, Karun Channa, Tinghui Zhou, Yiheng Zhu, Xiaoxia Sun, Charles Shang, Anirudh Sudarshan, Maurice Chu, Daiqing Li, Kangle Deng, Jean-Philippe Fauconnier, Tijmen Verhulsdonck, Maneesh Agrawala, Kayvon Fatahalian, Alexander Weiss, Christian Reiser, Ravi Kiran Chirravuri, Ravali Kandur, Alejandro Pelaez, Akash Garg, Michael Palleschi, Jessica Wang, Skylar Litz, Leon Liu, Anying Li, David Harmon, Derek Liu, Liangjun Feng, Denis Goupil, Lukas Kuczynski, Jihyun Yoon, Naveen Marri, Peiye Zhuang, Yinan Zhang, Brian Yin, Haomiao Jiang, Marcel van Workum, Thomas Lane, Bryce Erickson, Salil Pathare, Kyle Price, Anupam Singh, David BaszuckiAbstract
Foundation models trained on vast amounts of data have demonstrated remarkable reasoning and generation capabilities in the domains of text, images, audio and video. Our goal at Roblox is to build such a foundation model for 3D intelligence, a model that can support developers in producing all aspects of a Roblox experience, from generating 3D objects and scenes to rigging characters for animation to producing programmatic scripts describing object behaviors. We discuss three key design requirements for such a 3D foundation model and then present our first step towards building such a model. We expect that 3D geometric shapes will be a core data type and describe our solution for 3D shape tokenizer. We show how our tokenization scheme can be used in applications for text-to-shape generation, shape-to-text generation and text-to-scene generation. We demonstrate how these applications can collaborate with existing large language models (LLMs) to perform scene analysis and reasoning. We conclude with a discussion outlining our path to building a fully unified foundation model for 3D intelligence.
Abstract (translated)
基于大量数据训练的基础模型在文本、图像、音频和视频领域展示了出色的推理和生成能力。我们的目标是在Roblox上构建一个面向三维智能的基础模型，该模型能够支持开发者从生成三维物体和场景到为动画搭建角色再到编写描述对象行为的程序脚本等各个方面的工作。我们将讨论这样一个3D基础模型的关键设计要求，并介绍我们朝此方向迈出的第一步。我们预期3D几何形状将成为一种核心数据类型，并且会详细说明我们的3D形状标记化解决方案。我们会展示如何使用我们的标记化方案在文本到形状生成、形状到文本生成和文本到场景生成等应用中进行操作。此外，我们还会演示这些应用程序如何与现有的大规模语言模型（LLMs）协作以执行场景分析和推理工作。最后，我们将讨论构建一个完全统一的面向3D智能的基础模型的道路。 具体而言： 1. **核心设计要求**：为了创建能够支持开发者在Roblox上实现各种功能的3D基础模型，该模型需要具备理解、生成及修改三维数据的能力。 2. **初步步骤**：我们首先确定了3D几何形状作为核心数据类型，并开发了一个专门用于处理这些形状的标记化系统。这个3D形状标记器是构建更复杂模型的第一步。 3. **应用实例与协作**： - 文本到形状生成（Text-to-Shape Generation）和形状到文本生成（Shape-to-Text Generation）：通过使用我们的标记方案，可以将自然语言描述转换为具体的三维物体或场景，并且反过来也能从几何结构中提取出有意义的文本信息。 - 文本到场景生成（Text-to-Scene Generation）：这一功能使用户能够直接输入描述来创建复杂的三维环境，极大地简化了创作流程。 4. **与现有LLMs的合作**： 通过整合3D基础模型和现有的大规模语言模型（如GPT系列），我们可以利用LLM的文本处理能力来进行更高级别的场景分析及推理工作。这不仅能够增强用户交互体验，还为自动化内容生成开辟了新的可能。 5. **未来展望**：我们期望这些初步尝试将成为构建一个全面支持三维智能的基础模型的重要基石，在这一过程中推动更多创新和应用的发展。 综上所述，我们的目标是建立一套完整统一的3D基础模型体系，这将显著提高开发者在Roblox平台上的创造力与生产力。
URL
https://arxiv.org/abs/2503.15475
PDF
-
DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
2025-03-19 13:49:48Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, Hao ZhaoAbstract
Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. A key challenge lies in finding an efficient and generalizable geometric representation that seamlessly connects temporal and spatial synthesis. To address this, we propose DiST-4D, the first disentangled spatiotemporal diffusion framework for 4D driving scene generation, which leverages metric depth as the core geometric representation. DiST-4D decomposes the problem into two diffusion processes: DiST-T, which predicts future metric depth and multi-view RGB sequences directly from past observations, and DiST-S, which enables spatial NVS by training only on existing viewpoints while enforcing cycle consistency. This cycle consistency mechanism introduces a forward-backward rendering constraint, reducing the generalization gap between observed and unseen viewpoints. Metric depth is essential for both accurate reliable forecasting and accurate spatial NVS, as it provides a view-consistent geometric representation that generalizes well to unseen perspectives. Experiments demonstrate that DiST-4D achieves state-of-the-art performance in both temporal prediction and NVS tasks, while also delivering competitive performance in planning-related evaluations.
Abstract (translated)
当前的生成模型在合成动态4D驾驶场景时遇到了挑战，这些场景需要同时支持时间外推和空间新视角合成（NVS），而无需针对每个场景进行优化。关键难题在于寻找一种高效且通用化的几何表示方法，能够无缝地连接时间与空间的合成过程。为了解决这一问题，我们提出了DiST-4D，这是一种用于生成4D驾驶场景的分解时空扩散框架，它利用了度量深度作为核心几何表示。 DiST-4D将问题分解成两个扩散过程：DiST-T和DiST-S。其中，DiST-T能够直接从过去的观测数据预测未来的度量深度以及多视角RGB序列；而DiST-S则通过仅在现有视角上训练并强制执行循环一致性来实现空间NVS。这种循环一致性机制引入了前向-后向渲染约束，从而减少了观察到的和未见过的观点之间的泛化差距。 度量深度对于准确可靠的预测和精确的空间NVS都是至关重要的，因为它提供了一种视图一致性的几何表示方法，并且能够很好地推广应用于未知视角。实验结果表明，DiST-4D在时间预测任务和NVS任务中都达到了最先进的性能水平，在与规划相关的评估方面也表现出竞争性表现。
URL
https://arxiv.org/abs/2503.15208
PDF
-
SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis
2025-03-18 22:02:35Hou In Ivan Tam, Hou In Derek Pun, Austin T. Wang, Angel X. Chang, Manolis SavvaAbstract
Despite recent advances in text-conditioned 3D indoor scene generation, there remain gaps in the evaluation of these methods. Existing metrics primarily assess the realism of generated scenes by comparing them to a set of ground-truth scenes, often overlooking alignment with the input text - a critical factor in determining how effectively a method meets user requirements. We present SceneEval, an evaluation framework designed to address this limitation. SceneEval includes metrics for both explicit user requirements, such as the presence of specific objects and their attributes described in the input text, and implicit expectations, like the absence of object collisions, providing a comprehensive assessment of scene quality. To facilitate evaluation, we introduce SceneEval-100, a dataset of scene descriptions with annotated ground-truth scene properties. We evaluate recent scene generation methods using SceneEval and demonstrate its ability to provide detailed assessments of the generated scenes, highlighting strengths and areas for improvement across multiple dimensions. Our results show that current methods struggle at generating scenes that meet user requirements, underscoring the need for further research in this direction.
Abstract (translated)
尽管在文本条件下的3D室内场景生成方面取得了进展，但这些方法的评估仍然存在不足之处。现有的度量标准主要通过将生成的场景与一组真实的场景进行比较来衡量其逼真程度，往往忽略了与输入文本的一致性——这是决定一种方法能否有效满足用户需求的关键因素。我们提出了SceneEval这一评估框架，旨在解决这一局限性。SceneEval包括了针对明确的用户要求（如输入文本中描述的具体对象及其属性）和隐含期望（例如没有物体碰撞情况）的度量标准，从而对场景质量进行全面评估。为了便于评估，我们引入了包含注释的真实场景属性的100个场景描述的数据集SceneEval-100。我们使用SceneEval来评估最近的场景生成方法，并展示了它能够提供详细的生成场景评估结果，突出显示在多个维度上的优点和改进空间。我们的结果显示，当前的方法难以生成满足用户需求的场景，强调了在此方向上进行进一步研究的需求。
URL
https://arxiv.org/abs/2503.14756
PDF
-
Advances in 4D Generation: A Survey
2025-03-18 17:59:51Qiaowei Miao, Kehan Li, Jinsheng Quan, Zhiyuan Min, Shaojie Ma, Yichao Xu, Yi Yang, Yawei LuoAbstract
Generative artificial intelligence has witnessed remarkable advancements across multiple domains in recent years. Building on the successes of 2D and 3D content generation, 4D generation, which incorporates the temporal dimension into generative tasks, has emerged as a burgeoning yet rapidly evolving research area. This paper presents a comprehensive survey of this emerging field, systematically examining its theoretical foundations, key methodologies, and practical applications, with the aim of providing readers with a holistic understanding of the current state and future potential of 4D generation. We begin by introducing the core concepts of 4D data representations, encompassing both structured and unstructured formats, and their implications for generative tasks. Building upon this foundation, we delve into the enabling technologies that drive 4D generation, including advancements in spatiotemporal modeling, neural representations, and generative frameworks. We further review recent studies that employ diverse control mechanisms and representation strategies for generating 4D outputs, categorizing these approaches and summarizing their research trajectories. In addition, we explore the wide-ranging applications of 4D generation techniques, spanning dynamic object modeling, scene generation, digital human synthesis, 4D content editing, and autonomous driving. Finally, we analyze the key challenges inherent to 4D generation, such as data availability, computational efficiency, and spatiotemporal consistency, and propose promising directions for future research. Our code is publicly available at: \href{this https URL}{this https URL}.
Abstract (translated)
近年来，生成式人工智能在多个领域取得了显著进展。继二维和三维内容生成的成功之后，四维生成作为一种新兴但迅速发展的研究领域应运而生，它将时间维度融入到生成任务中。本文旨在对这一新兴领域的现状进行全面的综述，系统地审视其理论基础、关键方法和技术应用，为读者提供关于四维生成当前状态和未来潜力的整体理解。我们首先介绍四维数据表示的核心概念，包括结构化和非结构化格式，并探讨它们在生成任务中的影响。在此基础上，我们将深入研究推动四维生成的技术，如时空建模的进步、神经网络表示以及生成框架的发展。此外，本文还回顾了最近使用多种控制机制和表示策略来生成四维输出的研究工作，对这些方法进行分类并总结其研究路径。我们进一步探讨四维生成技术的广泛应用领域，包括动态对象建模、场景生成、数字人类合成、四维内容编辑及自动驾驶等。最后，本文分析了四维生成所面临的挑战，如数据可用性、计算效率和时空一致性，并提出未来研究的潜在方向。 我们的代码可在以下链接获取：[此 HTTPS URL](this https URL)。
URL
https://arxiv.org/abs/2503.14501
PDF
-
Bolt3D: Generating 3D Scenes in Seconds
2025-03-18 17:24:19Stanislaw Szymanowicz, Jason Y. Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T. Barron, Philipp HenzlerAbstract
We present a latent diffusion model for fast feed-forward 3D scene generation. Given one or more images, our model Bolt3D directly samples a 3D scene representation in less than seven seconds on a single GPU. We achieve this by leveraging powerful and scalable existing 2D diffusion network architectures to produce consistent high-fidelity 3D scene representations. To train this model, we create a large-scale multiview-consistent dataset of 3D geometry and appearance by applying state-of-the-art dense 3D reconstruction techniques to existing multiview image datasets. Compared to prior multiview generative models that require per-scene optimization for 3D reconstruction, Bolt3D reduces the inference cost by a factor of up to 300 times.
Abstract (translated)
我们提出了一种用于快速前馈生成三维场景的潜在扩散模型。给定一个或多个图像，我们的模型Bolt3D可以直接在单个GPU上以不到七秒的时间采样出高质量的三维场景表示。通过利用强大的、可扩展的现有二维扩散网络架构，我们的模型能够生成一致且高保真的三维场景表示。为了训练这个模型，我们创建了一个大规模的多视角一致数据集，该数据集包含了3D几何和外观信息，这是通过对现有的多视图图像数据集应用最先进的密集三维重建技术得到的。与之前的需要针对每个场景进行三维重建优化的多视图生成模型相比，Bolt3D将推理成本降低了高达300倍。
URL
https://arxiv.org/abs/2503.14445
PDF
-
SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
2025-03-18 06:41:02Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, Yunfeng AiAbstract
With the rapid advancement of autonomous driving technology, a lack of data has become a major obstacle to enhancing perception model accuracy. Researchers are now exploring controllable data generation using world models to diversify datasets. However, previous work has been limited to studying image generation quality on specific public datasets. There is still relatively little research on how to build data generation engines for real-world application scenes to achieve large-scale data generation for challenging scenes. In this paper, a simulator-conditioned scene generation engine based on world model is proposed. By constructing a simulation system consistent with real-world scenes, simulation data and labels, which serve as the conditions for data generation in the world model, for any scenes can be collected. It is a novel data generation pipeline by combining the powerful scene simulation capabilities of the simulation engine with the robust data generation capabilities of the world model. In addition, a benchmark with proportionally constructed virtual and real data, is provided for exploring the capabilities of world models in real-world scenes. Quantitative results show that these generated images significantly improve downstream perception models performance. Finally, we explored the generative performance of the world model in urban autonomous driving scenarios. All the data and code will be available at this https URL.
Abstract (translated)
随着自动驾驶技术的迅速发展，数据不足已成为提高感知模型精度的主要障碍。研究人员正探索使用世界模型进行可控的数据生成以丰富数据集。然而，之前的工作主要集中在特定公开数据集上的图像生成质量研究上，对于如何构建适用于真实应用场景的数据生成引擎以实现大规模挑战性场景下的数据生成的研究还相对较少。 本文提出了一种基于世界模型的仿真条件化场景生成引擎。通过构建与现实场景一致的仿真系统，可以收集任何场景所需的模拟数据和标签作为在世界模型中进行数据生成的条件。这是一种创新性的数据生成流程，它结合了仿真引擎强大的场景模拟能力和世界模型的数据生成能力。 此外，本文还提供了一个由虚拟数据和实际数据按比例构建的基准测试平台，用以探索世界模型在现实场景中的能力。定量结果表明，这些生成的图像显著提升了下游感知模型的表现性能。最后，我们研究了世界模型在城市自动驾驶场景下的生成性能。所有数据及代码将在该网址提供：[此 URL]
URL
https://arxiv.org/abs/2503.13952
PDF
-
ChatBEV: A Visual Language Model that Understands BEV Maps
2025-03-18 06:12:38Qingyao Xu, Siheng Chen, Guang Chen, Yanfeng Wang, Ya ZhangAbstract
Traffic scene understanding is essential for intelligent transportation systems and autonomous driving, ensuring safe and efficient vehicle operation. While recent advancements in VLMs have shown promise for holistic scene understanding, the application of VLMs to traffic scenarios, particularly using BEV maps, remains under explored. Existing methods often suffer from limited task design and narrow data amount, hindering comprehensive scene understanding. To address these challenges, we introduce ChatBEV-QA, a novel BEV VQA benchmark contains over 137k questions, designed to encompass a wide range of scene understanding tasks, including global scene understanding, vehicle-lane interactions, and vehicle-vehicle interactions. This benchmark is constructed using an novel data collection pipeline that generates scalable and informative VQA data for BEV maps. We further fine-tune a specialized vision-language model ChatBEV, enabling it to interpret diverse question prompts and extract relevant context-aware information from BEV maps. Additionally, we propose a language-driven traffic scene generation pipeline, where ChatBEV facilitates map understanding and text-aligned navigation guidance, significantly enhancing the generation of realistic and consistent traffic scenarios. The dataset, code and the fine-tuned model will be released.
Abstract (translated)
交通场景理解对于智能运输系统和自动驾驶至关重要，确保车辆的安全高效运行。虽然最近视觉语言模型（VLMs）在全面场景理解方面取得了显著进展，但将这些技术应用于基于鸟瞰图（BEV）的地图的交通场景中仍处于探索阶段。现有方法通常受限于任务设计狭窄及数据量有限的问题，这阻碍了对复杂交通场景的理解。为了解决这些问题，我们引入了一种新的基准测试ChatBEV-QA，这是一个包含超过137,000个问题的BEV视觉问答（VQA）基准测试，旨在涵盖广泛的场景理解任务，包括全局场景理解、车辆与车道之间的互动以及车辆间的互动。该基准测试通过一个创新的数据收集流程构建而成，能够生成用于BEV地图的大规模和信息丰富的VQA数据。 我们进一步微调了一个专门的视觉语言模型ChatBEV，使其能够解析各种问题提示，并从BEV地图中提取相关且上下文相关的资讯。此外，我们还提出了一种由语言驱动的交通场景生成流程，在此过程中，ChatBEV促进了对地图的理解以及文本与导航指导的一致性，显著提升了现实主义和一致性的交通场景生成。 该数据集、代码及微调后的模型将在未来发布。
URL
https://arxiv.org/abs/2503.13938
PDF
-
Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception
2025-03-17 17:59:50Dingkang Liang, Dingyuan Zhang, Xin Zhou, Sifan Tu, Tianrui Feng, Xiaofan Li, Yumeng Zhang, Mingyang Du, Xiao Tan, Xiang BaiAbstract
We present UniFuture, a simple yet effective driving world model that seamlessly integrates future scene generation and perception within a single framework. Unlike existing models focusing solely on pixel-level future prediction or geometric reasoning, our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions. Specifically, during the training, we first introduce a Dual-Latent Sharing scheme, which transfers image and depth sequence in a shared latent space, allowing both modalities to benefit from shared feature learning. Additionally, we propose a Multi-scale Latent Interaction mechanism, which facilitates bidirectional refinement between image and depth features at multiple spatial scales, effectively enhancing geometry consistency and perceptual alignment. During testing, our UniFuture can easily predict high-consistency future image-depth pairs by only using the current image as input. Extensive experiments on the nuScenes dataset demonstrate that UniFuture outperforms specialized models on future generation and perception tasks, highlighting the advantages of a unified, structurally-aware world model. The project page is at this https URL.
Abstract (translated)
我们介绍了UniFuture，这是一种简单而有效的驾驶世界模型，它在一个单一框架内无缝整合了未来场景生成和感知。与现有的仅专注于像素级未来预测或几何推理的模型不同，我们的方法同时建模未来的外观（即RGB图像）和几何结构（即深度），确保一致的预测结果。具体来说，在训练阶段，我们首先引入了一种双潜在共享方案，该方案在一个共享的潜在空间中传输图像和深度序列，使两种模式都能从共同特征学习中获益。此外，我们还提出了一种多尺度潜互动机制，这一机制促进了在多个空间尺度下图象与深度特征之间的双向细化过程，有效增强了几何一致性及感知对齐。 测试阶段，UniFuture可以通过仅使用当前图像作为输入来轻易预测高一致性的未来图像-深度配对。在nuScenes数据集上的广泛实验表明，UniFuture在未来的生成和感知任务中超越了专门化的模型，突显出统一的、结构意识强的世界模型的优势。项目的网页链接如下：[此URL]（请将方括号中的文字替换为实际网址）。
URL
https://arxiv.org/abs/2503.13587
PDF
-
SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steering
2025-03-15 07:20:08Byeongjun Park, Hyojun Go, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick KimAbstract
Recent progress in 3D/4D scene generation emphasizes the importance of physical alignment throughout video generation and scene reconstruction. However, existing methods improve the alignment separately at each stage, making it difficult to manage subtle misalignments arising from another stage. Here, we present SteerX, a zero-shot inference-time steering method that unifies scene reconstruction into the generation process, tilting data distributions toward better geometric alignment. To this end, we introduce two geometric reward functions for 3D/4D scene generation by using pose-free feed-forward scene reconstruction models. Through extensive experiments, we demonstrate the effectiveness of SteerX in improving 3D/4D scene generation.
Abstract (translated)
最近在3D和4D场景生成方面的进展强调了在整个视频生成和场景重建过程中物理对齐的重要性。然而，现有的方法是在每个阶段单独改进对齐，这使得难以管理由其他阶段产生的微妙不对齐问题。在这里，我们介绍了SteerX，这是一种零样本推理时间引导方法，它将场景重建统一到生成过程之中，使数据分布向更好的几何对齐方向倾斜。为此，我们通过使用无姿态的前馈场景重构模型引入了两个用于3D和4D场景生成的几何奖励函数。通过广泛的实验，我们展示了SteerX在改进3D和4D场景生成方面的有效性。
URL
https://arxiv.org/abs/2503.12024
PDF
-
WonderVerse: Extendable 3D Scene Generation with Video Generative Models
2025-03-12 08:44:51Hao Feng, Zhi Zuo, Jia-hui Pan, Ka-hei Hui, Yi-hua Shao, Qi Dou, Wei Xie, Zheng-zhe LiuAbstract
We introduce \textit{WonderVerse}, a simple but effective framework for generating extendable 3D scenes. Unlike existing methods that rely on iterative depth estimation and image inpainting, often leading to geometric distortions and inconsistencies, WonderVerse leverages the powerful world-level priors embedded within video generative foundation models to create highly immersive and geometrically coherent 3D environments. Furthermore, we propose a new technique for controllable 3D scene extension to substantially increase the scale of the generated environments. Besides, we introduce a novel abnormal sequence detection module that utilizes camera trajectory to address geometric inconsistency in the generated videos. Finally, WonderVerse is compatible with various 3D reconstruction methods, allowing both efficient and high-quality generation. Extensive experiments on 3D scene generation demonstrate that our WonderVerse, with an elegant and simple pipeline, delivers extendable and highly-realistic 3D scenes, markedly outperforming existing works that rely on more complex architectures.
Abstract (translated)
我们介绍了一个名为\textit{WonderVerse}的框架，这是一个简单却有效的生成可扩展三维场景的方法。与现有的依赖于迭代深度估计和图像修补方法不同，后者常常导致几何扭曲和不一致性，WonderVerse 利用了视频生成基础模型中嵌入的世界级先验知识来创建高度沉浸式且几何一致的3D环境。此外，我们提出了一种可控的3D场景扩展技术，以显著增加生成环境的规模。另外，我们引入了一个新的异常序列检测模块，该模块利用摄像机轨迹解决生成视频中的几何不一致性问题。最后，WonderVerse 兼容各种3D重建方法，既支持高效又高质量的生成。 在三维场景生成方面的广泛实验表明，我们的 WonderVerse 通过一个优雅且简单的管道，能够提供可扩展和高度逼真的3D场景，显著优于依赖更复杂架构的现有工作。
URL
https://arxiv.org/abs/2503.09160
PDF
-
Controllable 3D Outdoor Scene Generation via Scene Graphs
2025-03-10 10:26:08Yuheng Liu, Xinke Li, Yuning Zhang, Lu Qi, Xin Li, Wenping Wang, Chongshou Li, Xueting Li, Ming-Hsuan YangAbstract
Three-dimensional scene generation is crucial in computer vision, with applications spanning autonomous driving, gaming and the metaverse. Current methods either lack user control or rely on imprecise, non-intuitive conditions. In this work, we propose a method that uses, scene graphs, an accessible, user friendly control format to generate outdoor 3D scenes. We develop an interactive system that transforms a sparse scene graph into a dense BEV (Bird's Eye View) Embedding Map, which guides a conditional diffusion model to generate 3D scenes that match the scene graph description. During inference, users can easily create or modify scene graphs to generate large-scale outdoor scenes. We create a large-scale dataset with paired scene graphs and 3D semantic scenes to train the BEV embedding and diffusion models. Experimental results show that our approach consistently produces high-quality 3D urban scenes closely aligned with the input scene graphs. To the best of our knowledge, this is the first approach to generate 3D outdoor scenes conditioned on scene graphs.
Abstract (translated)
三维场景生成在计算机视觉领域至关重要，其应用范围涵盖自动驾驶、游戏以及元宇宙。当前的方法要么缺乏用户控制能力，要么依赖于不精确且不易理解的条件。在这项工作中，我们提出了一种使用场景图（一种易用且友好的用户控制格式）来生成户外3D场景的方法。我们开发了一个交互式系统，该系统将稀疏的场景图转换为密集的BEV（鸟瞰视图）嵌入地图，并指导有条件的扩散模型生成与场景图描述相匹配的3D场景。在推理过程中，用户可以轻松创建或修改场景图以生成大规模户外场景。我们创建了一个大型数据集，其中包括配对的场景图和三维语义场景，用于训练BEV嵌入和扩散模型。实验结果表明，我们的方法始终能够产生高质量、与输入场景图紧密一致的3D城市场景。据我们所知，这是第一个基于场景图生成三维户外场景的方法。
URL
https://arxiv.org/abs/2503.07152
PDF
-
Unlocking Generalization for Robotics via Modularity and Scale
Abstract
How can we build generalist robot systems? Scale may not be enough due to the significant multimodality of robotics tasks, lack of easily accessible data and the challenges of deploying on physical hardware. Meanwhile, most deployed robotic systems today are inherently modular and can leverage the independent generalization capabilities of each module to perform well. Therefore, this thesis seeks to tackle the task of building generalist robot agents by integrating these components into one: combining modularity with large-scale learning for general purpose robot control. The first question we consider is: how can we build modularity and hierarchy into learning systems? Our key insight is that rather than having the agent learn hierarchy and low-level control end-to-end, we can enforce modularity via planning to enable more efficient and capable robot learners. Next, we come to the role of scale in building generalist robot systems. To scale, neural networks require vast amounts of diverse data, expressive architectures to fit the data and a source of supervision to generate the data. We leverage a powerful supervision source: classical planning, which can generalize, but is expensive to run and requires access to privileged information to perform well in practice. We use these planners to supervise large-scale policy learning in simulation to produce generalist agents. Finally, we consider how to unify modularity with large-scale policy learning to build real-world robot systems capable of performing zero-shot manipulation. We do so by tightly integrating key ingredients of modular high and mid-level planning, learned local control, procedural scene generation and large-scale policy learning for sim2real transfer. We demonstrate that this recipe can produce a single, generalist agent that can solve challenging long-horizon manipulation tasks in the real world.
Abstract (translated)
如何构建通用机器人系统？由于机器人任务的多模态特性、缺乏易于获取的数据以及在物理硬件上部署所面临的挑战，单纯依赖规模可能并不足够。然而，当今大多数已部署的机器人系统本质上是模块化的，并能够利用每个模块独立泛化的能力来实现高性能。因此，本文旨在通过将这些组件整合为一个整体来解决构建通用型机器人代理的问题：结合模块化与大规模学习以用于通用目的机器人的控制。 首先，我们考虑的是如何在学习系统中引入模块化和层次结构？我们的关键见解是，与其让代理人从头到尾（端到端）学会层次结构和低级控制，不如通过规划来强制执行模块化，从而实现更高效且更具能力的机器人学习者。接下来，我们要讨论规模在构建通用型机器人系统中的作用。为了达到大规模，神经网络需要大量的多样化数据、能够适应这些数据的表达性架构以及生成数据的监督源。我们利用了一个强大的监督来源：经典规划，它可以泛化，但在实际运行中代价高昂，并且需要访问特权信息才能表现良好。我们使用这些规划器在模拟环境中监督大规模策略学习，以产生通用型代理。 最后，我们要考虑如何将模块化与大规模策略学习统一起来，以构建能够在零样本操作中执行的现实世界机器人系统。为此，我们将关键成分紧密结合：模块化的高级和中级计划、学习本地控制、程序化场景生成以及用于模拟到真实环境转换的大规模策略学习。我们证明了这一配方可以产生一个单一的、通用型代理，能够解决实际环境中具有挑战性的长时间操作任务。 总的来说，本文通过结合模块化设计与大规模学习的方法来探索构建能够在多种情况下执行复杂任务的一般性机器人系统的新途径。
URL
https://arxiv.org/abs/2503.06814
PDF
-
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
2025-03-06 19:34:15Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza FathiAbstract
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.
Abstract (translated)
使用3D资产生成场景是一项复杂的挑战，需要同时具备高层次的语义理解和低层次的几何推理能力。虽然多模态大型语言模型（MLLMs）在语义任务上表现出色，但它们在应用于3D场景生成时受到对3D几何理解有限的影响。在这篇论文中，我们探讨了如何最有效地将MLLMs应用到物体放置的任务中。为此，我们引入了一个新颖的框架FirePlace，该框架利用现有的MLLMs进行： 1. 3D几何推理，并从3D场景中提取相关的几何细节； 2. 构建并解决基于提取出的低层次几何的约束条件； 3. 剪枝以获得符合常识的最终放置位置。 通过结合几何推理和MLLMs对现实世界的理解，我们的方法可以提出既满足几何约束又能考虑高层次语义常识的对象放置建议。实验表明，这些能力使我们的方法能够更有效地在具有复杂几何结构的场景中放置物体，并且其质量超过了先前的工作。
URL
https://arxiv.org/abs/2503.04919
PDF
-
DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance
2025-03-05 17:31:45Zhao Yang, Zezhong Qian, Xiaofan Li, Weixiang Xu, Gongpeng Zhao, Ruohong Yu, Lingsi Zhu, Longjun LiuAbstract
Accurate and high-fidelity driving scene reconstruction demands the effective utilization of comprehensive scene information as conditional inputs. Existing methods predominantly rely on 3D bounding boxes and BEV road maps for foreground and background control, which fail to capture the full complexity of driving scenes and adequately integrate multimodal information. In this work, we present DualDiff, a dual-branch conditional diffusion model designed to enhance driving scene generation across multiple views and video sequences. Specifically, we introduce Occupancy Ray-shape Sampling (ORS) as a conditional input, offering rich foreground and background semantics alongside 3D spatial geometry to precisely control the generation of both elements. To improve the synthesis of fine-grained foreground objects, particularly complex and distant ones, we propose a Foreground-Aware Mask (FGM) denoising loss function. Additionally, we develop the Semantic Fusion Attention (SFA) mechanism to dynamically prioritize relevant information and suppress noise, enabling more effective multimodal fusion. Finally, to ensure high-quality image-to-video generation, we introduce the Reward-Guided Diffusion (RGD) framework, which maintains global consistency and semantic coherence in generated videos. Extensive experiments demonstrate that DualDiff achieves state-of-the-art (SOTA) performance across multiple datasets. On the NuScenes dataset, DualDiff reduces the FID score by 4.09% compared to the best baseline. In downstream tasks, such as BEV segmentation, our method improves vehicle mIoU by 4.50% and road mIoU by 1.70%, while in BEV 3D object detection, the foreground mAP increases by 1.46%. Code will be made available at this https URL.
Abstract (translated)
准确且高保真的驾驶场景重建需要有效地利用全面的场景信息作为条件输入。现有的方法主要依赖于三维边界框和鸟瞰图道路地图来控制前景和背景，这无法捕捉驾驶场景的全部复杂性，并且不足以充分融合多模态信息。在本工作中，我们提出了DualDiff，这是一种双分支条件扩散模型，旨在增强从多个视角和视频序列生成驾驶场景的能力。具体来说，我们引入了Occupancy Ray-shape Sampling（ORS）作为条件输入，提供丰富的前景和背景语义以及三维空间几何信息来精确控制两者的生成。 为了改进细粒度前景对象的合成，特别是复杂且距离较远的对象，我们提出了一种基于前景的掩码去噪损失函数（FGM）。此外，我们开发了语义融合注意机制（SFA），以动态优先处理相关信息并抑制噪声，从而实现更有效的多模态融合。 最后，为了确保高质量的图像到视频生成，我们引入了引导式扩散框架（RGD），它在生成的视频中保持全局一致性和语义连贯性。广泛的实验表明，DualDiff在多个数据集上达到了最先进的性能。在NuScenes数据集上，与最佳基线相比，DualDiff将FID得分降低了4.09%。在下游任务（如鸟瞰图分割）中，我们的方法使车辆的mIoU提高了4.50%，道路的mIoU提高了1.70%，而在鸟瞰图3D对象检测中，前景的mAP增加了1.46%。 代码将在[此链接](https://this-is-the-url-to-be-filled-in-later.com)提供。
URL
https://arxiv.org/abs/2503.03689
PDF
-
A Survey on Text-Driven 360-Degree Panorama Generation
2025-02-20 18:19:57Hai Wang, Xiaoyu Xiang, Weihao Xia, Jing-Hao XueAbstract
The advent of text-driven 360-degree panorama generation, enabling the synthesis of 360-degree panoramic images directly from textual descriptions, marks a transformative advancement in immersive visual content creation. This innovation significantly simplifies the traditionally complex process of producing such content. Recent progress in text-to-image diffusion models has accelerated the rapid development in this emerging field. This survey presents a comprehensive review of text-driven 360-degree panorama generation, offering an in-depth analysis of state-of-the-art algorithms and their expanding applications in 360-degree 3D scene generation. Furthermore, we critically examine current limitations and propose promising directions for future research. A curated project page with relevant resources and research papers is available at this https URL.
Abstract (translated)
文本驱动的360度全景图像生成技术的出现，使得可以直接从文字描述中合成360度全景图，标志着沉浸式视觉内容创作领域的革命性进步。这一创新大大简化了传统上复杂的内容生产过程。最近在文本到图像扩散模型方面取得的进步加速了该新兴领域的发展。本综述对文本驱动的360度全景生成进行了全面回顾，并深入分析了当前最先进的算法及其在360度三维场景生成中的扩展应用。此外，我们还批判性地审视了现有局限性并提出了未来研究的有前景的方向。相关资源和研究论文可在以下网址（此 https URL）上查阅。
URL
https://arxiv.org/abs/2502.14799
PDF
-
Rolling Ahead Diffusion for Traffic Scene Simulation
2025-02-13 18:45:56Yunpeng Liu, Matthew Niedoba, William Harvey, Adam Scibior, Berend Zwartsenberg, Frank WoodAbstract
Realistic driving simulation requires that NPCs not only mimic natural driving behaviors but also react to the behavior of other simulated agents. Recent developments in diffusion-based scenario generation focus on creating diverse and realistic traffic scenarios by jointly modelling the motion of all the agents in the scene. However, these traffic scenarios do not react when the motion of agents deviates from their modelled trajectories. For example, the ego-agent can be controlled by a stand along motion planner. To produce reactive scenarios with joint scenario models, the model must regenerate the scenario at each timestep based on new observations in a Model Predictive Control (MPC) fashion. Although reactive, this method is time-consuming, as one complete possible future for all NPCs is generated per simulation step. Alternatively, one can utilize an autoregressive model (AR) to predict only the immediate next-step future for all NPCs. Although faster, this method lacks the capability for advanced planning. We present a rolling diffusion based traffic scene generation model which mixes the benefits of both methods by predicting the next step future and simultaneously predicting partially noised further future steps at the same time. We show that such model is efficient compared to diffusion model based AR, achieving a beneficial compromise between reactivity and computational efficiency.
Abstract (translated)
现实中的驾驶模拟要求NPC不仅模仿自然的驾驶行为，还能够对其他仿真代理的行为作出反应。近期基于扩散模型的情景生成技术着重于通过共同建模场景中所有代理的运动来创建多样且逼真的交通情景。然而，这些交通情景在代理的运动偏离其预设轨迹时并不会做出反应。例如，主控代理（ego-agent）可以由独立的路径规划器控制。为了利用联合情景模型生成响应式的情景，模型必须基于新的观察结果以模型预测控制（MPC）的方式，在每个时间步重新生成场景。尽管这种方法具备响应性，但它非常耗时，因为每模拟一步就需要为所有NPC生成一个完整的可能未来。 另一种方法是使用自回归模型（AR），仅预测所有NPC的下一步未来的运动。虽然此法较快，但缺乏对未来进行深入规划的能力。我们提出了一种滚动扩散基交通场景生成模型，该模型结合了上述两种方法的优点：它同时预测接下来一步的情景和部分模糊化后的未来步骤。我们展示了这种模型相比于基于扩散模型的自回归模型在效率上具有优势，并且能够在响应性和计算效率之间取得有利的平衡。 这种方法通过混合预测方式，即预测下一步的具体行为同时对更远期的行为做出概率性推测，从而实现了更高的灵活性和效率，在保持场景真实性的前提下提高了系统的响应速度。
URL
https://arxiv.org/abs/2502.09587
PDF
-
MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
2025-02-09 12:23:40Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, Jieyi Long, Xiangde Liu, Guangyao ZhaiAbstract
Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: this https URL.
Abstract (translated)
可控的3D场景生成在虚拟现实和室内设计等领域有着广泛的应用，生成的场景需要表现出高度的真实感与几何形状上的可控制性。场景图提供了一种合适的数据表示方式，便于实现这些应用需求。然而，当前基于图形的方法进行场景生成时，受限于文本输入，并且对于灵活多变的用户输入适应性不足，这限制了精确控制物体几何结构的能力。 为了解决这个问题，我们提出了MMGDreamer，这是一种用于场景生成的双分支扩散模型，集成了新颖的混合模态图、视觉增强模块和关系预测器。这种混合模态图允许对象节点整合文本和视觉两种模式，并支持节点间的可选关系连接。这提升了对灵活用户输入的适应性，并使得精确控制生成场景中物体的几何形状成为可能。 此外，MMGDreamer中的视觉增强模块通过使用文本嵌入来构建视觉表示，从而丰富了仅基于文本节点的视觉保真度。我们的关系预测器利用节点表示来推断节点间缺失的关系，进而产生更为连贯的场景布局。 广泛的实验结果表明，MMGDreamer在物体几何控制方面表现出色，并达到了目前最先进的场景生成性能水平。项目页面链接：[此URL](this https URL)（请替换为实际链接）。
URL
https://arxiv.org/abs/2502.05874
PDF
-
LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generation
2025-02-04 02:51:37Yang Zhou, Zongjin He, Qixuan Li, Chao WangAbstract
Recently, the field of text-guided 3D scene generation has garnered significant attention. High-quality generation that aligns with physical realism and high controllability is crucial for practical 3D scene applications. However, existing methods face fundamental limitations: (i) difficulty capturing complex relationships between multiple objects described in the text, (ii) inability to generate physically plausible scene layouts, and (iii) lack of controllability and extensibility in compositional scenes. In this paper, we introduce LayoutDreamer, a framework that leverages 3D Gaussian Splatting (3DGS) to facilitate high-quality, physically consistent compositional scene generation guided by text. Specifically, given a text prompt, we convert it into a directed scene graph and adaptively adjust the density and layout of the initial compositional 3D Gaussians. Subsequently, dynamic camera adjustments are made based on the training focal point to ensure entity-level generation quality. Finally, by extracting directed dependencies from the scene graph, we tailor physical and layout energy to ensure both realism and flexibility. Comprehensive experiments demonstrate that LayoutDreamer outperforms other compositional scene generation quality and semantic alignment methods. Specifically, it achieves state-of-the-art (SOTA) performance in the multiple objects generation metric of T3Bench.
Abstract (translated)
最近，文本引导的三维场景生成领域引起了广泛关注。高质量地生成符合物理真实性和高度可控性的三维场景对于实际应用至关重要。然而，现有的方法面临一些根本性的限制：（i）难以捕捉文本描述中多个对象之间的复杂关系；（ii）无法生成合乎物理逻辑的场景布局；（iii）在组合式场景中的可控制性与扩展性不足。为此，在本文中我们介绍了LayoutDreamer框架，该框架利用3D高斯点阵(3DGS)技术来实现高质量、物理上一致且由文本引导的组合式三维场景生成。具体而言，给定一个文本提示后，我们将它转换为有向场景图，并根据初始的组合式3D高斯分布自适应地调整其密度和布局。随后，基于训练焦点进行动态相机调整以确保实体级别的生成质量。最后，通过从场景图中提取定向依赖关系，我们定制物理能量与布局能量，以此保证现实性和灵活性。全面的实验表明，LayoutDreamer在组合式三维场景生成质量和语义对齐方法方面优于其他模型，并且在T3Bench的多对象生成度量标准上达到了最新的技术水平（SOTA）。
URL
https://arxiv.org/abs/2502.01949
PDF
-
PhiP-G: Physics-Guided Text-to-3D Compositional Scene Generation
2025-02-02 07:47:03Qixuan Li, Chao Wang, Zongjin He, Yan PengAbstract
Text-to-3D asset generation has achieved significant optimization under the supervision of 2D diffusion priors. However, when dealing with compositional scenes, existing methods encounter several challenges: 1). failure to ensure that composite scene layouts comply with physical laws; 2). difficulty in accurately capturing the assets and relationships described in complex scene descriptions; 3). limited autonomous asset generation capabilities among layout approaches leveraging large language models (LLMs). To avoid these compromises, we propose a novel framework for compositional scene generation, PhiP-G, which seamlessly integrates generation techniques with layout guidance based on a world model. Leveraging LLM-based agents, PhiP-G analyzes the complex scene description to generate a scene graph, and integrating a multimodal 2D generation agent and a 3D Gaussian generation method for targeted assets creation. For the stage of layout, PhiP-G employs a physical pool with adhesion capabilities and a visual supervision agent, forming a world model for layout prediction and planning. Extensive experiments demonstrate that PhiP-G significantly enhances the generation quality and physical rationality of the compositional scenes. Notably, PhiP-G attains state-of-the-art (SOTA) performance in CLIP scores, achieves parity with the leading methods in generation quality as measured by the T$^3$Bench, and improves efficiency by 24x.
Abstract (translated)
在二维扩散先验的监督下，文本到三维资产生成已经实现了显著优化。然而，在处理组合场景时，现有方法面临以下挑战：1) 无法确保复合场景布局符合物理规律；2) 难以准确捕捉复杂场景描述中的资产和关系；3) 利用大型语言模型（LLMs）的布局方法在自主资产生成能力方面存在局限。为避免这些妥协，我们提出了一种新颖的框架PhiP-G，该框架无缝地将生成技术与基于世界模型的布局指导相结合。 通过利用基于大语言模型的代理，PhiP-G 分析复杂场景描述来生成场景图，并整合多模态二维生成代理和三维高斯生成方法以创建特定资产。在布局阶段，PhiP-G 采用具有粘合能力的物理池和视觉监督代理，形成一个世界模型用于布局预测和规划。 广泛的实验表明，PhiP-G 显著提升了组合场景生成的质量和物理合理性。特别地，PhiP-G 在CLIP评分中达到最先进的（SOTA）性能，在T$^3$Bench 测量的生成质量上与领先方法持平，并且在效率方面提高了24倍。
URL
https://arxiv.org/abs/2502.00708
PDF
-
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
2025-01-24 18:59:51Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, Xiang BaiAbstract
Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model (LLM), enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at this https URL.
Abstract (translated)
驾驶世界模型（DWMs）已成为自动驾驶不可或缺的一部分，通过实现未来场景预测来支持自动驾驶。然而，现有的DWM仅限于生成新场景，并未能整合对驾驶环境的理解，即无法解读和推理驾驶环境中的情况。在本文中，我们提出了一种统一的驾驶世界模型HERMES。我们在驾驶场景下通过一个统一框架无缝地集成了三维场景理解和未来场景演变（生成）。具体而言，HERMES利用鸟瞰图（BEV）表示来整合多视角空间信息的同时保持几何关系和互动性。此外，我们还引入了世界查询机制，该机制通过在大型语言模型中应用因果注意力将世界知识融入到BEV特征中，从而为理解和生成任务提供上下文增强。我们在nuScenes和OmniDrive-nuScenes数据集上进行了全面的研究以验证我们的方法的有效性。HERMES取得了最先进的性能，在减少生成误差方面比现有技术降低了32.4%，在理解指标如CIDEr的评估中提高了8.0%。该模型及其代码将在[此链接](https://this https URL)公开发布。
URL
https://arxiv.org/abs/2501.14729
PDF
