Scene_Generation
-
Recent Advance in 3D Object and Scene Generation: A Survey
2025-04-16 03:22:06Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In recent years, the demand for 3D content has grown exponentially with intelligent upgrading of interactive media, extended reality (XR), and Metaverse industries. In order to overcome the limitation of traditional manual modeling approaches, such as labor-intensive workflows and prolonged production cycles, revolutionary advances have been achieved through the convergence of novel 3D representation paradigms and artificial intelligence generative technologies. In this survey, we conduct a systematically review of the cutting-edge achievements in static 3D object and scene generation, as well as establish a comprehensive technical framework through systematic categorization. Specifically, we initiate our analysis with mainstream 3D object representations, followed by in-depth exploration of two principal technical pathways in object generation: data-driven supervised learning methods and deep generative model-based approaches. Regarding scene generation, we focus on three dominant paradigms: layout-guided compositional synthesis, 2D prior-based scene generation, and rule-driven modeling. Finally, we critically examine persistent challenges in 3D generation and propose potential research directions for future investigation. This survey aims to provide readers with a structured understanding of state-of-the-art 3D generation technologies while inspiring researchers to undertake more exploration in this domain.
Abstract (translated)
近年来，随着交互式媒体、扩展现实（XR）和元宇宙行业的智能化升级，对三维内容的需求呈指数级增长。为了克服传统手动建模方法的限制，如劳动密集型工作流程和延长的生产周期，通过新型三维表示范例与人工智能生成技术的融合，取得了革命性的进展。在本次综述中，我们系统地回顾了静态3D物体和场景生成领域的最新成就，并通过系统的分类建立了全面的技术框架。具体而言，我们的分析始于主流的3D对象表示形式，随后深入探讨两个主要的对象生成技术路径：数据驱动监督学习方法和基于深度生成模型的方法。对于场景生成，我们重点关注三个主导范式：布局引导组合合成、基于2D先验的场景生成以及规则驱动建模。最后，我们批判性地审视了3D生成中持续存在的挑战，并提出了未来研究方向的可能性。本次综述旨在为读者提供对最新3D生成技术结构化理解的同时，激励研究人员在此领域进行更多探索。
URL
https://arxiv.org/abs/2504.11734
PDF
-
Decoupled Diffusion Sparks Adaptive Scene Generation
2025-04-14 17:59:57Yunsong Zhou, Naisheng Ye, William Ljungbergh, Tianyu Li, Jiazhi Yang, Zetong Yang, Hongzi Zhu, Christoffer Petersson, Hongyang LiAbstract
Controllable scene generation could reduce the cost of diverse data collection substantially for autonomous driving. Prior works formulate the traffic layout generation as predictive progress, either by denoising entire sequences at once or by iteratively predicting the next frame. However, full sequence denoising hinders online reaction, while the latter's short-sighted next-frame prediction lacks precise goal-state guidance. Further, the learned model struggles to generate complex or challenging scenarios due to a large number of safe and ordinal driving behaviors from open datasets. To overcome these, we introduce Nexus, a decoupled scene generation framework that improves reactivity and goal conditioning by simulating both ordinal and challenging scenarios from fine-grained tokens with independent noise states. At the core of the decoupled pipeline is the integration of a partial noise-masking training strategy and a noise-aware schedule that ensures timely environmental updates throughout the denoising process. To complement challenging scenario generation, we collect a dataset consisting of complex corner cases. It covers 540 hours of simulated data, including high-risk interactions such as cut-in, sudden braking, and collision. Nexus achieves superior generation realism while preserving reactivity and goal orientation, with a 40% reduction in displacement error. We further demonstrate that Nexus improves closed-loop planning by 20% through data augmentation and showcase its capability in safety-critical data generation.
Abstract (translated)
可控场景生成可以大幅降低自动驾驶中多样化数据收集的成本。先前的研究将交通布局的生成视为预测性进展，要么一次去除整个序列中的噪声，要么通过迭代预测下一帧来实现。然而，一次性全序列去噪会阻碍在线反应能力，而后者仅基于下一帧的短期预测又缺乏精确的目标状态指导。此外，由于开放数据集中存在大量安全和常规驾驶行为，学习模型难以生成复杂或具有挑战性的场景。 为了克服这些问题，我们引入了Nexus框架，这是一个解耦的场景生成框架，通过模拟带有独立噪声状态的细粒度令牌，来改善反应性和目标导向性，同时可以生成正常情况及有挑战性的场景。该框架的核心在于集成部分噪声屏蔽训练策略和感知噪声的时间表安排，以确保在整个去噪过程中及时更新环境。 为了补充对具有挑战性场景的生成，我们收集了一个包含复杂边缘案例的数据集，其中包括540小时模拟数据（如切入、突然刹车和碰撞等高风险互动）。Nexus在保持反应性和目标导向的同时实现了更真实的场景生成，并将位移误差减少了40%。此外，我们还展示了通过数据增强方法来提升闭环规划的20%，并证明了其在安全关键性数据生成方面的能力。
URL
https://arxiv.org/abs/2504.10485
PDF
-
GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting
2025-04-14 09:04:01Junlin Hao, Peiheng Wang, Haoyang Wang, Xinggong Zhang, Zongming GuoAbstract
Single-image 3D scene reconstruction presents significant challenges due to its inherently ill-posed nature and limited input constraints. Recent advances have explored two promising directions: multiview generative models that train on 3D consistent datasets but struggle with out-of-distribution generalization, and 3D scene inpainting and completion frameworks that suffer from cross-view inconsistency and suboptimal error handling, as they depend exclusively on depth data or 3D smoothness, which ultimately degrades output quality and computational performance. Building upon these approaches, we present GaussVideoDreamer, which advances generative multimedia approaches by bridging the gap between image, video, and 3D generation, integrating their strengths through two key innovations: (1) A progressive video inpainting strategy that harnesses temporal coherence for improved multiview consistency and faster convergence. (2) A 3D Gaussian Splatting consistency mask to guide the video diffusion with 3D consistent multiview evidence. Our pipeline combines three core components: a geometry-aware initialization protocol, Inconsistency-Aware Gaussian Splatting, and a progressive video inpainting strategy. Experimental results demonstrate that our approach achieves 32% higher LLaVA-IQA scores and at least 2x speedup compared to existing methods while maintaining robust performance across diverse scenes.
Abstract (translated)
单张图像的三维场景重建面临巨大的挑战，由于其本质上是一个病态问题，并且输入信息有限。最近的研究主要探索了两个有前景的方向：多视图生成模型训练在3D一致的数据集上，但难以处理出分布数据的一般化；以及依赖于深度数据或三维平滑性的3D场景修复和补全框架，在跨视角一致性方面存在不足，并且误差处理欠佳，最终导致输出质量降低及计算性能下降。基于这些方法的进展，我们提出了GaussVideoDreamer，该模型通过连接图像、视频与三维生成之间的差距，利用两个关键创新来推动生成多媒体的方法：（1）渐进式视频修复策略，借助时间一致性实现多视图一致性的改进和更快收敛。（2）3D高斯点阵一致性掩模，为视频扩散过程提供基于3D一致性多视角证据的指导。我们的流程整合了三个核心组件：几何感知初始化协议、跨一致性意识高斯点阵以及渐进式视频修复策略。实验结果表明，相较于现有的方法，我们提出的方法在LLaVA-IQA评分上提高了32%，同时至少提升了两倍的速度，并保持了在各种场景中的稳定性能。
URL
https://arxiv.org/abs/2504.10001
PDF
-
Text To 3D Object Generation For Scalable Room Assembly
2025-04-12 20:13:07Sonia Laguna, Alberto Garcia-Garcia, Marie-Julie Rakotosaona, Stylianos Moschoglou, Leonhard Helminger, Sergio Orts-EscolanoAbstract
Modern machine learning models for scene understanding, such as depth estimation and object tracking, rely on large, high-quality datasets that mimic real-world deployment scenarios. To address data scarcity, we propose an end-to-end system for synthetic data generation for scalable, high-quality, and customizable 3D indoor scenes. By integrating and adapting text-to-image and multi-view diffusion models with Neural Radiance Field-based meshing, this system generates highfidelity 3D object assets from text prompts and incorporates them into pre-defined floor plans using a rendering tool. By introducing novel loss functions and training strategies into existing methods, the system supports on-demand scene generation, aiming to alleviate the scarcity of current available data, generally manually crafted by artists. This system advances the role of synthetic data in addressing machine learning training limitations, enabling more robust and generalizable models for real-world applications.
Abstract (translated)
现代的机器学习模型，如场景理解中的深度估计和对象跟踪，依赖于大量高质量的数据集，这些数据集能够模拟实际部署环境。为了解决数据稀缺的问题，我们提出了一种端到端系统，用于生成可扩展、高质量且可定制的3D室内场景的合成数据。通过将文本到图像和多视角扩散模型与基于神经辐射场(meshing)的技术集成并适应，该系统可以从文本提示中生成高保真的3D对象资产，并利用渲染工具将其整合进预定义的平面图中。通过对现有方法引入新的损失函数和训练策略，此系统支持按需场景生成，旨在缓解目前由艺术家手工制作的数据稀缺问题。该系统推进了合成数据在解决机器学习训练限制中的作用，使得为实际应用开发出更稳健且更具泛化能力的模型成为可能。
URL
https://arxiv.org/abs/2504.09328
PDF
-
EP-Diffuser: An Efficient Diffusion Model for Traffic Scene Generation and Prediction via Polynomial Representations
2025-04-07 18:45:49Yue Yao, Mohamed-Khalil Bouzidi, Daniel Goehring, Joerg ReichardtAbstract
As the prediction horizon increases, predicting the future evolution of traffic scenes becomes increasingly difficult due to the multi-modal nature of agent motion. Most state-of-the-art (SotA) prediction models primarily focus on forecasting the most likely future. However, for the safe operation of autonomous vehicles, it is equally important to cover the distribution for plausible motion alternatives. To address this, we introduce EP-Diffuser, a novel parameter-efficient diffusion-based generative model designed to capture the distribution of possible traffic scene evolutions. Conditioned on road layout and agent history, our model acts as a predictor and generates diverse, plausible scene continuations. We benchmark EP-Diffuser against two SotA models in terms of accuracy and plausibility of predictions on the Argoverse 2 dataset. Despite its significantly smaller model size, our approach achieves both highly accurate and plausible traffic scene predictions. We further evaluate model generalization ability in an out-of-distribution (OoD) test setting using Waymo Open dataset and show superior robustness of our approach. The code and model checkpoints can be found here: this https URL.
Abstract (translated)
随着预测时间范围的增加，由于代理运动的多模态特性，对未来交通场景演化的预测难度也在增大。目前最先进的（SotA）预测模型主要集中在预测最有可能的未来情景上。然而，对于自主车辆的安全操作而言，涵盖可能的替代运动方案的概率分布同样重要。为了解决这个问题，我们引入了EP-Diffuser，这是一种新颖的参数高效扩散生成模型，旨在捕捉交通场景演化的各种可能性分布。基于道路布局和代理历史信息，我们的模型作为预测器工作，并产生多样且合理的场景延续。 我们在Argoverse 2数据集上对EP-Diffuser与两个SotA模型在预测准确性和合理性方面进行了基准测试。尽管模型大小显著较小，但我们的方法实现了高度精确且合理的交通场景预测。此外，我们使用Waymo Open数据集在外分布（OoD）测试设置中评估了模型的泛化能力，并展示了我们方法的优越鲁棒性。 代码和模型检查点可以在以下链接找到：[此URL](this https URL)。
URL
https://arxiv.org/abs/2504.05422
PDF
-
PanoDreamer: Consistent Text to 360-Degree Scene Generation
2025-04-07 14:57:01Zhexiao Xiong, Zhang Chen, Zhong Li, Yi Xu, Nathan JacobsAbstract
Automatically generating a complete 3D scene from a text description, a reference image, or both has significant applications in fields like virtual reality and gaming. However, current methods often generate low-quality textures and inconsistent 3D structures. This is especially true when extrapolating significantly beyond the field of view of the reference image. To address these challenges, we propose PanoDreamer, a novel framework for consistent, 3D scene generation with flexible text and image control. Our approach employs a large language model and a warp-refine pipeline, first generating an initial set of images and then compositing them into a 360-degree panorama. This panorama is then lifted into 3D to form an initial point cloud. We then use several approaches to generate additional images, from different viewpoints, that are consistent with the initial point cloud and expand/refine the initial point cloud. Given the resulting set of images, we utilize 3D Gaussian Splatting to create the final 3D scene, which can then be rendered from different viewpoints. Experiments demonstrate the effectiveness of PanoDreamer in generating high-quality, geometrically consistent 3D scenes.
Abstract (translated)
从文本描述、参考图像或两者结合自动生成完整的三维场景，在虚拟现实和游戏等领域具有重要的应用价值。然而，目前的方法往往生成质量较低的纹理，并且在构建不一致的三维结构时尤其如此，尤其是在超出参考图像视野范围的情况下。为了解决这些问题，我们提出了PanoDreamer，这是一种新颖的框架，用于通过灵活的文字和图像控制来生成一致性的三维场景。 我们的方法采用大型语言模型和一种扭曲-细化管道技术：首先生成一系列初始图像，然后将这些图像组合成360度全景图。接着，我们将该全景图提升至三维空间以形成初步点云。随后，我们使用几种不同的方法从不同视角生成额外的图像，并确保这些新生成的图像与初步点云保持一致，同时扩展和细化初步点云。 基于最终的一系列图像，我们利用3D高斯光束技术来创建最终的三维场景，该场景可以从多个视角进行渲染。实验结果表明，PanoDreamer在生成高质量且几何上一致的三维场景方面具有显著的效果。
URL
https://arxiv.org/abs/2504.05152
PDF
-
Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model
2025-04-03 17:00:44Shengjun Zhang, Jinzhao Li, Xin Fei, Hao Liu, Yueqi DuanAbstract
In this paper, we propose Scene Splatter, a momentum-based paradigm for video diffusion to generate generic scenes from single image. Existing methods, which employ video generation models to synthesize novel views, suffer from limited video length and scene inconsistency, leading to artifacts and distortions during further reconstruction. To address this issue, we construct noisy samples from original features as momentum to enhance video details and maintain scene consistency. However, for latent features with the perception field that spans both known and unknown regions, such latent-level momentum restricts the generative ability of video diffusion in unknown regions. Therefore, we further introduce the aforementioned consistent video as a pixel-level momentum to a directly generated video without momentum for better recovery of unseen regions. Our cascaded momentum enables video diffusion models to generate both high-fidelity and consistent novel views. We further finetune the global Gaussian representations with enhanced frames and render new frames for momentum update in the next step. In this manner, we can iteratively recover a 3D scene, avoiding the limitation of video length. Extensive experiments demonstrate the generalization capability and superior performance of our method in high-fidelity and consistent scene generation.
Abstract (translated)
在这篇论文中，我们提出了一种名为Scene Splatter的基于动量的范式，用于从单张图像生成通用场景。现有方法使用视频生成模型来合成新颖视图时，会受到视频长度有限和场景不一致的影响，从而导致在进一步重建过程中出现伪影和失真。为了解决这个问题，我们构建了以原始特征为基础的噪声样本作为动量，以此增强视频细节并保持场景一致性。然而，在具有感知范围涵盖已知区域与未知区域的潜在特征的情况下，这种潜在层次上的动量限制了视频扩散模型在未知区域内生成的能力。因此，我们进一步将前述一致性的视频引入为像素级别的动量，以改善没有使用动量直接生成的视频中未见区域的恢复效果。我们的级联动量使视频扩散模型能够生成既高保真又具有一致性的新颖视图。此外，我们将全局高斯表示进行精细调整，并利用增强后的帧渲染新帧来更新下一步中的动量。通过这种方式，我们可以迭代地恢复一个3D场景，避免了视频长度的限制。广泛的实验展示了我们方法在生成高质量和一致场景方面的泛化能力和卓越性能。
URL
https://arxiv.org/abs/2504.02764
PDF
-
LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images
2025-04-03 07:18:48Ming-Jia Yang, Yu-Xiao Guo, Yang Liu, Bin Zhou, Xin TongAbstract
Generating realistic, room-level indoor scenes with semantically plausible and detailed appearances from in-the-wild images is crucial for various applications in VR, AR, and robotics. The success of NeRF-based generative methods indicates a promising direction to address this challenge. However, unlike their success at the object level, existing scene-level generative methods require additional information, such as multiple views, depth images, or semantic guidance, rather than relying solely on RGB images. This is because NeRF-based methods necessitate prior knowledge of camera poses, which is challenging to approximate for indoor scenes due to the complexity of defining alignment and the difficulty of globally estimating poses from a single image, given the unseen parts behind the camera. To address this challenge, we redefine global poses within the framework of Local-Pose-Alignment (LPA) -- an anchor-based multi-local-coordinate system that uses a selected number of anchors as the roots of these coordinates. Building on this foundation, we introduce LPA-GAN, a novel NeRF-based generative approach that incorporates specific modifications to estimate the priors of camera poses under LPA. It also co-optimizes the pose predictor and scene generation processes. Our ablation study and comparisons with straightforward extensions of NeRF-based object generative methods demonstrate the effectiveness of our approach. Furthermore, visual comparisons with other techniques reveal that our method achieves superior view-to-view consistency and semantic normality.
Abstract (translated)
从野外图像生成具有语义合理且细节丰富的房间级别室内场景对于VR、AR和机器人技术等多种应用至关重要。基于NeRF（神经辐射场）的生成方法的成功表明，这是一个很有前景的方向来解决这一挑战。然而，与它们在物体级别的成功不同，现有的场景级生成方法需要额外的信息，例如多视角图像、深度图或语义指导，而不仅仅是依赖于RGB图像。这是因为基于NeRF的方法需要相机姿态的先验知识，而这对于室内场景来说是具有挑战性的，因为定义对齐的复杂性以及从单张图像全局估计姿态的难度（尤其是考虑到摄像机后方未见的部分）。为了解决这一挑战，我们重新定义了全局姿态，并在局部姿态对齐（Local-Pose-Alignment, LPA）框架内进行操作。这是一个基于锚点的多本地坐标系统，它使用选定数量的锚点作为这些坐标的根。 在此基础上，我们引入了一种新的NeRF生成方法——LPA-GAN，该方法在LPA框架下专门进行了修改以估计相机姿态的先验知识，并且优化了姿态预测和场景生成过程。我们的消融研究以及与基于NeRF的对象生成方法的直接扩展进行比较表明了这种方法的有效性。此外，与其它技术进行的视觉对比显示，我们的方法实现了更好的视图间一致性及语义合理性。 通过这种创新的方法，LPA-GAN能够在仅依赖于RGB图像的情况下有效地生成室内场景，并且在保持高真实性和细节的同时解决了姿态估计的问题，从而为虚拟现实和增强现实中高质量室内外环境建模提供了新的可能。
URL
https://arxiv.org/abs/2504.02337
PDF
-
WonderTurbo: Generating Interactive 3D World in 0.72 Seconds
2025-04-03 04:10:47Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Weijie Wang, Haoyun Li, Guosheng Zhao, Jie Li, Wenkang Qin, Guan Huang, Wenjun MeiAbstract
Interactive 3D generation is gaining momentum and capturing extensive attention for its potential to create immersive virtual experiences. However, a critical challenge in current 3D generation technologies lies in achieving real-time interactivity. To address this issue, we introduce WonderTurbo, the first real-time interactive 3D scene generation framework capable of generating novel perspectives of 3D scenes within 0.72 seconds. Specifically, WonderTurbo accelerates both geometric and appearance modeling in 3D scene generation. In terms of geometry, we propose StepSplat, an innovative method that constructs efficient 3D geometric representations through dynamic updates, each taking only 0.26 seconds. Additionally, we design QuickDepth, a lightweight depth completion module that provides consistent depth input for StepSplat, further enhancing geometric accuracy. For appearance modeling, we develop FastPaint, a 2-steps diffusion model tailored for instant inpainting, which focuses on maintaining spatial appearance consistency. Experimental results demonstrate that WonderTurbo achieves a remarkable 15X speedup compared to baseline methods, while preserving excellent spatial consistency and delivering high-quality output.
Abstract (translated)
交互式3D生成技术正逐渐兴起，并因其创造沉浸式虚拟体验的潜力而备受关注。然而，当前3D生成技术的一个关键挑战在于实现实时互动性。为了解决这一问题，我们推出了WonderTurbo——首个能够实现实时交互式3D场景生成框架，它能够在0.72秒内生成新的3D场景视角。具体而言，WonderTurbo通过加速几何和外观建模来提高3D场景生成的速度。 在几何方面，我们提出了StepSplat方法，这是一种创新的动态更新技术，能够构建高效的3D几何表示，并且每次仅需花费0.26秒。此外，我们设计了QuickDepth轻量级深度完成模块，为StepSplat提供了一致性的深度输入，从而进一步提高几何精度。 在外观建模方面，我们开发了FastPaint——一个专为即时修复绘制而定制的两步扩散模型，专注于保持空间外观的一致性。实验结果表明，与基准方法相比，WonderTurbo实现了15倍的速度提升，并且能够维持出色的空间一致性并输出高质量的结果。
URL
https://arxiv.org/abs/2504.02261
PDF
-
WorldPrompter: Traversable Text-to-Scene Generation
2025-04-02 18:04:32Zhaoyang Zhang, Yannick Hold-Geoffroy, Milo\v{s} Ha\v{s}an, Chen Ziwen, Fujun Luan, Julie Dorsey, Yiwei HuAbstract
Scene-level 3D generation is a challenging research topic, with most existing methods generating only partial scenes and offering limited navigational freedom. We introduce WorldPrompter, a novel generative pipeline for synthesizing traversable 3D scenes from text prompts. We leverage panoramic videos as an intermediate representation to model the 360° details of a scene. WorldPrompter incorporates a conditional 360° panoramic video generator, capable of producing a 128-frame video that simulates a person walking through and capturing a virtual environment. The resulting video is then reconstructed as Gaussian splats by a fast feedforward 3D reconstructor, enabling a true walkable experience within the 3D scene. Experiments demonstrate that our panoramic video generation model achieves convincing view consistency across frames, enabling high-quality panoramic Gaussian splat reconstruction and facilitating traversal over an area of the scene. Qualitative and quantitative results also show it outperforms the state-of-the-art 360° video generators and 3D scene generation models.
Abstract (translated)
场景级别的三维生成是一个具有挑战性的研究课题，大多数现有的方法只能生成部分场景，并且提供有限的导航自由度。我们引入了WorldPrompter，这是一种新颖的生成管道，可以从文本提示中合成可穿越的三维场景。我们利用全景视频作为中间表示来建模场景的360°细节。 WorldPrompter 包含了一个条件性的 360° 全景视频生成器，能够产生一个模拟人在虚拟环境中行走并捕捉环境的128帧视频。然后，通过快速前馈式的三维重建算法将产生的视频重构为高斯点（Gaussian splats），从而在三维场景中实现真正的可行走体验。 实验表明，我们的全景视频生成模型实现了跨帧令人信服的视角一致性，这使得高质量的全景高斯点重建成为可能，并且支持穿越场景中的大片区域。定性和定量的结果也显示它超越了现有的最先进的 360° 视频生成器和三维场景生成模型的表现。 该研究通过结合文本驱动的视频生成与高效的 3D 场景重构技术，为实现高度交互式的虚拟现实体验提供了新的途径。
URL
https://arxiv.org/abs/2504.02045
PDF
-
VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
2025-04-02 17:59:21Hanyang Wang, Fangfu Liu, Jiawei Chi, Yueqi DuanAbstract
Recovering 3D scenes from sparse views is a challenging task due to its inherent ill-posed problem. Conventional methods have developed specialized solutions (e.g., geometry regularization or feed-forward deterministic model) to mitigate the issue. However, they still suffer from performance degradation by minimal overlap across input views with insufficient visual information. Fortunately, recent video generative models show promise in addressing this challenge as they are capable of generating video clips with plausible 3D structures. Powered by large pretrained video diffusion models, some pioneering research start to explore the potential of video generative prior and create 3D scenes from sparse views. Despite impressive improvements, they are limited by slow inference time and the lack of 3D constraint, leading to inefficiencies and reconstruction artifacts that do not align with real-world geometry structure. In this paper, we propose VideoScene to distill the video diffusion model to generate 3D scenes in one step, aiming to build an efficient and effective tool to bridge the gap from video to 3D. Specifically, we design a 3D-aware leap flow distillation strategy to leap over time-consuming redundant information and train a dynamic denoising policy network to adaptively determine the optimal leap timestep during inference. Extensive experiments demonstrate that our VideoScene achieves faster and superior 3D scene generation results than previous video diffusion models, highlighting its potential as an efficient tool for future video to 3D applications. Project Page: this https URL
Abstract (translated)
从稀疏视角恢复三维场景是一个极具挑战性的任务，因为其本质上是不适定问题。传统的方法开发了专门的解决方案（例如几何正则化或前馈确定性模型）来缓解这些问题。然而，它们仍然在输入视图之间最小重叠且视觉信息不足的情况下遭受性能下降的问题。幸运的是，最近的视频生成模型显示出解决这一挑战的潜力，因为它们能够生成具有合理三维结构的视频片段。借助大规模预训练的视频扩散模型，一些开创性的研究开始探索视频生成先验的潜力，并从稀疏视图中创建三维场景。尽管取得了令人印象深刻的进步，但这些方法受限于推理时间慢以及缺乏三维约束条件，导致效率低下和重建过程中出现与现实世界几何结构不符的艺术瑕疵。 在本文中，我们提出了VideoScene框架，旨在将视频扩散模型精简为一步生成三维场景的方法，目标是建立一个高效且有效的工具，以弥合从视频到三维的差距。具体来说，我们设计了一个3D感知跳跃流精炼策略来跳过耗时冗余信息，并训练动态去噪策略网络在推理过程中自适应地确定最佳跳跃时间步长。 大量的实验表明，我们的VideoScene方法比之前的视频扩散模型更快且更有效地生成三维场景结果，这突显了其作为未来视频到3D应用高效工具的潜力。项目页面：[此URL](https://this-url.com/)（原文中的链接地址请替换为实际链接）。
URL
https://arxiv.org/abs/2504.01956
PDF
-
WorldScore: A Unified Evaluation Benchmark for World Generation
2025-04-01 17:20:23Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun WuAbstract
We introduce the WorldScore benchmark, the first unified benchmark for world generation. We decompose world generation into a sequence of next-scene generation tasks with explicit camera trajectory-based layout specifications, enabling unified evaluation of diverse approaches from 3D and 4D scene generation to video generation models. The WorldScore benchmark encompasses a curated dataset of 3,000 test examples that span diverse worlds: static and dynamic, indoor and outdoor, photorealistic and stylized. The WorldScore metrics evaluate generated worlds through three key aspects: controllability, quality, and dynamics. Through extensive evaluation of 19 representative models, including both open-source and closed-source ones, we reveal key insights and challenges for each category of models. Our dataset, evaluation code, and leaderboard can be found at this https URL
Abstract (translated)
我们介绍了WorldScore基准测试，这是第一个用于世界生成的统一基准。我们将世界生成分解为一系列基于明确摄像机轨迹布局规范的下一场景生成任务，从而能够对从3D和4D场景生成到视频生成模型等多样方法进行统一评估。WorldScore基准测试包括一个精心策划的数据集，包含3,000个测试示例，涵盖了多样的世界：静态与动态、室内与室外、写实主义和风格化。 WorldScore指标通过三个关键方面来评价生成的世界：可控性、质量和动态特性。通过对19种代表性模型（包括开源和闭源模型）进行广泛的评估，我们揭示了每类模型的关键见解和挑战。我们的数据集、评估代码及排行榜可在此链接获取：[https URL]
URL
https://arxiv.org/abs/2504.00983
PDF
-
Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
2025-03-28 15:03:41Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, Felix HeideAbstract
We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene - comprising a lane graph and agent bounding boxes - and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead employs a novel vectorized latent diffusion model for initial scene generation that directly operates on the vectorized scene elements and an autoregressive Transformer for data-driven agent behaviour simulation. Scenario Dreamer additionally supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. Extensive experiments show that Scenario Dreamer outperforms existing generative simulators in realism and efficiency: the vectorized scene-generation base model achieves superior generation quality with around 2x fewer parameters, 6x lower generation latency, and 10x fewer GPU training hours compared to the strongest baseline. We confirm its practical utility by showing that reinforcement learning planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments.
Abstract (translated)
我们介绍了Scenario Dreamer，这是一种完全基于数据驱动的自动驾驶车辆规划生成模拟器。它可以生成初始交通场景（包括车道图和代理边框）以及闭环代理行为。现有的用于生成驾驶仿真环境的方法将初始交通场景编码为栅格化图像，并因此需要参数繁重的网络来执行不必要的计算，因为栅格化场景中有许多空白像素。此外，我们发现使用基于规则的代理行为方法缺乏多样性和现实性。 与现有方法不同，Scenario Dreamer采用了一种新颖的向量化潜在扩散模型进行初始场景生成，该模型直接操作向量化的场景元素，并且使用自回归Transformer来进行数据驱动的代理行为模拟。此外，Scenario Dreamer通过扩散填补技术支持场景扩展，从而能够生成不受限的仿真环境。 大量的实验表明，与现有的生成式模拟器相比，Scenario Dreamer在真实性和效率方面表现更佳：向量化场景生成基础模型实现了更高的生成质量，参数数量减少了大约2倍，生成延迟降低了6倍，并且GPU训练时间也减少了10倍。我们通过展示强化学习规划代理在Scenario Dreamer环境中面临更大挑战的事实来确认其实际效用，特别是在长时间和对抗性驾驶环境中的表现优于传统的非生成式仿真环境。
URL
https://arxiv.org/abs/2503.22496
PDF
-
Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency
2025-03-26 17:59:44Tianqi Liu, Zihao Huang, Zhaoxi Chen, Guangcong Wang, Shoukang Hu, Liao Shen, Huiqiang Sun, Zhiguo Cao, Wei Li, Ziwei LiuAbstract
We present Free4D, a novel tuning-free framework for 4D scene generation from a single image. Existing methods either focus on object-level generation, making scene-level generation infeasible, or rely on large-scale multi-view video datasets for expensive training, with limited generalization ability due to the scarcity of 4D scene data. In contrast, our key insight is to distill pre-trained foundation models for consistent 4D scene representation, which offers promising advantages such as efficiency and generalizability. 1) To achieve this, we first animate the input image using image-to-video diffusion models followed by 4D geometric structure initialization. 2) To turn this coarse structure into spatial-temporal consistent multiview videos, we design an adaptive guidance mechanism with a point-guided denoising strategy for spatial consistency and a novel latent replacement strategy for temporal coherence. 3) To lift these generated observations into consistent 4D representation, we propose a modulation-based refinement to mitigate inconsistencies while fully leveraging the generated information. The resulting 4D representation enables real-time, controllable rendering, marking a significant advancement in single-image-based 4D scene generation.
Abstract (translated)
我们提出了一种名为Free4D的新框架，该框架无需调优即可从单张图像生成4D场景。现有方法要么专注于对象级别的生成，使得场景级别的生成变得不可行；要么依赖于大规模的多视角视频数据集进行昂贵的训练，并且由于缺乏4D场景数据而具有有限的泛化能力。相比之下，我们的关键洞察是通过蒸馏预训练的基础模型来实现一致的4D场景表示，这种方法提供了诸如效率和泛化能力等显著优势。 为了实现这一目标： 1. 我们首先使用图像到视频的扩散模型对输入图像进行动画处理，并随后采用4D几何结构初始化。 2. 为将这种粗略结构转变为时空一致的多视角视频，我们设计了一种自适应引导机制，包括用于空间一致性的一点引导去噪策略和一种新颖的潜在替换策略以实现时间连贯性。 3. 要将这些生成的观察提升到一致的4D表示中，我们提出了一种基于调制的细化方法，通过这种方式可以缓解不一致性，并充分利用生成的信息。 最终得到的4D表示能够实现实时、可控渲染，在单张图像基础上进行4D场景生成方面取得了重大进展。
URL
https://arxiv.org/abs/2503.20785
PDF
-
Global-Local Tree Search for Language Guided 3D Scene Generation
2025-03-24 09:21:13Wei Deng, Mengshi Qi, Huadong MaAbstract
Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at this https URL .
Abstract (translated)
大型视觉-语言模型（VLMs），如GPT-4，在多个领域取得了显著的成功。然而，关于使用VLM生成三维室内场景的研究相对较少。本文将这一任务视为一个受空间和布局常识约束的规划问题，并提出了一种新的全局局部树搜索算法来解决该问题。 在全局层面上，本方法依次放置每个对象并探索每次放置过程中的多种放置方式。为了减少问题空间表示为树结构的深度，我们将场景结构分层分解，即按房间层级、区域层级、楼层物体层级和支持物体层级进行拆解。算法独立生成不同区域中的地面物体和不同地面物体上的支撑物。 在局部层面，我们将每个对象的放置任务进一步细分为多个步骤，并通过探索问题空间的树形图来搜索最优解。为了利用VLM模型产生物体的位置信息，我们把俯视视角的空间离散化为密集网格，并向每个格子填充多样化的表情符号以使其独特可辨认。然后用带有表情符号网格的提示询问VLM，从而得到由表情名称描述的一个合理位置。 通过量化和定性的实验结果表明，我们的方法能够生成比现有最新技术更合理的三维场景。我们的源代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2503.18476
PDF
-
MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
2025-03-24 09:18:01Zhenyu Pan, Han LiuAbstract
We present MetaSpatial, the first reinforcement learning (RL)-based framework designed to enhance 3D spatial reasoning in vision-language models (VLMs), enabling real-time 3D scene generation without the need for hard-coded optimizations. MetaSpatial addresses two core challenges: (i) the lack of internalized 3D spatial reasoning in VLMs, which limits their ability to generate realistic layouts, and (ii) the inefficiency of traditional supervised fine-tuning (SFT) for layout generation tasks, as perfect ground truth annotations are unavailable. Our key innovation is a multi-turn RL-based optimization mechanism that integrates physics-aware constraints and rendered image evaluations, ensuring generated 3D layouts are coherent, physically plausible, and aesthetically consistent. Methodologically, MetaSpatial introduces an adaptive, iterative reasoning process, where the VLM refines spatial arrangements over multiple turns by analyzing rendered outputs, improving scene coherence progressively. Empirical evaluations demonstrate that MetaSpatial significantly enhances the spatial consistency and formatting stability of various scale models. Post-training, object placements are more realistic, aligned, and functionally coherent, validating the effectiveness of RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game development applications. Our code, data, and training pipeline are publicly available at this https URL.
Abstract (translated)
我们介绍了MetaSpatial，这是一个基于强化学习（RL）的框架，旨在增强视觉语言模型（VLMs）中的3D空间推理能力，使其能够在无需硬编码优化的情况下实现实时3D场景生成。MetaSpatial解决了两个核心挑战：(i) VLM中缺乏内在化的3D空间推理能力，这限制了它们生成逼真布局的能力；(ii) 传统监督微调（SFT）方法在布局生成任务中的低效性，因为无法获得完美的地面真实标注。 我们的关键创新是一个基于多轮RL的优化机制，该机制集成了物理感知约束和渲染图像评估，确保生成的3D布局具有连贯性、物理合理性及美学一致性。从方法论的角度来看，MetaSpatial引入了一个自适应迭代推理过程，在这个过程中，VLM通过分析渲染输出来逐步改进空间布置，从而提高场景的一致性。 实证研究表明，MetaSpatial显著增强了各种规模模型的空间一致性和格式稳定性。经过训练后，物体放置变得更加逼真、对齐且功能上连贯，这验证了RL在元宇宙、AR/VR、数字孪生和游戏开发应用中的3D空间推理能力的有效性。我们的代码、数据及培训流程可在以下网址公开获取：[此处插入URL]。
URL
https://arxiv.org/abs/2503.18470
PDF
-
Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes
2025-03-23 17:48:44Kelly O. Marshall, Omid Poursaeed, Sergiu Oprea, Amit Kumar, Anushrut Jignasu, Chinmay Hegde, Yilei Li, Rakesh RanjanAbstract
3D indoor scene generation is an important problem for the design of digital and real-world environments. To automate this process, a scene generation model should be able to not only generate plausible scene layouts, but also take into consideration visual features and style preferences. Existing methods for this task exhibit very limited control over these attributes, only allowing text inputs in the form of simple object-level descriptions or pairwise spatial relationships. Our proposed method Decorum enables users to control the scene generation process with natural language by adopting language-based representations at each stage. This enables us to harness recent advancements in Large Language Models (LLMs) to model language-to-language mappings. In addition, we show that using a text-based representation allows us to select furniture for our scenes using a novel object retrieval method based on multimodal LLMs. Evaluations on the benchmark 3D-FRONT dataset show that our methods achieve improvements over existing work in text-conditioned scene synthesis and object retrieval.
Abstract (translated)
三维室内场景生成是数字和现实世界环境设计中的一个重要问题。为了实现这一过程的自动化，一个场景生成模型不仅要能够生成合理的场景布局，还应考虑视觉特征和风格偏好。目前的方法在控制这些属性方面表现出非常有限的能力，仅允许使用简单的对象级别描述或空间关系对文本输入进行操作。 我们提出的方法Decorum通过在每个阶段采用基于语言的表示形式，使用户可以用自然语言控制场景生成过程。这使得我们可以利用大型语言模型（LLMs）的最新进展来建模语言到语言的映射。此外，我们还展示了使用文本基表示法可以让我们借助一种基于多模式LLM的新颖对象检索方法为我们的场景选择家具。 在基准数据集3D-FRONT上的评估表明，与现有的以文本条件化场景合成和物体检索的方法相比，我们的方法实现了改进。
URL
https://arxiv.org/abs/2503.18155
PDF
-
SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation
2025-03-23 09:11:04Haoliang Shang, Hanyu Wu, Guangyao Zhai, Boyang Sun, Fangjinhua Wang, Federico Tombari, Marc PollefeysAbstract
Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.
Abstract (translated)
场景图捕捉了物体之间的复杂关系，作为内容生成和操作的强大先验知识。然而，合理地操纵场景图（无论是添加节点还是修改边）仍然是一个具有挑战性的未被充分探索的任务。例如，向图中添加一个节点或推理某个节点与其他所有节点的关系在计算上是不可行的，因为即使是单个边的改动也可能由于图内部复杂的相互依赖关系而引发冲突。 为了解决这些挑战，我们提出了SG-Tailor，这是一种自回归模型，它能够预测任意两个节点之间的无冲突关系。SG-Tailor不仅推断了对象间的关系，包括生成新添加节点的常识性边缘连接，而且还解决了由于边修改产生的冲突以产生连贯且可以操作的图供下游任务使用。 对于节点添加，该模型查询目标节点和图中的其他节点来预测适当的关系。 在边修改时，SG-Tailor采用了一种剪切并缝合（Cut-And-Stitch）策略解决冲突，并全局调整整个图形结构。 广泛的实验表明，与竞争方法相比，SG-Tailor的性能显著更优，并且可以无缝集成作为场景生成和机器人操作任务中的插件模块。
URL
https://arxiv.org/abs/2503.18988
PDF
-
HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation
2025-03-21 04:36:57Hou In Derek Pun, Hou In Ivan Tam, Austin T. Wang, Xiaoliang Huo, Angel X. Chang, Manolis SavvaAbstract
Despite advances in indoor 3D scene layout generation, synthesizing scenes with dense object arrangements remains challenging. Existing methods primarily focus on large furniture while neglecting smaller objects, resulting in unrealistically empty scenes. Those that place small objects typically do not honor arrangement specifications, resulting in largely random placement not following the text description. We present HSM, a hierarchical framework for indoor scene generation with dense object arrangements across spatial scales. Indoor scenes are inherently hierarchical, with surfaces supporting objects at different scales, from large furniture on floors to smaller objects on tables and shelves. HSM embraces this hierarchy and exploits recurring cross-scale spatial patterns to generate complex and realistic indoor scenes in a unified manner. Our experiments show that HSM outperforms existing methods by generating scenes that are more realistic and better conform to user input across room types and spatial configurations.
Abstract (translated)
尽管在室内三维场景布局生成方面已经取得了进展，但合成包含密集物体排列的场景仍然具有挑战性。现有方法主要关注大型家具而忽视了较小的物品，导致生成的场景显得过于空旷。那些放置小物件的方法通常不遵循具体的排列规范，导致位置分布很大程度上是随机的，并不符合文本描述的要求。我们提出了HSM（分层框架），用于在不同空间尺度下生成包含密集物体布置的室内场景。室内场景本质上具有层次性，从地面上的大件家具到桌面上或架子上的小型物品，在不同的尺度上都有支持其摆放的表面。HSM采纳了这种层级结构，并利用重复出现的空间规模间的模式来统一生成复杂且真实的室内场景。实验结果表明，与现有方法相比，HSM在不同房间类型和空间配置下能生成更逼真、更能符合用户输入要求的场景。
URL
https://arxiv.org/abs/2503.16848
PDF
-
NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
2025-03-20 17:37:43Han-Hung Lee, Qinghong Han, Angel X. ChangAbstract
In this paper, we explore the task of generating expansive outdoor scenes, ranging from castles to high-rises. Unlike indoor scene generation, which has been a primary focus of prior work, outdoor scene generation presents unique challenges, including wide variations in scene heights and the need for a method capable of rapidly producing large landscapes. To address this, we propose an efficient approach that encodes scene chunks as uniform vector sets, offering better compression and performance than the spatially structured latents used in prior methods. Furthermore, we train an explicit outpainting model for unbounded generation, which improves coherence compared to prior resampling-based inpainting schemes while also speeding up generation by eliminating extra diffusion steps. To facilitate this task, we curate NuiScene43, a small but high-quality set of scenes, preprocessed for joint training. Notably, when trained on scenes of varying styles, our model can blend different environments, such as rural houses and city skyscrapers, within the same scene, highlighting the potential of our curation process to leverage heterogeneous scenes for joint training.
Abstract (translated)
在这篇论文中，我们探讨了生成广阔户外场景的任务，这些场景从城堡到高层建筑不等。与先前研究主要关注的室内场景生成不同，户外场景生成面临着独特的挑战，包括场景高度变化大以及需要一种能够快速生产大规模景观的方法。为了解决这些问题，我们提出了一种高效的方法，该方法将场景块编码成均匀向量集合，相比之前方法使用的空间结构化潜在变量，这种方法提供了更好的压缩和性能表现。 此外，我们还训练了一个显式的“outpainting”模型用于无边界生成，与先前基于重采样的填充方案相比，这种方案能够提高连贯性，并通过消除额外的扩散步骤加快了生成速度。为了便于这项任务，我们整理了NuiScene43，这是一个虽小但质量高的一组场景集，经过预处理以进行联合训练。 值得注意的是，在不同风格场景上训练时，我们的模型可以在同一场景中融合不同的环境，如乡村房屋和城市摩天大楼，这突显出我们的数据整理过程能够利用异构场景进行联合训练的潜力。
URL
https://arxiv.org/abs/2503.16375
PDF
