Scene_Generation
-
GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation
2025-12-29 03:40:05Tianchen Deng, Xuefeng Chen, Yi Chen, Qu Chen, Yuyao Xu, Lijin Yang, Le Xu, Yu Zhang, Bo Zhang, Wuxiong Huang, Hesheng WangAbstract
Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub this https URL.
Abstract (translated)
随着生成模型的进步，驾驶世界模型（DWMs）正在迅速发展。然而，现有的DWM缺乏对3D场景的理解能力，并且只能根据输入数据生成内容，而无法解释或推理关于驾驶环境的信息。此外，当前的方法用点云或BEV特征表示3D空间信息，这些方法不能准确地将文本信息与底层的3D场景相匹配。 为了解决这些问题，我们提出了一种基于3D高斯场景表示的新统一DWM框架，它能够同时实现对3D场景的理解和多模态场景生成，并且还支持理解和生成任务的情境丰富化。我们的方法通过将丰富的语言特征嵌入到每个高斯原语中直接将文本信息与3D场景进行对齐，从而实现了早期的模式对齐。 此外，我们设计了一种新颖的任务感知的语言引导采样策略，该策略可以去除冗余的3D高斯模型，并注入准确且紧凑的3D令牌至LLM（大型语言模型）。另外，我们还设计了一个双条件多模态生成模型，在其中我们的视觉-语言模型捕获的信息被用作高级语言条件与低级图像条件结合使用，共同引导多模态生成过程。 我们在nuScenes和NuInteract数据集上进行了全面的研究以验证框架的有效性。我们的方法达到了最先进的性能。我们将在GitHub上公开发布代码，该链接为[此URL]（请将此URL替换为您实际的发布地址）。
URL
https://arxiv.org/abs/2512.23180
PDF
-
Bridging Modalities and Transferring Knowledge: Enhanced Multimodal Understanding and Recognition
2025-12-23 16:46:58Gorjan RadevskiAbstract
This manuscript explores multimodal alignment, translation, fusion, and transference to enhance machine understanding of complex inputs. We organize the work into five chapters, each addressing unique challenges in multimodal machine learning. Chapter 3 introduces Spatial-Reasoning Bert for translating text-based spatial relations into 2D arrangements between clip-arts. This enables effective decoding of spatial language into visual representations, paving the way for automated scene generation aligned with human spatial understanding. Chapter 4 presents a method for translating medical texts into specific 3D locations within an anatomical atlas. We introduce a loss function leveraging spatial co-occurrences of medical terms to create interpretable mappings, significantly enhancing medical text navigability. Chapter 5 tackles translating structured text into canonical facts within knowledge graphs. We develop a benchmark for linking natural language to entities and predicates, addressing ambiguities in text extraction to provide clearer, actionable insights. Chapter 6 explores multimodal fusion methods for compositional action recognition. We propose a method fusing video frames and object detection representations, improving recognition robustness and accuracy. Chapter 7 investigates multimodal knowledge transference for egocentric action recognition. We demonstrate how multimodal knowledge distillation enables RGB-only models to mimic multimodal fusion-based capabilities, reducing computational requirements while maintaining performance. These contributions advance methodologies for spatial language understanding, medical text interpretation, knowledge graph enrichment, and action recognition, enhancing computational systems' ability to process complex, multimodal inputs across diverse applications.
Abstract (translated)
这份手稿探讨了多模态对齐、翻译、融合和迁移，以增强机器对复杂输入的理解。我们将这项工作分为五章，每章都针对多模态机器学习中的独特挑战。 第三章介绍了Spatial-Reasoning Bert，它能够将基于文本的空间关系转换为剪贴画之间的二维排列。这使得空间语言的有效解码成为可能，并转化为视觉表示，从而开启了与人类空间理解一致的自动场景生成之路。 第四章提出了一种方法，用于将医学文本翻译成解剖图谱中的特定三维位置。我们引入了一个损失函数，利用医学术语的空间共现来创建可解释映射，显著增强了对医学文本导航的理解能力。 第五章解决了将结构化文本转换为知识图中规范事实的问题。我们建立了一个基准，用以连接自然语言与实体和谓语，解决文本提取中的歧义问题，提供更清晰、可行的见解。 第六章探讨了用于组成性动作识别的多模态融合方法。我们提出了一种结合视频帧和对象检测表示的方法，提高了识别的鲁棒性和准确性。 第七章研究了面向第一人称视角动作识别的多模态知识迁移。我们展示了如何通过多模态知识蒸馏使RGB-only模型模仿基于多模态融合的能力，从而在减少计算需求的同时保持性能。 这些贡献推进了空间语言理解、医学文本解读、知识图谱丰富以及行动识别的方法论，增强了计算系统处理复杂多元输入的能力，涵盖各种应用。
URL
https://arxiv.org/abs/2512.20501
PDF
-
AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning
2025-12-19 17:55:48Ran Gong, Xiaohan Zhang, Jinghuan Shang, Maria Vittoria Minniti, Jigarkumar Patel, Valerio Pepe, Riedana Yan, Ahmet Gundogdu, Ivan Kapelyukh, Ali Abbas, Xiaoqiang Yan, Harsh Patel, Laura Herlant, Karl SchmeckpeperAbstract
Generalist robot learning remains constrained by data: large-scale, diverse, and high-quality interaction data are expensive to collect in the real world. While simulation has become a promising way for scaling up data collection, the related tasks, including simulation task design, task-aware scene generation, expert demonstration synthesis, and sim-to-real transfer, still demand substantial human effort. We present AnyTask, an automated framework that pairs massively parallel GPU simulation with foundation models to design diverse manipulation tasks and synthesize robot data. We introduce three AnyTask agents for generating expert demonstrations aiming to solve as many tasks as possible: 1) ViPR, a novel task and motion planning agent with VLM-in-the-loop Parallel Refinement; 2) ViPR-Eureka, a reinforcement learning agent with generated dense rewards and LLM-guided contact sampling; 3) ViPR-RL, a hybrid planning and learning approach that jointly produces high-quality demonstrations with only sparse rewards. We train behavior cloning policies on generated data, validate them in simulation, and deploy them directly on real robot hardware. The policies generalize to novel object poses, achieving 44% average success across a suite of real-world pick-and-place, drawer opening, contact-rich pushing, and long-horizon manipulation tasks. Our project website is at this https URL .
Abstract (translated)
通用型机器人学习仍受数据限制：在现实世界中收集大规模、多样化和高质量的交互数据成本高昂。尽管仿真已经成为扩大数据采集的一种有前景的方法，包括任务设计、感知任务的场景生成、专家演示合成以及从仿真到实际环境的数据转移等相关的任务仍然需要大量的手工劳动。我们提出了AnyTask，这是一个自动化的框架，它结合了大规模并行GPU仿真和基础模型来设计多样化的操作任务，并合成人机交互数据。我们介绍了三种用于生成旨在解决尽可能多的任务的专家演示的AnyTask代理：1）ViPR，一种新颖的任务与运动规划代理，采用了视觉语言模型循环中的平行细化；2）ViPR-Eureka，一个强化学习代理，具有生成密集奖励和大型语言模型引导接触采样的功能；3）ViPR-RL，一种混合型计划和学习方法，仅用稀疏奖励就能产生高质量的演示。我们在合成数据上训练行为克隆策略，在仿真中验证这些策略，并直接部署在真实的机器人硬件上。所训练的政策能够泛化到新的物体姿态，实现在一系列真实世界的抓取放置、抽屉打开、接触密集型推动以及长时程操作任务中的平均44%的成功率。 我们项目的官方网站位于[这里](https://this-is-the-url-of-your-project-website.com)（请将URL替换为实际项目网站地址）。
URL
https://arxiv.org/abs/2512.17853
PDF
-
3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework
2025-12-19 11:20:52Tobias Sautter, Jan-Niklas Dihlmann, Hendrik P. A. LenschAbstract
Recent advances in 3D scene generation produce visually appealing output, but current representations hinder artists' workflows that require modifiable 3D textured mesh scenes for visual effects and game development. Despite significant advances, current textured mesh scene reconstruction methods are far from artist ready, suffering from incorrect object decomposition, inaccurate spatial relationships, and missing backgrounds. We present 3D-RE-GEN, a compositional framework that reconstructs a single image into textured 3D objects and a background. We show that combining state of the art models from specific domains achieves state of the art scene reconstruction performance, addressing artists' requirements. Our reconstruction pipeline integrates models for asset detection, reconstruction, and placement, pushing certain models beyond their originally intended domains. Obtaining occluded objects is treated as an image editing task with generative models to infer and reconstruct with scene level reasoning under consistent lighting and geometry. Unlike current methods, 3D-RE-GEN generates a comprehensive background that spatially constrains objects during optimization and provides a foundation for realistic lighting and simulation tasks in visual effects and games. To obtain physically realistic layouts, we employ a novel 4-DoF differentiable optimization that aligns reconstructed objects with the estimated ground plane. 3D-RE-GEN~achieves state of the art performance in single image 3D scene reconstruction, producing coherent, modifiable scenes through compositional generation guided by precise camera recovery and spatial optimization.
Abstract (translated)
最近在三维场景生成领域的进展产生了视觉上吸引人的输出，但当前的表示方法阻碍了艺术家的工作流程，这些工作需要可修改的具有纹理的三维网格场景用于视觉效果和游戏开发。尽管已经取得了重大进步，目前的纹理网格场景重建方法仍远未达到适合艺术家使用的要求，它们在对象分解、空间关系准确性以及背景缺失方面存在不足。 我们提出了3D-RE-GEN，这是一种组合框架，可以将单张图像重构为具有纹理的三维物体和背景。研究表明，结合来自特定领域的最先进的模型可以实现最佳场景重建性能，并满足艺术家的需求。我们的重建管道集成了资产检测、重建和放置的模型，在某些情况下超越了原定的应用领域。获取被遮挡的对象被视为一项图像编辑任务，使用生成模型进行推理并在一致的光照和几何结构下重构整个场景。与当前的方法不同，3D-RE-GEN 生成全面的背景，该背景在优化过程中空间上限制物体，并为视觉效果和游戏中的现实主义照明和仿真提供基础。 为了获得物理上真实的布局，我们采用了一种新颖的四自由度可微分优化方法，将重建的对象与估计的地平面对齐。3D-RE-GEN 在单图像三维场景重构方面取得了最先进的性能，通过由精确相机恢复和空间优化引导的组合生成方法产生了连贯且可修改的场景。
URL
https://arxiv.org/abs/2512.17459
PDF
-
Sceniris: A Fast Procedural Scene Generation Framework
2025-12-18 18:55:03Jinghuan Shang, Harsh Patel, Ran Gong, Karl SchmeckpeperAbstract
Synthetic 3D scenes are essential for developing Physical AI and generative models. Existing procedural generation methods often have low output throughput, creating a significant bottleneck in scaling up dataset creation. In this work, we introduce Sceniris, a highly efficient procedural scene generation framework for rapidly generating large-scale, collision-free scene variations. Sceniris also provides an optional robot reachability check, providing manipulation-feasible scenes for robot tasks. Sceniris is designed for maximum efficiency by addressing the primary performance limitations of the prior method, Scene Synthesizer. Leveraging batch sampling and faster collision checking in cuRobo, Sceniris achieves at least 234x speed-up over Scene Synthesizer. Sceniris also expands the object-wise spatial relationships available in prior work to support diverse scene requirements. Our code is available at this https URL
Abstract (translated)
合成的3D场景对于开发物理人工智能和生成模型至关重要。现有的程序化生成方法往往具有较低的输出吞吐量，成为扩大数据集创建规模的一个重要瓶颈。在这项工作中，我们介绍了Sceniris，这是一个高效的程序化场景生成框架，可以快速生成大规模且无碰撞的场景变化。此外，Sceniris还提供了一个可选的机器人可达性检查功能，为机器人的任务提供了操作可行性的场景。 Sceniris通过解决先前方法Scene Synthesizer的主要性能限制而设计成了效率最高的状态。它利用批处理采样和cuRobo中的更快碰撞检测来实现至少比Scene Synthesizer快234倍的速度提升。此外，Sceniris还将之前工作中可用的对象级空间关系扩展到支持多样化的场景需求。 我们的代码可以在以下链接中获取：[此URL]（请将"[此URL]"替换为实际提供的GitHub或其它代码托管平台的URL）。
URL
https://arxiv.org/abs/2512.16896
PDF
-
Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting
2025-12-17 14:59:21Arthur Moreau, Richard Shaw, Michal Nazarczuk, Jisu Shin, Thomas Tanay, Zhensong Zhang, Songcen Xu, Eduardo P\'erez-PelliteroAbstract
Feed-forward 3D Gaussian Splatting (3DGS) models enable real-time scene generation but are hindered by suboptimal pixel-aligned primitive placement, which relies on a dense, rigid grid and limits both quality and efficiency. We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level, replacing the pixel grid with an adaptive, "Off The Grid" distribution. Inspired by keypoint detection, our multi-resolution decoder learns to distribute primitives across image patches. This module is trained end-to-end with a 3D reconstruction backbone using self-supervised learning. Our resulting pose-free model generates photorealistic scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models. It outperforms competitors while using far fewer primitives, demonstrating a more accurate and efficient allocation that captures fine details and reduces artifacts. Moreover, we observe that by learning to render 3D Gaussians, our 3D reconstruction backbone improves camera pose estimation, suggesting opportunities to train these foundational models without labels.
Abstract (translated)
前馈3D高斯点阵模型（3DGS）能够实现实时场景生成，但由于像素对齐的基本元素放置不理想而受限。这种限制依赖于密集且刚性的网格结构，从而影响了质量和效率的提升。我们引入了一种新的前馈架构，该架构能够在亚像素级别检测3D高斯基本体，并用一种自适应、"离网"(Off The Grid)分布取代传统的像素网格。受关键点检测的启发，我们的多分辨率解码器能够学习在图像补丁中分配这些基本元素。这一模块与用于三维重建的骨干网络一起，使用自我监督学习方法进行端到端训练。 我们得到的结果是没有姿态约束的模型可以在几秒钟内生成逼真的场景，并且在前馈模型的新视角合成方面达到了最先进的水平。它不仅超越了竞争对手，在使用远少于其他模型的基本体数量的同时，展示了一种更为准确和高效的分配方式，该方法能够捕捉到细节并减少伪影。此外，我们观察到通过学习渲染3D高斯分布，我们的三维重建骨干网络可以提高相机姿态估计的准确性，这表明有可能在没有标签的情况下训练这些基础模型。
URL
https://arxiv.org/abs/2512.15508
PDF
-
I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
2025-12-15 18:59:13Lu Ling, Yunhao Ge, Yichen Sheng, Aniket BeraAbstract
Generalization remains the central challenge for interactive 3D scene generation. Existing learning-based approaches ground spatial understanding in limited scene dataset, restricting generalization to new layouts. We instead reprogram a pre-trained 3D instance generator to act as a scene level learner, replacing dataset-bounded supervision with model-centric spatial supervision. This reprogramming unlocks the generator transferable spatial knowledge, enabling generalization to unseen layouts and novel object compositions. Remarkably, spatial reasoning still emerges even when the training scenes are randomly composed objects. This demonstrates that the generator's transferable scene prior provides a rich learning signal for inferring proximity, support, and symmetry from purely geometric cues. Replacing widely used canonical space, we instantiate this insight with a view-centric formulation of the scene space, yielding a fully feed-forward, generalizable scene generator that learns spatial relations directly from the instance model. Quantitative and qualitative results show that a 3D instance generator is an implicit spatial learner and reasoner, pointing toward foundation models for interactive 3D scene understanding and generation. Project page: this https URL
Abstract (translated)
泛化仍然是互动式三维场景生成的核心挑战。现有的基于学习的方法将空间理解建立在有限的场景数据集上，从而限制了其对新布局的一般化能力。我们则重新编程了一个预训练的3D实例生成器，使其作为场景级的学习者来运作，并用模型中心的空间监督取代了局限于数据集的监督。这种重编程解锁了生成器可迁移的空间知识，使它能够推广到未见过的布局和新物体组合上。值得注意的是，即使在由随机组合的对象构成的训练场景中，空间推理仍然出现。这表明生成器的可转移场景先验提供了丰富的学习信号，可以从纯粹的几何线索中推断出接近性、支撑性和对称性。 通过用以视角为中心的形式化替代广泛使用的标准空间概念，我们实现了这一洞察，并提出了一种完全前馈的、通用化的场景生成器，该生成器直接从实例模型中学得空间关系。定量和定性的结果表明，一个3D实例生成器是一个隐含的空间学习者和推理工具，这指向了用于交互式3D场景理解和生成的基础模型的发展方向。 项目页面: [此链接](this https URL)
URL
https://arxiv.org/abs/2512.13683
PDF
-
RoomPilot: Controllable Synthesis of Interactive Indoor Environments via Multimodal Semantic Parsing
2025-12-12 02:33:09Wentang Chen, Shougao Zhang, Yiman Zhang, Tianhao Zhou, Ruihui LiAbstract
Generating controllable and interactive indoor scenes is fundamental to applications in game development, architectural visualization, and embodied AI training. Yet existing approaches either handle a narrow range of input modalities or rely on stochastic processes that hinder controllability. To overcome these limitations, we introduce RoomPilot, a unified framework that parses diverse multi-modal inputs--textual descriptions or CAD floor plans--into an Indoor Domain-Specific Language (IDSL) for indoor structured scene generation. The key insight is that a well-designed IDSL can act as a shared semantic representation, enabling coherent, high-quality scene synthesis from any single modality while maintaining interaction semantics. In contrast to conventional procedural methods that produce visually plausible but functionally inert layouts, RoomPilot leverages a curated dataset of interaction-annotated assets to synthesize environments exhibiting realistic object behaviors. Extensive experiments further validate its strong multi-modal understanding, fine-grained controllability in scene generation, and superior physical consistency and visual fidelity, marking a significant step toward general-purpose controllable 3D indoor scene generation.
Abstract (translated)
生成可控且互动的室内场景对于游戏开发、建筑可视化以及具身人工智能训练等应用来说至关重要。然而，现有的方法要么只能处理有限种类的输入模态（如文本描述或CAD平面图），要么依赖于随机过程来创建这些场景，这使得难以实现精确控制。为了克服这些问题，我们引入了RoomPilot——一个统一框架，它能够解析多种多样的多模态输入（包括文本描述和CAD楼层计划）并将其转换为室内专用领域的语言（IDSL），进而生成结构化的室内场景。关键的理念是：经过精心设计的IDSL可以作为共享语义表示，从单一模态中生成一致且高质量的场景，并保持交互性。 与传统的程序化方法不同，这些方法虽然能够产生视觉上可信但功能相对静止的布局，RoomPilot利用一个注有互动数据的数据集来合成表现出真实物体行为的环境。大量的实验进一步验证了它在多模态理解、场景生成中的精细控制以及物理一致性和视觉逼真度方面的卓越表现，标志着向着通用可控3D室内场景生成迈出了一大步。
URL
https://arxiv.org/abs/2512.11234
PDF
-
SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model
2025-12-11 18:59:56Yukai Shi, Weiyu Li, Zihao Wang, Hongyang Li, Xingyu Chen, Ping Tan, Lei ZhangAbstract
We propose a decoupled 3D scene generation framework called SceneMaker in this work. Due to the lack of sufficient open-set de-occlusion and pose estimation priors, existing methods struggle to simultaneously produce high-quality geometry and accurate poses under severe occlusion and open-set settings. To address these issues, we first decouple the de-occlusion model from 3D object generation, and enhance it by leveraging image datasets and collected de-occlusion datasets for much more diverse open-set occlusion patterns. Then, we propose a unified pose estimation model that integrates global and local mechanisms for both self-attention and cross-attention to improve accuracy. Besides, we construct an open-set 3D scene dataset to further extend the generalization of the pose estimation model. Comprehensive experiments demonstrate the superiority of our decoupled framework on both indoor and open-set scenes. Our codes and datasets is released at this https URL.
Abstract (translated)
在本文中，我们提出了一种称为SceneMaker的解耦3D场景生成框架。由于缺乏足够的开放集去遮挡和姿态估计先验知识，在严重遮挡和开放集设置下同时产生高质量几何形状和准确的姿态方面，现有方法面临挑战。为了解决这些问题，我们首先将去遮挡模型从三维物体生成中分离出来，并通过利用图像数据集和收集的去遮挡数据集来增强其多样性，以处理更加多样化的开放集遮挡模式。此外，我们提出了一种统一的姿态估计模型，该模型结合了全局和局部机制的自我注意和交叉注意力，以提高准确性。另外，我们构建了一个开放集3D场景数据集，进一步扩展姿态估计模型的泛化能力。全面的实验表明，在室内和开放集场景中，我们的解耦框架具有优越性。我们的代码和数据集可在[此处](https://此URL/)获取。
URL
https://arxiv.org/abs/2512.10957
PDF
-
Self-Evolving 3D Scene Generation from a Single Image
2025-12-09 18:44:21Kaizhi Zheng, Yue Fan, Jing Gu, Zishuo Xu, Xuehai He, Xin Eric WangAbstract
Generating high-quality, textured 3D scenes from a single image remains a fundamental challenge in vision and graphics. Recent image-to-3D generators recover reasonable geometry from single views, but their object-centric training limits generalization to complex, large-scale scenes with faithful structure and texture. We present EvoScene, a self-evolving, training-free framework that progressively reconstructs complete 3D scenes from single images. The key idea is combining the complementary strengths of existing models: geometric reasoning from 3D generation models and visual knowledge from video generation models. Through three iterative stages--Spatial Prior Initialization, Visual-guided 3D Scene Mesh Generation, and Spatial-guided Novel View Generation--EvoScene alternates between 2D and 3D domains, gradually improving both structure and appearance. Experiments on diverse scenes demonstrate that EvoScene achieves superior geometric stability, view-consistent textures, and unseen-region completion compared to strong baselines, producing ready-to-use 3D meshes for practical applications.
Abstract (translated)
从单张图像生成高质量、具有纹理的三维场景仍然是计算机视觉和图形学中的一个基本挑战。最近的一些图像到三维模型的生成器能够从单一视角恢复合理的几何结构，但它们以对象为中心的训练方式限制了其在复杂大规模场景中实现忠实结构和纹理的一般化能力。我们提出了EvoScene，这是一个自我演化的、无需训练的框架，它可以逐步地从单张图片中重建完整的三维场景。核心思想是结合现有模型的优势：即三维生成模型中的几何推理能力和视频生成模型中的视觉知识。 通过三个迭代阶段——空间先验初始化、基于视觉引导的三维场景网格生成和基于空间引导的新视角生成——EvoScene在二维和三维域之间交替进行，逐步提升结构和外观的质量。实验结果表明，在多样化的场景中，EvoScene相比于强大的基准模型实现了更优的几何稳定性、视图一致性的纹理以及未见区域的完成度，并且可以生成适合实际应用的即用型3D网格。
URL
https://arxiv.org/abs/2512.08905
PDF
-
Optimization-Guided Diffusion for Interactive Scene Generation
2025-12-08 15:56:18Shiaho Li, Naisheng Ye, Tianyu Li, Kashyap Chitta, Tuo An, Peng Su, Boyang Wang, Haiou Liu, Chen Lv, Hongyang LiAbstract
Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are essential for this task are rare and underrepresented in driving datasets. Data-driven scene generation offers a low-cost alternative by synthesizing complex traffic behaviors from existing driving logs. However, existing models often lack controllability or yield samples that violate physical or social constraints, limiting their usability. We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling from a scene generation model. OMEGA re-anchors each reverse diffusion step via constrained optimization, steering the generation towards physically plausible and behaviorally coherent trajectories. Building on this framework, we formulate ego-attacker interactions as a game-theoretic optimization in the distribution space, approximating Nash equilibria to generate realistic, safety-critical adversarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes from 32.35% to 72.27% for free exploration capabilities, and from 11% to 80% for controllability-focused generation. Our approach can also generate $5\times$ more near-collision frames with a time-to-collision under three seconds while maintaining the overall scene realism.
Abstract (translated)
真实且多样化的多代理驾驶场景对于评估自动驾驶车辆至关重要，但这些场景中关键的安全事件（对任务来说是必不可少的）在驾驶数据集中非常罕见并且代表性不足。基于数据驱动的场景生成提供了一种低成本的选择，通过从现有的驾驶日志中合成复杂的交通行为来实现这一点。然而，现有模型往往缺乏可控性或产生违反物理和社会约束的样本，限制了它们的应用。 我们提出了OMEGA框架，这是一个优化引导、无需训练的框架，在基于扩散的方法进行场景生成采样过程中强制执行结构一致性和交互意识。通过在每个反向扩散步骤中应用有约束条件的优化方法，OMEGA重新定位每次迭代，使生成的过程朝着物理上合理且行为连贯的方向前进。 在此框架基础上，我们将自我车辆与攻击者之间的互动作为分布空间中的博弈论优化问题来处理，逼近纳什均衡以生成现实且具有安全挑战性的对抗性场景。在nuPlan和Waymo数据集上的实验表明，OMEGA提高了生成的场景的真实感、一致性和可控性，在自由探索能力方面将物理和行为上有效场景的比例从32.35%提升至72.27%，而在聚焦于可控性生成的情况下，这一比例则从11%大幅提升到80%。此外，我们的方法还能在保持整体场景真实性的前提下，产生五倍数量的接近碰撞帧，这些帧的时间距离（time-to-collision）小于三秒。 通过这样的改进，OMEGA不仅提高了合成驾驶场景的质量和真实性，还为自动驾驶车辆的安全性和性能评估提供了更有价值的数据。
URL
https://arxiv.org/abs/2512.07661
PDF
-
MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation
2025-12-08 15:17:01Zhiqi Li, Wenhuan Li, Tengfei Wang, Zhenwei Wang, Junta Wu, Haoyuan Wang, Yunhan Yang, Zehuan Huang, Yang Li, Peidong Liu, Chunchao GuoAbstract
Compositionality is critical for 3D object and scene generation, but existing part-aware 3D generation methods suffer from poor scalability due to quadratic global attention costs when increasing the number of components. In this work, we present MoCA, a compositional 3D generative model with two key designs: (1) importance-based component routing that selects top-k relevant components for sparse global attention, and (2) unimportant components compression that preserve contextual priors of unselected components while reducing computational complexity of global attention. With these designs, MoCA enables efficient, fine-grained compositional 3D asset creation with scalable number of components. Extensive experiments show MoCA outperforms baselines on both compositional object and scene generation tasks. Project page: this https URL
Abstract (translated)
组合性对于生成三维物体和场景至关重要，但现有的具有部件感知能力的3D生成方法在增加组件数量时由于二次全局注意力成本而面临扩展性的挑战。在这项工作中，我们提出了MoCA，这是一种组合式3D生成模型，具备两个关键设计：(1) 基于重要性进行组件路由，选择前k个相关的组件用于稀疏的全局注意力；(2) 对不重要的组件进行压缩，在减少全局注意力计算复杂度的同时保留未选组件的上下文先验。通过这两个设计，MoCA能够实现高效、细粒度的组合式3D资产创建，并且支持可扩展数量的组件。大量的实验表明，MoCA在组合式物体和场景生成任务上超越了基准方法。项目页面：[此链接](this https URL)
URL
https://arxiv.org/abs/2512.07628
PDF
-
MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Models
2025-12-03 20:33:18Shaoheng Fang, Chaohui Yu, Fan Wang, Qixing HuangAbstract
We introduce MVRoom, a controllable novel view synthesis (NVS) pipeline for 3D indoor scenes that uses multi-view diffusion conditioned on a coarse 3D layout. MVRoom employs a two-stage design in which the 3D layout is used throughout to enforce multi-view consistency. The first stage employs novel representations to effectively bridge the 3D layout and consistent image-based condition signals for multi-view generation. The second stage performs image-conditioned multi-view generation, incorporating a layout-aware epipolar attention mechanism to enhance multi-view consistency during the diffusion process. Additionally, we introduce an iterative framework that generates 3D scenes with varying numbers of objects and scene complexities by recursively performing multi-view generation (MVRoom), supporting text-to-scene generation. Experimental results demonstrate that our approach achieves high-fidelity and controllable 3D scene generation for NVS, outperforming state-of-the-art baseline methods both quantitatively and qualitatively. Ablation studies further validate the effectiveness of key components within our generation pipeline.
Abstract (translated)
我们介绍了MVRoom，这是一种用于室内三维场景的可控新颖视角合成（NVS）流水线，它使用基于粗略3D布局的多视图扩散。MVRoom采用两阶段设计，在整个过程中利用3D布局来强制执行多视图一致性。第一阶段采用了新的表示方法，有效地将3D布局与一致的图像条件信号结合起来，用于多视图生成。第二阶段进行受图像条件限制的多视图生成，并引入了一种具有视角意识的极线注意力机制，在扩散过程中增强多视图的一致性。 此外，我们还介绍了一个迭代框架，该框架通过递归执行多视图生成（MVRoom）来生成包含不同数量对象和场景复杂度的3D场景，支持从文本到场景的生成。实验结果表明，我们的方法在NVS方面实现了高质量且可控的三维场景生成，在定量和定性评估中均优于现有的基准方法。消融研究进一步验证了我们生成管道中关键组件的有效性。
URL
https://arxiv.org/abs/2512.04248
PDF
-
TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image
2025-12-01 02:38:52Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu SuAbstract
Generating high-fidelity, physically interactive 3D simulated tabletop scenes is essential for embodied AI--especially for robotic manipulation policy learning and data synthesis. However, current text- or image-driven 3D scene generation methods mainly focus on large-scale scenes, struggling to capture the high-density layouts and complex spatial relations that characterize tabletop scenes. To address these challenges, we propose TabletopGen, a training-free, fully automatic framework that generates diverse, instance-level interactive 3D tabletop scenes. TabletopGen accepts a reference image as input, which can be synthesized by a text-to-image model to enhance scene diversity. We then perform instance segmentation and completion on the reference to obtain per-instance images. Each instance is reconstructed into a 3D model followed by canonical coordinate alignment. The aligned 3D models then undergo pose and scale estimation before being assembled into a collision-free, simulation-ready tabletop scene. A key component of our framework is a novel pose and scale alignment approach that decouples the complex spatial reasoning into two stages: a Differentiable Rotation Optimizer for precise rotation recovery and a Top-view Spatial Alignment mechanism for robust translation and scale estimation, enabling accurate 3D reconstruction from 2D reference. Extensive experiments and user studies show that TabletopGen achieves state-of-the-art performance, markedly surpassing existing methods in visual fidelity, layout accuracy, and physical plausibility, capable of generating realistic tabletop scenes with rich stylistic and spatial diversity. Our code will be publicly available.
Abstract (translated)
生成高保真、物理互动的3D桌面仿真场景对于具身人工智能（embodied AI）至关重要，尤其是在机器人操作策略学习和数据合成方面。然而，当前基于文本或图像驱动的3D场景生成方法主要关注大规模场景的创建，难以捕捉到桌面场景中高密度布局和复杂空间关系的特点。为了应对这些挑战，我们提出了TabletopGen，这是一个无需训练、完全自动化的框架，可以生成多样化的实例级别的互动3D桌面场景。TabletopGen接受一张参考图像作为输入，该图像是通过文本到图像的模型合成而来，以增强场景多样性。然后，在参考图像上进行实例分割和补全操作，获取每个实例对应的图片。接着将每一个实例重构为一个3D模型，并进行标准坐标对齐。对齐后的3D模型会经历姿态与尺度估计过程，最后组装成一个无碰撞的、可以模拟的真实桌面场景。 我们框架的一个关键组成部分是新的姿态和尺度对齐方法，它将复杂的空间推理分解为两个阶段：第一阶段使用可微旋转优化器进行精确旋转恢复；第二阶段则采用俯视图空间对齐机制来稳健地估计位移与缩放比例。这种方法使得从2D参考图像到3D重建的准确度得到提升。 通过广泛的实验和用户研究，TabletopGen在视觉保真度、布局准确性以及物理真实性方面达到了最先进的水平，显著超越了现有方法，并能够生成具有丰富风格和空间多样性的现实桌面场景。我们的代码将公开发布。
URL
https://arxiv.org/abs/2512.01204
PDF
-
CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration
2025-11-29 14:01:13Boshi Tang, Henry Zheng, Rui Huang, Gao HuangAbstract
High-quality 3D scene generation from a single image is crucial for AR/VR and embodied AI applications. Early approaches struggle to generalize due to reliance on specialized models trained on curated small datasets. While recent advancements in large-scale 3D foundation models have significantly enhanced instance-level generation, coherent scene generation remains a challenge, where performance is limited by inaccurate per-object pose estimations and spatial inconsistency. To this end, this paper introduces CC-FMO, a zero-shot, camera-conditioned pipeline for single-image to 3D scene generation that jointly conforms to the object layout in input image and preserves instance fidelity. CC-FMO employs a hybrid instance generator that combines semantics-aware vector-set representation with detail-rich structured latent representation, yielding object geometries that are both semantically plausible and high-quality. Furthermore, CC-FMO enables the application of foundational pose estimation models in the scene generation task via a simple yet effective camera-conditioned scale-solving algorithm, to enforce scene-level coherence. Extensive experiments demonstrate that CC-FMO consistently generates high-fidelity camera-aligned compositional scenes, outperforming all state-of-the-art methods.
Abstract (translated)
从单张图像生成高质量的3D场景对于AR/VR和具身AI应用至关重要。早期的方法由于依赖于在精心策划的小型数据集上训练的专业模型，难以进行泛化。尽管近期通过大规模3D基础模型在实例级别的生成方面取得了显著进展，但连贯场景生成仍然是一个挑战，其性能受限于不准确的每个对象姿态估计和空间不一致性。为此，本文介绍了CC-FMO（相机条件下的零样本单图像到3D场景生成管道），该系统既符合输入图像中的物体布局又保持实例真实性。CC-FMO采用了一种结合语义感知向量集表示与细节丰富的结构化潜在表示的混合实例生成器，从而产生在语义上合理且高质量的对象几何形状。此外，通过简单的相机条件下的尺度解决算法，CC-FMO使基础姿态估计模型能够在场景生成任务中应用，以强制执行场景级别的连贯性。广泛的实验表明，CC-FMO能够持续地生成高保真的相机对齐组合场景，并超越所有最先进的方法。
URL
https://arxiv.org/abs/2512.00493
PDF
-
GeoWorld: Unlocking the Potential of Geometry Models to Facilitate High-Fidelity 3D Scene Generation
2025-11-28 13:55:45Yuhao Wan, Lijuan Liu, Jingzhi Zhou, Zihan Zhou, Xuying Zhang, Dongbo Zhang, Shaohui Jiao, Qibin Hou, Ming-Ming ChengAbstract
Previous works leveraging video models for image-to-3D scene generation tend to suffer from geometric distortions and blurry content. In this paper, we renovate the pipeline of image-to-3D scene generation by unlocking the potential of geometry models and present our GeoWorld. Instead of exploiting geometric information obtained from a single-frame input, we propose to first generate consecutive video frames and then take advantage of the geometry model to provide full-frame geometry features, which contain richer information than single-frame depth maps or camera embeddings used in previous methods, and use these geometry features as geometrical conditions to aid the video generation model. To enhance the consistency of geometric structures, we further propose a geometry alignment loss to provide the model with real-world geometric constraints and a geometry adaptation module to ensure the effective utilization of geometry features. Extensive experiments show that our GeoWorld can generate high-fidelity 3D scenes from a single image and a given camera trajectory, outperforming prior methods both qualitatively and quantitatively. Project Page: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.23191
PDF
-
PAT3D: Physics-Augmented Text-to-3D Scene Generation
2025-11-26 23:23:58Guying Lin, Kemeng Huang, Michael Liu, Ruihan Gao, Hanke Chen, Lyuhao Chen, Beijia Lu, Taku Komura, Yuan Liu, Jun-Yan Zhu, Minchen LiAbstract
We introduce PAT3D, the first physics-augmented text-to-3D scene generation framework that integrates vision-language models with physics-based simulation to produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then converted into initial conditions for simulation. A differentiable rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we introduce a simulation-in-the-loop optimization procedure that guarantees physical stability and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality. Beyond high-quality generation, PAT3D uniquely enables simulation-ready 3D scenes for downstream tasks such as scene editing and robotic manipulation. Code and data will be released upon acceptance.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21978
PDF
-
LaGen: Towards Autoregressive LiDAR Scene Generation
2025-11-26 10:39:16Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Junjie Li, Juyong Zhang, Yukang Feng, Jianwen Sun, Songbur Wong, Junqi You, Junchi YanAbstract
Generative world models for autonomous driving (AD) have become a trending topic. Unlike the widely studied image modality, in this work we explore generative world models for LiDAR data. Existing generation methods for LiDAR data only support single frame generation, while existing prediction approaches require multiple frames of historical input and can only deterministically predict multiple frames at once, lacking interactivity. Both paradigms fail to support long-horizon interactive generation. To this end, we introduce LaGen, which to the best of our knowledge is the first framework capable of frame-by-frame autoregressive generation of long-horizon LiDAR scenes. LaGen is able to take a single-frame LiDAR input as a starting point and effectively utilize bounding box information as conditions to generate high-fidelity 4D scene point clouds. In addition, we introduce a scene decoupling estimation module to enhance the model's interactive generation capability for object-level content, as well as a noise modulation module to mitigate error accumulation during long-horizon generation. We construct a protocol based on nuScenes for evaluating long-horizon LiDAR scene generation. Experimental results comprehensively demonstrate LaGen outperforms state-of-the-art LiDAR generation and prediction models, especially on the later frames.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21256
PDF
-
MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments
2025-11-26 08:20:20Xu Hu, Yiyang Feng, Junran Peng, Jiawei He, Liyi Chen, Chuanchen Luo, Xucheng Yin, Qing Li, Zhaoxiang ZhangAbstract
The development of embodied agents for complex commercial environments is hindered by a critical gap in existing robotics datasets and benchmarks, which primarily focus on household or tabletop settings with short-horizon tasks. To address this limitation, we introduce MarketGen, a scalable simulation platform with automatic scene generation for complex supermarket environments. MarketGen features a novel agent-based Procedural Content Generation (PCG) framework. It uniquely supports multi-modal inputs (text and reference images) and integrates real-world design principles to automatically generate complete, structured, and realistic supermarkets. We also provide an extensive and diverse 3D asset library with a total of 1100+ supermarket goods and parameterized facilities assets. Building on this generative foundation, we propose a novel benchmark for assessing supermarket agents, featuring two daily tasks in a supermarket: (1) Checkout Unloading: long-horizon tabletop tasks for cashier agents, and (2) In-Aisle Item Collection: complex mobile manipulation tasks for salesperson agents. We validate our platform and benchmark through extensive experiments, including the deployment of a modular agent system and successful sim-to-real transfer. MarketGen provides a comprehensive framework to accelerate research in embodied AI for complex commercial applications.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21161
PDF
-
FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain
2025-11-26 06:58:57YuAn Wang, Xiaofan Li, Chi Huang, Wenhao Zhang, Hao Li, Bosheng Wang, Xun Sun, Jun WangAbstract
In controllable driving-scene reconstruction and 3D scene generation, maintaining geometric fidelity while synthesizing visually plausible appearance under large viewpoint shifts is crucial. However, effective fusion of geometry-based 3DGS and appearance-driven diffusion models faces inherent challenges, as the absence of pixel-wise, 3D-consistent editing criteria often leads to over-restoration and geometric drift. To address these issues, we introduce \textbf{FaithFusion}, a 3DGS-diffusion fusion framework driven by pixel-wise Expected Information Gain (EIG). EIG acts as a unified policy for coherent spatio-temporal synthesis: it guides diffusion as a spatial prior to refine high-uncertainty regions, while its pixel-level weighting distills the edits back into 3DGS. The resulting plug-and-play system is free from extra prior conditions and structural this http URL experiments on the Waymo dataset demonstrate that our approach attains SOTA performance across NTA-IoU, NTL-IoU, and FID, maintaining an FID of 107.47 even at 6 meters lane shift. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21113
PDF
