Scene_Generation
-
REACT3D: Recovering Articulations for Interactive Physical 3D Scenes
2025-10-13 12:37:59Zhao Huang, Boyang Sun, Alexandros Delitzas, Jiaqi Chen, Marc PollefeysAbstract
Interactive 3D scenes are increasingly vital for embodied intelligence, yet existing datasets remain limited due to the labor-intensive process of annotating part segmentation, kinematic types, and motion trajectories. We present REACT3D, a scalable zero-shot framework that converts static 3D scenes into simulation-ready interactive replicas with consistent geometry, enabling direct use in diverse downstream tasks. Our contributions include: (i) openable-object detection and segmentation to extract candidate movable parts from static scenes, (ii) articulation estimation that infers joint types and motion parameters, (iii) hidden-geometry completion followed by interactive object assembly, and (iv) interactive scene integration in widely supported formats to ensure compatibility with standard simulation platforms. We achieve state-of-the-art performance on detection/segmentation and articulation metrics across diverse indoor scenes, demonstrating the effectiveness of our framework and providing a practical foundation for scalable interactive scene generation, thereby lowering the barrier to large-scale research on articulated scene understanding. Our project page is \textit{\hypersetup{urlcolor=black}\href{this https URL}{this http URL}}.
Abstract (translated)
交互式3D场景对于具身智能（embodied intelligence）的重要性日益增加，然而由于对部件分割、运动类型和轨迹标注的劳动密集型过程，现有的数据集仍然受限。我们提出了REACT3D，这是一个可扩展的零样本框架，能够将静态3D场景转换为具有连贯几何结构的交互式副本，可以直接用于各种下游任务中。我们的贡献包括： (i) 可开启物体检测和分割：从静态场景中提取候选移动部件； (ii) 轴组估计：推断关节类型和运动参数； (iii) 隐藏几何补全并进行交互对象组装； (iv) 在广泛支持的格式下实现交互式场景集成，以确保与标准仿真平台兼容。 我们在各种室内场景上的检测/分割和轴组度量上取得了最先进的性能，展示了我们框架的有效性，并为大规模交互式场景生成提供了一个实用的基础，从而降低了研究复杂连贯场景理解的大规模障碍。我们的项目页面位于 [此链接](this https URL)。
URL
https://arxiv.org/abs/2510.11340
PDF
-
From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
2025-10-11 17:14:24Joy Hsu, Emily Jin, Jiajun Wu, Niloy J. MitraAbstract
Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.
Abstract (translated)
现实世界的场景，如ScanNet中的场景，难以捕捉，且可用的数据非常有限。生成具有多变物体姿态的逼真场景仍然是一个开放且具挑战性的任务。在这项工作中，我们提出了FactoredScenes框架，该框架通过利用房间的基础结构，并从日常生活场景中学习物体姿势的变化来合成逼真的3D场景。我们引入了一种分层的概念分解表示方法，将场景拆分为房间程序和物体姿态的组织化概念。 为了编码结构，FactoredScenes 学习了一个函数库，捕捉可重复使用的布局模式，然后利用大型语言模型生成高级程序，并通过学习到的库进行正则化处理。为了表示场景的变化，FactoredScenes 学习了一个条件于程序的模型来分层次预测物体姿态，并在场景中检索和放置3D对象。 我们展示了FactoredScenes能够生成难以与真实的ScanNet房间相区分的真实世界房间，证明了其有效性和创新性。
URL
https://arxiv.org/abs/2510.10292
PDF
-
MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
2025-10-05 06:37:26Zhenyu Pan, Yucheng Lu, Han LiuAbstract
We present MetaFind, a scene-aware tri-modal compositional retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches mainly rely on general-purpose 3D shape representation models. Our key innovation is a flexible retrieval mechanism that supports arbitrary combinations of text, image, and 3D modalities as queries, enhancing spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play equivariant layout encoder ESSGNN that captures spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene, regardless of coordinate frame transformations. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared to baseline methods.
Abstract (translated)
我们介绍了MetaFind，这是一个场景感知的三模态组合检索框架，旨在通过从大规模存储库中检索3D资产来增强元宇宙中的场景生成。MetaFind解决了两个核心挑战：(i) 不一致的资产检索忽略了空间、语义和风格约束；(ii) 缺乏专门针对3D资产检索的标准检索范式，因为现有方法主要依赖于通用的3D形状表示模型。 我们的关键技术创新在于提供了一种灵活的检索机制，支持任意组合文本、图像和3D模态作为查询，通过联合建模对象级别的特征（包括外观）和场景级别布局结构来增强空间推理和风格一致性。从方法论上看，MetaFind引入了一个即插即用等变布局编码器ESSGNN，能够捕捉空间关系和物体的外观特征，并确保检索到的3D资产在上下文和风格上与现有场景一致，无论坐标系转换如何。 该框架支持迭代场景构建，通过持续地将检索结果适应于当前场景更新。经验评估表明，在各种检索任务中，MetaFind相较于基线方法显著提升了空间和风格一致性。
URL
https://arxiv.org/abs/2510.04057
PDF
-
Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft
2025-10-03 17:35:16Junchao Huang, Xinting Hu, Boyao Han, Shaoshuai Shi, Zhuotao Tian, Tianyu He, Li JiangAbstract
Autoregressive video diffusion models have proved effective for world modeling and interactive scene generation, with Minecraft gameplay as a representative application. To faithfully simulate play, a model must generate natural content while exploring new scenes and preserve spatial consistency when revisiting explored areas. Under limited computation budgets, it must compress and exploit historical cues within a finite context window, which exposes a trade-off: Temporal-only memory lacks long-term spatial consistency, whereas adding spatial memory strengthens consistency but may degrade new scene generation quality when the model over-relies on insufficient spatial context. We present Memory Forcing, a learning framework that pairs training protocols with a geometry-indexed spatial memory. Hybrid Training exposes distinct gameplay regimes, guiding the model to rely on temporal memory during exploration and incorporate spatial memory for revisits. Chained Forward Training extends autoregressive training with model rollouts, where chained predictions create larger pose variations and encourage reliance on spatial memory for maintaining consistency. Point-to-Frame Retrieval efficiently retrieves history by mapping currently visible points to their source frames, while Incremental 3D Reconstruction maintains and updates an explicit 3D cache. Extensive experiments demonstrate that Memory Forcing achieves superior long-term spatial consistency and generative quality across diverse environments, while maintaining computational efficiency for extended sequences.
Abstract (translated)
自回归视频扩散模型已被证明在世界建模和交互式场景生成方面非常有效，以《我的世界》游戏玩法作为代表性应用。为了真实地模拟游戏体验，一个模型必须在探索新场景时生成自然内容，并且当重新访问已探索的区域时保持空间一致性。在计算资源有限的情况下，该模型需要压缩并在有限的时间窗口内利用历史线索，这暴露了一个权衡：仅时间记忆缺乏长期的空间一致性，而添加空间记忆虽然可以增强一致性，但可能会因过度依赖不足的空间上下文而导致新场景生成质量下降。 我们提出了Memory Forcing（内存强制）框架，这是一个结合了几何索引空间内存的训练协议的学习框架。混合训练通过区分不同的游戏模式，引导模型在探索阶段依赖时间记忆，并在重访时引入空间记忆以保持一致性。链式前向训练则是在自回归训练的基础上添加了模型预测序列，在这种情况下，连续的预测会创建更大的姿态变化，并鼓励模型依赖空间内存来维持一致性。点到帧检索通过将当前可见点映射回其源帧来进行历史记录的有效检索，而增量3D重建则维护并更新显式的三维缓存。 广泛的实验表明，Memory Forcing框架在各种环境中实现了卓越的长期空间一致性和生成质量，同时保持了对延长序列计算效率的需求。
URL
https://arxiv.org/abs/2510.03198
PDF
-
Text-to-Scene with Large Reasoning Models
2025-09-30 11:08:11Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Nick Tuninga, Roger WattenhoferAbstract
Prompt-driven scene synthesis allows users to generate complete 3D environments from textual descriptions. Current text-to-scene methods often struggle with complex geometries and object transformations, and tend to show weak adherence to complex instructions. We address these limitations by introducing Reason-3D, a text-to-scene model powered by large reasoning models (LRMs). Reason-3D integrates object retrieval using captions covering physical, functional, and contextual attributes. Reason-3D then places the selected objects based on implicit and explicit layout constraints, and refines their positions with collision-aware spatial reasoning. Evaluated on instructions ranging from simple to complex indoor configurations, Reason-3D significantly outperforms previous methods in human-rated visual fidelity, adherence to constraints, and asset retrieval quality. Beyond its contribution to the field of text-to-scene generation, our work showcases the advanced spatial reasoning abilities of modern LRMs. Additionally, we release the codebase to further the research in object retrieval and placement with LRMs.
Abstract (translated)
提示驱动的场景合成允许用户根据文本描述生成完整的3D环境。当前的文本到场景方法在处理复杂几何形状和对象变换时常常遇到困难，并且对复杂指令的遵循度较低。为了解决这些限制，我们引入了Reason-3D，这是一个由大型推理模型（LRMs）驱动的文本到场景模型。Reason-3D集成了基于描述物理属性、功能属性和上下文属性的对象检索功能。随后，它根据隐式和显式的布局约束放置选定对象，并通过碰撞感知的空间推理来优化它们的位置。 在从简单到复杂的室内配置指令上进行评估时，Reason-3D在人类评价的视觉逼真度、遵循约束的程度以及资产检索质量方面显著优于以前的方法。除了对文本到场景生成领域的贡献外，我们的工作还展示了现代LRMs先进的空间推理能力。此外，我们发布了代码库，以进一步推动LRMs在对象检索和放置方面的研究。
URL
https://arxiv.org/abs/2509.26091
PDF
-
M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
2025-09-28 08:16:08Yiheng Zhang, Zhuojiang Cai, Mingdao Wang, Meitong Guo, Tianxiao Li, Li Lin, Yuwang WangAbstract
In text-driven 3D scene generation, object layout serves as a crucial intermediate representation that bridges high-level language instructions with detailed geometric output. It not only provides a structural blueprint for ensuring physical plausibility but also supports semantic controllability and interactive editing. However, the learning capabilities of current 3D indoor layout generation models are constrained by the limited scale, diversity, and annotation quality of existing datasets. To address this, we introduce M3DLayout, a large-scale, multi-source dataset for 3D indoor layout generation. M3DLayout comprises 15,080 layouts and over 258k object instances, integrating three distinct sources: real-world scans, professional CAD designs, and procedurally generated scenes. Each layout is paired with detailed structured text describing global scene summaries, relational placements of large furniture, and fine-grained arrangements of smaller items. This diverse and richly annotated resource enables models to learn complex spatial and semantic patterns across a wide variety of indoor environments. To assess the potential of M3DLayout, we establish a benchmark using a text-conditioned diffusion model. Experimental results demonstrate that our dataset provides a solid foundation for training layout generation models. Its multi-source composition enhances diversity, notably through the Inf3DLayout subset which provides rich small-object information, enabling the generation of more complex and detailed scenes. We hope that M3DLayout can serve as a valuable resource for advancing research in text-driven 3D scene synthesis.
Abstract (translated)
在基于文本驱动的三维场景生成中，物体布局作为一种关键的中间表示形式，连接了高层次的语言指令和详细的几何输出。它不仅提供了确保物理合理性的结构蓝图，还支持语义可控性和交互式编辑。然而，当前三维室内布局生成模型的学习能力受到现有数据集规模有限、多样性不足以及标注质量不高的限制。为了解决这些问题，我们引入了M3DLayout，这是一个大规模的多源数据集，用于三维室内布局生成。M3DLayout包含15,080个布局和超过258k个物体实例，整合了三个不同的来源：真实世界的扫描、专业CAD设计以及程序化生成场景。每个布局都配有一份详细的结构化文本描述，包括全局场景概要、大型家具的相对位置关系以及小型物品的精细布置安排。这种多样性和丰富标注的数据资源使模型能够学习到复杂的空间和语义模式，在各种室内环境中具有广泛的适用性。 为了评估M3DLayout的潜力，我们使用基于条件文本扩散模型建立了一个基准测试平台。实验结果表明，我们的数据集为训练布局生成模型提供了坚实的基础。其多源构成增强了多样性，特别是Inf3DLayout子集提供的丰富的小物体信息，使更复杂和详细的场景生成成为可能。我们希望M3DLayout能够作为一个有价值的资源，推动基于文本驱动的三维场景合成研究的进步。
URL
https://arxiv.org/abs/2509.23728
PDF
-
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
2025-09-28 03:21:12Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In the field of 3D content generation, single image scene reconstruction methods still struggle to simultaneously ensure the quality of individual assets and the coherence of the overall scene in complex environments, while texture editing techniques often fail to maintain both local continuity and multi-view consistency. In this paper, we propose a novel system ZeroScene, which leverages the prior knowledge of large vision models to accomplish both single image-to-3D scene reconstruction and texture editing in a zero-shot manner. ZeroScene extracts object-level 2D segmentation and depth information from input images to infer spatial relationships within the scene. It then jointly optimizes 3D and 2D projection losses of the point cloud to update object poses for precise scene alignment, ultimately constructing a coherent and complete 3D scene that encompasses both foreground and background. Moreover, ZeroScene supports texture editing of objects in the scene. By imposing constraints on the diffusion model and introducing a mask-guided progressive image generation strategy, we effectively maintain texture consistency across multiple viewpoints and further enhance the realism of rendered results through Physically Based Rendering (PBR) material estimation. Experimental results demonstrate that our framework not only ensures the geometric and appearance accuracy of generated assets, but also faithfully reconstructs scene layouts and produces highly detailed textures that closely align with text prompts.
Abstract (translated)
在3D内容生成领域，单张图像场景重建方法仍难以同时保证复杂环境中各个资产的质量和整个场景的连贯性。此外，纹理编辑技术往往无法兼顾局部连续性和多视角一致性。为此，本文提出了一种新的系统ZeroScene，该系统利用大型视觉模型的先验知识，以零样本的方式实现单张图像到3D场景重建及纹理编辑。 ZeroScene从输入图像中提取对象级别的2D分割和深度信息，以此推断出场景中的空间关系。随后，通过联合优化点云的3D与2D投影损失来更新物体姿态，从而精确地对齐场景，并最终构建一个包含前景与背景且连贯完整的3D场景。 此外，ZeroScene还支持场景中对象的纹理编辑。通过对扩散模型施加约束并引入基于蒙版引导的渐进式图像生成策略，我们能够有效保持多视角下的纹理一致性，并通过物理基础渲染（PBR）材质估计进一步提升渲染结果的真实感。 实验结果显示，我们的框架不仅保证了生成资产的几何和外观准确性，还忠实还原场景布局，并产生与文本提示高度一致且细节丰富的纹理。
URL
https://arxiv.org/abs/2509.23607
PDF
-
WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
2025-09-27 16:47:44Ziyue Zhu, Zhanqian Wu, Zhenxin Zhu, Lijun Zhou, Haiyang Sun, Bing Wan, Kun Ma, Guang Chen, Hangjun Ye, Jin Xie, jian YangAbstract
Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose \textbf{WorldSplat}, a novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-track videos through two key steps: ((i)) We introduce a 4D-aware latent diffusion model integrating multi-modal information to produce pixel-aligned 4D Gaussians in a feed-forward manner. ((ii)) Subsequently, we refine the novel view videos rendered from these Gaussians using a enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that \textbf{WorldSplat} effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos.
Abstract (translated)
最近在驾驶场景生成和重建方面的进展展示了其增强自动驾驶系统的能力，通过生产可扩展且可控的训练数据。现有的生成方法主要集中在合成多样性和高保真的驾驶视频上；然而，由于三维一致性有限和视角覆盖稀疏的问题，它们难以支持便捷、高质量的新视点合成（NVS）。相比之下，最近的3D/4D重建方法在真实世界的驾驶场景中显著提高了NVS的质量，但天生缺乏生成能力。为了克服这种在场景生成与重建之间的困境，我们提出了**WorldSplat**，这是一种新颖的前馈框架，用于生成四维驾驶场景。我们的方法通过两个关键步骤有效地产生了具有一致性的多轨道视频：（i）我们引入了一种4D感知的潜在扩散模型，该模型集成了多模态信息，并以前馈方式生成像素对齐的4D高斯分布。（ii）随后，利用增强的视频扩散模型来细化从这些高斯分布渲染出的新视点视频。在基准数据集上进行的一系列实验表明，**WorldSplat**能够有效生成高度保真、时间和空间一致性的多轨道新视点驾驶视频。
URL
https://arxiv.org/abs/2509.23402
PDF
-
MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
2025-09-26 12:46:00Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, Jiangmiao PangAbstract
The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce MesaTask-10K, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with manually crafted layouts that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a Spatial Reasoning Chain that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present MesaTask, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts. Project page is at this https URL
Abstract (translated)
机器人解释人类指令并执行操作任务的能力需要具有与训练任务相关的桌面场景的可用性。然而，传统的创建这些场景的方法依赖于耗时的手动布局设计或完全随机化的布局设计，这两种方法在合理性和与任务的相关性方面都存在局限。 本文提出了一项新任务——目标导向型桌面场景生成（task-oriented tabletop scene generation），这项任务由于高层次的任务指令与实际的桌面场景之间的巨大差距而面临重大挑战。为了支持对此类具有挑战性的任务的研究，我们引入了MesaTask-10K数据集，这是一个大规模的数据集，包含大约10,700个合成桌面场景，这些场景的手动布局设计确保了现实主义和复杂的对象间关系。 为弥合任务与场景之间的差距，我们提出了一个空间推理链（Spatial Reasoning Chain），该链条将生成过程分解为物体推断、空间相互关联推理以及最终的3D布局场景图构建。我们提出了一种基于大型语言模型（LLM）的框架——MesaTask，该框架利用了上述推理链，并通过加入DPO算法进一步增强了其功能，以生成与给定任务描述相符且物理上合理的桌面场景。 详尽的实验表明，在生成符合任务要求、布局真实感强的桌面场景方面，MesaTask相较于基准方法具有显著的优势。项目页面位于[此处](https://该项目的网址在这里)。
URL
https://arxiv.org/abs/2509.22281
PDF
-
Drag4D: Align Your Motion with Text-Driven 3D Scene Generation
2025-09-26 05:23:45Minjun Kang, Inkyu Shin, Taeyeop Lee, In So Kweon, Kuk-Jin YoonAbstract
We introduce Drag4D, an interactive framework that integrates object motion control within text-driven 3D scene generation. This framework enables users to define 3D trajectories for the 3D objects generated from a single image, seamlessly integrating them into a high-quality 3D background. Our Drag4D pipeline consists of three stages. First, we enhance text-to-3D background generation by applying 2D Gaussian Splatting with panoramic images and inpainted novel views, resulting in dense and visually complete 3D reconstructions. In the second stage, given a reference image of the target object, we introduce a 3D copy-and-paste approach: the target instance is extracted in a full 3D mesh using an off-the-shelf image-to-3D model and seamlessly composited into the generated 3D scene. The object mesh is then positioned within the 3D scene via our physics-aware object position learning, ensuring precise spatial alignment. Lastly, the spatially aligned object is temporally animated along a user-defined 3D trajectory. To mitigate motion hallucination and ensure view-consistent temporal alignment, we develop a part-augmented, motion-conditioned video diffusion model that processes multiview image pairs together with their projected 2D trajectories. We demonstrate the effectiveness of our unified architecture through evaluations at each stage and in the final results, showcasing the harmonized alignment of user-controlled object motion within a high-quality 3D background.
Abstract (translated)
我们介绍了一种名为Drag4D的交互式框架，该框架集成了基于文本驱动的三维场景生成中的物体运动控制。此框架允许用户为从单张图像生成的三维物体定义三维轨迹，并将它们无缝地融入高质量的三维背景中。 Drag4D的工作流程分为三个阶段： 1. **增强型文本到三维背景生成**：我们利用2D高斯光斑技术结合全景图和修复后的新型视角，增强了由文本驱动生成三维场景的能力。这使得生成的三维重建更加密集且视觉上完整。 2. **基于参考图像的目标物体3D复制与粘贴方法**：给定目标物体的参考图像后，在这一阶段中，我们将该实例从全3D网格模型中提取出来，并将其无缝地合并到生成的三维场景中。我们采用现成的图像到三维模型转换技术来实现这一点。接下来，通过我们的物理感知对象位置学习方法，将物体网格放置在三维场景中的正确位置上，确保了空间对齐的准确性。 3. **时间动画处理**：最后一步是根据用户定义的三维轨迹，在时间维度上动态调整已经精确对齐的对象。为了减少运动错觉并保证多视角的一致性，我们开发了一种部分增强、以运动条件为基础的视频扩散模型。该模型能同时处理多视角图像对及其投影后的二维轨迹。 通过在每个阶段以及最终结果中的评估，我们展示了用户控制物体运动与高质量三维背景之间的和谐统一，证明了我们统一架构的有效性和优越性。
URL
https://arxiv.org/abs/2509.21888
PDF
-
4D Driving Scene Generation With Stereo Forcing
2025-09-24 15:37:17Hao Lu, Zhuang Ma, Guangfeng Jiang, Wenhang Ge, Bohan Li, Yuzhan Cai, Wenzhao Zheng, Yunpeng Zhang, Yingcong ChenAbstract
Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. Bridging generation and novel view synthesis remains a major challenge. We present PhiGenesis, a unified framework for 4D scene generation that extends video generation techniques with geometric and temporal consistency. Given multi-view image sequences and camera parameters, PhiGenesis produces temporally continuous 4D Gaussian splatting representations along target 3D trajectories. In its first stage, PhiGenesis leverages a pre-trained video VAE with a novel range-view adapter to enable feed-forward 4D reconstruction from multi-view images. This architecture supports single-frame or video inputs and outputs complete 4D scenes including geometry, semantics, and motion. In the second stage, PhiGenesis introduces a geometric-guided video diffusion model, using rendered historical 4D scenes as priors to generate future views conditioned on trajectories. To address geometric exposure bias in novel views, we propose Stereo Forcing, a novel conditioning strategy that integrates geometric uncertainty during denoising. This method enhances temporal coherence by dynamically adjusting generative influence based on uncertainty-aware perturbations. Our experimental results demonstrate that our method achieves state-of-the-art performance in both appearance and geometric reconstruction, temporal generation and novel view synthesis (NVS) tasks, while simultaneously delivering competitive performance in downstream evaluations. Homepage is at \href{this https URL}{PhiGensis}.
Abstract (translated)
当前的生成模型在同时支持时间外推和空间新视图合成（NVS）的情况下，难以生成具有动态4D驾驶场景的数据。连接生成与新视图合成仍然是一个主要挑战。我们提出了PhiGenesis，这是一个用于4D场景生成的统一框架，它通过几何和时间一致性扩展了视频生成技术。给定多视角图像序列和摄像机参数后，PhiGenesis能够沿着目标3D轨迹产生连续的时间4D高斯散射表示。 在第一个阶段中，PhiGenesis利用带有新颖范围视图适配器的预训练视频VAE（变分自编码器），使从多视角图像进行前向传播式的4D重建成为可能。这种架构支持单帧或视频输入，并输出包括几何、语义和运动在内的完整4D场景。 在第二个阶段，PhiGenesis引入了一个基于几何引导的视频扩散模型，使用渲染的历史4D场景作为先验条件来生成未来的视图，这些未来视图根据轨迹进行调节。为了解决新视角中几何暴露偏差的问题，我们提出了Stereo Forcing，这是一种新的条件策略，在去噪过程中整合了几何不确定性。这种技术通过基于不确定性的扰动动态调整生成影响来增强时间一致性。 实验结果表明，我们的方法在外观和几何重建、时间生成和新视图合成（NVS）任务中均达到了最先进的性能，并且同时在下游评估中提供了竞争性表现。 PhiGenesis的主页位于[此处](https://this https URL "PhiGensis")。
URL
https://arxiv.org/abs/2509.20251
PDF
-
SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
2025-09-24 09:06:41Yandan Yang, Baoxiong Jia, Shujie Zhang, Siyuan HuangAbstract
Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that SceneWeaver not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation. Project website: this https URL.
Abstract (translated)
随着嵌入式人工智能（Embodied AI）的发展，室内场景合成变得越来越重要。这不仅需要视觉上逼真的三维环境，还需要物理上合理且功能多样的空间。尽管最近的方法在提升视觉保真度方面取得了进展，但它们通常仍局限于固定场景类别，缺乏足够的物体级细节和物理一致性，并且难以适应复杂的用户指令。 在这项工作中，我们介绍了SceneWeaver，这是一个基于代理的框架，通过工具驱动的迭代优化统一了各种场景合成范式。SceneWeaver的核心是一个语言模型驱动的计划器，该计划器从一套可扩展的场景生成工具中进行选择，这些工具有数据驱动的生成模型、视觉方法和大型语言模型（LLM）方法等。计划过程通过自我评估物理合理性、视觉逼真度和与用户输入的语义一致性来进行指导。 这种闭环式的思考-行动-反思设计使代理能够识别语义不一致，调用特定工具，并在连续迭代中更新环境。在常见房间类型和开放式词汇表房间类型的广泛实验中，我们发现SceneWeaver不仅在物理、视觉和语义指标上优于先前的方法，而且还能有效地推广到具有多样指令的复杂场景，标志着向着通用3D环境生成迈出了一步。 项目网站：[此链接](https://this-url.com/)（请将“this https URL”替换为实际网址）。
URL
https://arxiv.org/abs/2509.20414
PDF
-
Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation
2025-09-23 17:58:01Sherwin Bahmani, Tianchang Shen, Jiawei Ren, Jiahui Huang, Yifeng Jiang, Haithem Turki, Andrea Tagliasacchi, David B. Lindell, Zan Gojcic, Sanja Fidler, Huan Ling, Jun Gao, Xuanchi RenAbstract
The ability to generate virtual environments is crucial for applications ranging from gaming to physical AI domains such as robotics, autonomous driving, and industrial AI. Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data, which is not always readily available. Recent advancements in video diffusion models have shown remarkable imagination capabilities, yet their 2D nature limits the applications to simulation where a robot needs to navigate and interact with the environment. In this paper, we propose a self-distillation framework that aims to distill the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for multi-view training data. Specifically, we augment the typical RGB decoder with a 3DGS decoder, which is supervised by the output of the RGB decoder. In this approach, the 3DGS decoder can be purely trained with synthetic data generated by video diffusion models. At inference time, our model can synthesize 3D scenes from either a text prompt or a single image for real-time rendering. Our framework further extends to dynamic 3D scene generation from a monocular input video. Experimental results show that our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
Abstract (translated)
生成虚拟环境的能力对于从游戏到物理AI领域（如机器人、自动驾驶和工业AI）的应用至关重要。目前基于学习的三维重建方法依赖于捕捉到的真实世界多视角数据，而这种数据并不总是易于获取。最近视频扩散模型的进步展示了令人印象深刻的想象能力，但其二维特性限制了在需要机器人导航与环境互动等模拟场景中的应用。在这篇论文中，我们提出了一种自我蒸馏框架，旨在将视频扩散模型中的隐含三维知识转化为显式的三维高斯点（3DGS）表示形式，从而无需多视角训练数据。具体来说，我们在典型的RGB解码器上添加了一个3DGS解码器，并通过RGB解码器的输出对其进行监督。在此方法中，3DGS解码器可以完全依赖于视频扩散模型生成的合成数据进行训练。在推理时，我们的模型可以从文本提示或单张图像实时渲染三维场景。此外，该框架还可以扩展到从单一视角输入视频生成动态三维场景。实验结果表明，我们提出的框架在静态和动态三维场景生成方面达到了最先进的性能。
URL
https://arxiv.org/abs/2509.19296
PDF
-
Octree Latent Diffusion for Semantic 3D Scene Generation and Completion
2025-09-20 00:53:13Xujia Zhang, Brendan Crowe, Christoffer HeckmanAbstract
The completion, extension, and generation of 3D semantic scenes are an interrelated set of capabilities that are useful for robotic navigation and exploration. Existing approaches seek to decouple these problems and solve them oneoff. Additionally, these approaches are often domain-specific, requiring separate models for different data distributions, e.g. indoor vs. outdoor scenes. To unify these techniques and provide cross-domain compatibility, we develop a single framework that can perform scene completion, extension, and generation in both indoor and outdoor scenes, which we term Octree Latent Semantic Diffusion. Our approach operates directly on an efficient dual octree graph latent representation: a hierarchical, sparse, and memory-efficient occupancy structure. This technique disentangles synthesis into two stages: (i) structure diffusion, which predicts binary split signals to construct a coarse occupancy octree, and (ii) latent semantic diffusion, which generates semantic embeddings decoded by a graph VAE into voxellevel semantic labels. To perform semantic scene completion or extension, our model leverages inference-time latent inpainting, or outpainting respectively. These inference-time methods use partial LiDAR scans or maps to condition generation, without the need for retraining or finetuning. We demonstrate highquality structure, coherent semantics, and robust completion from single LiDAR scans, as well as zero-shot generalization to out-of-distribution LiDAR data. These results indicate that completion-through-generation in a dual octree graph latent space is a practical and scalable alternative to regression-based pipelines for real-world robotic perception tasks.
Abstract (translated)
三维语义场景的完成、扩展和生成是一系列相互关联的能力，这些能力对于机器人导航与探索非常有用。现有方法试图将这些问题解耦，并单独解决每一个问题。此外，这些方法往往是特定领域的，需要为不同的数据分布（如室内场景与室外场景）分别构建模型。为了统一这些技术并提供跨域兼容性，我们开发了一个单一的框架，能够在室内和室外场景中执行场景完成、扩展和生成任务，我们将该框架命名为Octree Latent Semantic Diffusion。 我们的方法直接在一种高效且稀疏的记忆节约型占用结构——双八叉树图隐式表示上操作。这种技术将合成过程分为两个阶段：(i) 结构扩散，预测二进制分割信号以构建一个粗略的占用八叉树；(ii) 隐含语义扩散，生成由图VAE解码成体素级别语义标签的语义嵌入。 为了执行语义场景完成或扩展任务，我们的模型利用了推断时的隐式插值（完成）和外推（扩展）。这些推断方法使用部分LiDAR扫描或地图作为条件进行生成，而无需重新训练或微调。我们展示了从单次LiDAR扫描中获得高质量结构、连贯语义以及稳健完成能力的能力，并且还能够零样本泛化到分布外的LiDAR数据。 这些结果表明，在双八叉树图隐式空间中的通过生成进行场景完成是一种实用而可扩展的方法，可以作为基于回归的工作流管道在现实世界的机器人感知任务中的一种替代方案。
URL
https://arxiv.org/abs/2509.16483
PDF
-
SPATIALGEN: Layout-guided 3D Indoor Scene Generation
2025-09-18 14:12:32Chuan Fang, Heng Li, Yixun Liang, Jia Zheng, Yongsen Mao, Yuan Liu, Rui Tang, Zihan Zhou, Ping TanAbstract
Creating high-fidelity 3D models of indoor environments is essential for applications in design, virtual reality, and robotics. However, manual 3D modeling remains time-consuming and labor-intensive. While recent advances in generative AI have enabled automated scene synthesis, existing methods often face challenges in balancing visual quality, diversity, semantic consistency, and user control. A major bottleneck is the lack of a large-scale, high-quality dataset tailored to this task. To address this gap, we introduce a comprehensive synthetic dataset, featuring 12,328 structured annotated scenes with 57,440 rooms, and 4.7M photorealistic 2D renderings. Leveraging this dataset, we present SpatialGen, a novel multi-view multi-modal diffusion model that generates realistic and semantically consistent 3D indoor scenes. Given a 3D layout and a reference image (derived from a text prompt), our model synthesizes appearance (color image), geometry (scene coordinate map), and semantic (semantic segmentation map) from arbitrary viewpoints, while preserving spatial consistency across modalities. SpatialGen consistently generates superior results to previous methods in our experiments. We are open-sourcing our data and models to empower the community and advance the field of indoor scene understanding and generation.
Abstract (translated)
创建高保真度的室内环境3D模型对于设计、虚拟现实和机器人技术的应用至关重要。然而，手动进行3D建模仍然耗时且劳动密集型。虽然最近在生成式AI方面的进展已经使得场景自动化合成成为可能，但现有的方法往往面临着平衡视觉质量、多样性、语义一致性以及用户控制的挑战。一个重要瓶颈在于缺少一个大规模高质量的数据集来专门应对这一任务。为解决这一缺口，我们介绍了一个全面的人工数据集，该数据集中包含了12,328个结构化标注场景和57,440个房间，并有4.7M张逼真的2D渲染图像。利用这一数据集，我们提出了一种新颖的多视角多模态扩散模型——SpatialGen，能够生成视觉上真实且语义一致的室内三维场景。给定一个3D布局和一张参考图片（由文本提示派生），我们的模型可以从任意视角合成立体感外观（颜色图像）、几何结构（场景坐标图）以及语义信息（语义分割图），同时在不同模态间保持空间一致性。在实验中，SpatialGen始终比先前的方法产生更好的结果。我们计划开放源代码并公开数据和模型以赋能社区，并推动室内场景理解和生成领域的发展。
URL
https://arxiv.org/abs/2509.14981
PDF
-
Causal Reasoning Elicits Controllable 3D Scene Generation
2025-09-18 01:03:21Shen Chen, Ruiyu Zhao, Jiale Zhou, Zongkai Wu, Jenq-Neng Hwang, Lei LiAbstract
Existing 3D scene generation methods often struggle to model the complex logical dependencies and physical constraints between objects, limiting their ability to adapt to dynamic and realistic environments. We propose CausalStruct, a novel framework that embeds causal reasoning into 3D scene generation. Utilizing large language models (LLMs), We construct causal graphs where nodes represent objects and attributes, while edges encode causal dependencies and physical constraints. CausalStruct iteratively refines the scene layout by enforcing causal order to determine the placement order of objects and applies causal intervention to adjust the spatial configuration according to physics-driven constraints, ensuring consistency with textual descriptions and real-world dynamics. The refined scene causal graph informs subsequent optimization steps, employing a Proportional-Integral-Derivative(PID) controller to iteratively tune object scales and positions. Our method uses text or images to guide object placement and layout in 3D scenes, with 3D Gaussian Splatting and Score Distillation Sampling improving shape accuracy and rendering stability. Extensive experiments show that CausalStruct generates 3D scenes with enhanced logical coherence, realistic spatial interactions, and robust adaptability.
Abstract (translated)
现有的三维场景生成方法通常难以模拟物体之间的复杂逻辑依赖和物理约束，这限制了它们适应动态和现实环境的能力。我们提出了一种名为CausalStruct的新框架，该框架将因果推理嵌入到3D场景生成中。通过使用大型语言模型（LLMs），我们构建了因果图，其中节点代表对象及其属性，而边则编码因果依赖性和物理约束。CausalStruct 通过强制执行因果顺序来迭代地精炼场景布局以确定物体的放置顺序，并采用因果干预方法根据物理学驱动的约束调整空间配置，确保与文本描述和现实世界动态的一致性。精炼后的场景因果图指导后续优化步骤，利用比例-积分-微分（PID）控制器迭代调优对象的比例和位置。我们的方法使用文本或图像来引导3D场景中的物体放置和布局，并通过3D高斯点投影和评分蒸馏采样提高形状精度和渲染稳定性。广泛的实验表明，CausalStruct 能够生成具有增强逻辑连贯性、逼真空间交互以及强大适应性的三维场景。
URL
https://arxiv.org/abs/2509.15249
PDF
-
A Synthetic Data Pipeline for Supporting Manufacturing SMEs in Visual Assembly Control
2025-09-16 13:48:55Jonas Werheid, Shengjie He, Aymen Gannouni, Anas Abdelrazeq, Robert H. SchmittAbstract
Quality control of assembly processes is essential in manufacturing to ensure not only the quality of individual components but also their proper integration into the final product. To assist in this matter, automated assembly control using computer vision methods has been widely implemented. However, the costs associated with image acquisition, annotation, and training of computer vision algorithms pose challenges for integration, especially for small- and medium-sized enterprises (SMEs), which often lack the resources for extensive training, data collection, and manual image annotation. Synthetic data offers the potential to reduce manual data collection and labeling. Nevertheless, its practical application in the context of assembly quality remains limited. In this work, we present a novel approach for easily integrable and data-efficient visual assembly control. Our approach leverages simulated scene generation based on computer-aided design (CAD) data and object detection algorithms. The results demonstrate a time-saving pipeline for generating image data in manufacturing environments, achieving a mean Average Precision (mAP@0.5:0.95) up to 99,5% for correctly identifying instances of synthetic planetary gear system components within our simulated training data, and up to 93% when transferred to real-world camera-captured testing data. This research highlights the effectiveness of synthetic data generation within an adaptable pipeline and underscores its potential to support SMEs in implementing resource-efficient visual assembly control solutions.
Abstract (translated)
装配过程中的质量控制在制造业中至关重要，不仅确保了各个组件的质量，还保证了它们能够正确地集成到最终产品中。为了协助这一目标，广泛实施了基于计算机视觉方法的自动化装配控制系统。然而，图像获取、注释以及训练计算机视觉算法的相关成本对整合带来了挑战，尤其是对于资源有限的小型和中型企业（SMEs），这些企业通常缺乏进行大量培训、数据收集及手动图像标注的能力。合成数据提供了减少手动数据采集和标记潜力的可能性，但其在装配质量上下文中的实际应用仍然较为有限。 在此研究工作中，我们提出了一种新型的方法，旨在实现易于集成且高效的数据驱动视觉装配控制。我们的方法利用基于计算机辅助设计（CAD）数据的仿真场景生成，并结合了目标检测算法。实验结果表明，这为制造业环境中图像数据的生成提供了一个节省时间的工作流程，在针对合成行星齿轮系统组件实例的模拟训练数据上实现了高达99.5%的平均精度（mAP@0.5:0.95），而在转移到实际相机捕捉到的测试数据时，这一数字达到了最高93%。 这项研究强调了在具有适应性工作流程中生成合成数据的有效性，并突显了其支持SMEs实施资源高效视觉装配控制系统解决方案的巨大潜力。
URL
https://arxiv.org/abs/2509.13089
PDF
-
Narrative-to-Scene Generation: An LLM-Driven Pipeline for 2D Game Environments
2025-08-31 01:45:56Yi-Chun Chen, Arnav JhalaAbstract
Recent advances in large language models(LLMs) enable compelling story generation, but connecting narrative text to playable visual environments remains an open challenge in procedural content generation(PCG). We present a lightweight pipeline that transforms short narrative prompts into a sequence of 2D tile-based game scenes, reflecting the temporal structure of stories. Given an LLM-generated narrative, our system identifies three key time frames, extracts spatial predicates in the form of "Object-Relation-Object" triples, and retrieves visual assets using affordance-aware semantic embeddings from the GameTileNet dataset. A layered terrain is generated using Cellular Automata, and objects are placed using spatial rules grounded in the predicate structure. We evaluated our system in ten diverse stories, analyzing tile-object matching, affordance-layer alignment, and spatial constraint satisfaction across frames. This prototype offers a scalable approach to narrative-driven scene generation and lays the foundation for future work on multi-frame continuity, symbolic tracking, and multi-agent coordination in story-centered PCG.
Abstract (translated)
最近在大型语言模型（LLMs）方面的进展使得生成引人入胜的故事成为可能，但将叙述文本与可玩的视觉环境联系起来仍然是程序化内容生成（PCG）中的一个开放性挑战。我们提出了一种轻量级流水线，能够将简短的故事提示转换为一系列2D基于瓷砖的游戏场景，这些场景反映了故事的时间结构。 给定由大型语言模型生成的叙述文本后，我们的系统会识别出三个关键时间框架，并提取空间谓词的形式化为“对象-关系-对象”三元组。然后使用带有GameTileNet数据集中感知功能的语义嵌入来检索视觉资产。地形通过细胞自动机分层生成，而根据谓词结构的空间规则放置物体。 我们在十种不同类型的故事情节中评估了该系统，分析了瓷砖与物体之间的匹配度、功能层次的一致性以及跨框架的空间约束满足情况。 这一原型提供了一种基于叙事驱动场景生成的可扩展方法，并为未来工作奠定了基础，这些未来的工作将涉及多帧连续性、符号跟踪和以故事为中心的多智能体协调。
URL
https://arxiv.org/abs/2509.04481
PDF
-
LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding
2025-08-26 17:04:49Julian Ost, Andrea Ramazzina, Amogh Joshi, Maximilian B\"omer, Mario Bijelic, Felix HeideAbstract
Large-scale scene data is essential for training and testing in robot learning. Neural reconstruction methods have promised the capability of reconstructing large physically-grounded outdoor scenes from captured sensor data. However, these methods have baked-in static environments and only allow for limited scene control -- they are functionally constrained in scene and trajectory diversity by the captures from which they are reconstructed. In contrast, generating driving data with recent image or video diffusion models offers control, however, at the cost of geometry grounding and causality. In this work, we aim to bridge this gap and present a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal novel view synthesis with object permanence and explicit 3D geometry estimation. The proposed method combines the generation of a proxy geometry and environment representation with score distillation from learned 2D image priors. We find that this approach allows for high controllability, enabling the prompt-guided geometry and high-fidelity texture and structure that can be conditioned on map layouts -- producing realistic and geometrically consistent 3D generations of complex driving scenes.
Abstract (translated)
大规模场景数据对于机器人学习的训练和测试至关重要。神经重建方法承诺可以从捕获的传感器数据中重构大型且物理基础坚实的室外场景。然而，这些方法内置了静态环境限制，并仅允许有限的场景控制——它们的功能受限于用于重建的数据捕捉中的多样性和轨迹多样性。 相比之下，使用最近的图像或视频扩散模型生成驾驶数据可以提供更多的控制能力，但代价是失去了几何精确度和因果关系。在这项工作中，我们旨在弥合这一差距，并提出了一种直接生成大规模3D驾驶场景的方法，该方法具有准确的几何形状，并且能够进行因果驱动的新视图合成、物体持久性和显式的3D几何估计。 所提出的这种方法结合了代理几何体和环境表示的生成以及从学习到的2D图像先验中提取分数。我们发现，这种方法允许高度可控性，可以基于地图布局对提示引导下的几何形状和高保真的纹理及结构进行条件设定——从而能够产生现实且几何上一致的大规模驾驶场景的3D生成。
URL
https://arxiv.org/abs/2508.19204
PDF
-
HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generation
2025-08-25 09:32:57Xiping Wang, Yuxi Wang, Mengqi Zhou, Junsong Fan, Zhaoxiang ZhangAbstract
Realistic 3D indoor scene generation is crucial for virtual reality, interior design, embodied intelligence, and scene understanding. While existing methods have made progress in coarse-scale furniture arrangement, they struggle to capture fine-grained object placements, limiting the realism and utility of generated environments. This gap hinders immersive virtual experiences and detailed scene comprehension for embodied AI applications. To address these issues, we propose Hierarchical Layout Generation (HLG), a novel method for fine-grained 3D scene generation. HLG is the first to adopt a coarse-to-fine hierarchical approach, refining scene layouts from large-scale furniture placement to intricate object arrangements. Specifically, our fine-grained layout alignment module constructs a hierarchical layout through vertical and horizontal decoupling, effectively decomposing complex 3D indoor scenes into multiple levels of granularity. Additionally, our trainable layout optimization network addresses placement issues, such as incorrect positioning, orientation errors, and object intersections, ensuring structurally coherent and physically plausible scene generation. We demonstrate the effectiveness of our approach through extensive experiments, showing superior performance in generating realistic indoor scenes compared to existing methods. This work advances the field of scene generation and opens new possibilities for applications requiring detailed 3D environments. We will release our code upon publication to encourage future research.
Abstract (translated)
生成现实主义的三维室内场景对于虚拟现实、室内设计、具身智能和场景理解至关重要。尽管现有的方法在粗略规模的家具布局方面已取得进展，但在捕捉细微的对象放置细节上仍存在困难，这限制了生成环境的真实性和实用性。这种差距阻碍了沉浸式虚拟体验的发展以及对复杂AI应用场景中详细场景的理解。为了解决这些问题，我们提出了一种名为层级布局生成（HLG）的新方法，用于精细级的3D场景生成。HLG首次采用了一个从粗到细的分层方式，通过细化大型家具放置来实现复杂的对象安排布局。 具体而言，我们的精细化布局对齐模块通过对垂直和水平方向进行解耦来构建一个层级式布局，有效地将复杂的三维室内场景分解为多个级别的详细程度。此外，我们可训练的布局优化网络解决了摆放问题，例如不正确的定位、方向错误以及物体之间的碰撞，确保生成的场景在结构上是连贯且物理上可信的。 通过广泛的实验，我们展示了这种方法的有效性，在生成逼真的室内场景方面优于现有的方法。这项工作推动了场景生成领域的发展，并为需要详细3D环境的应用开辟了新的可能性。我们将会在发表时发布我们的代码以促进未来的相关研究。
URL
https://arxiv.org/abs/2508.17832
PDF
