Scene_Generation
-
SceneLCM: End-to-End Layout-Guided Interactive Indoor Scene Generation with Latent Consistency Model
2025-06-08 11:30:31Yangkai Lin, Jiabao Lei, Kui JiaAbstract
Our project page: this https URL. Automated generation of complex, interactive indoor scenes tailored to user prompt remains a formidable challenge. While existing methods achieve indoor scene synthesis, they struggle with rigid editing constraints, physical incoherence, excessive human effort, single-room limitations, and suboptimal material quality. To address these limitations, we propose SceneLCM, an end-to-end framework that synergizes Large Language Model (LLM) for layout design with Latent Consistency Model(LCM) for scene optimization. Our approach decomposes scene generation into four modular pipelines: (1) Layout Generation. We employ LLM-guided 3D spatial reasoning to convert textual descriptions into parametric blueprints(3D layout). And an iterative programmatic validation mechanism iteratively refines layout parameters through LLM-mediated dialogue loops; (2) Furniture Generation. SceneLCM employs Consistency Trajectory Sampling(CTS), a consistency distillation sampling loss guided by LCM, to form fast, semantically rich, and high-quality representations. We also offer two theoretical justification to demonstrate that our CTS loss is equivalent to consistency loss and its distillation error is bounded by the truncation error of the Euler solver; (3) Environment Optimization. We use a multiresolution texture field to encode the appearance of the scene, and optimize via CTS loss. To maintain cross-geometric texture coherence, we introduce a normal-aware cross-attention decoder to predict RGB by cross-attending to the anchors locations in geometrically heterogeneous instance. (4)Physically Editing. SceneLCM supports physically editing by integrating physical simulation, achieved persistent physical realism. Extensive experiments validate SceneLCM's superiority over state-of-the-art techniques, showing its wide-ranging potential for diverse applications.
Abstract (translated)
我们的项目页面：此 HTTPS URL。自动生成复杂且互动性强的室内场景以响应用户提示仍然是一个重大挑战。虽然现有方法能够合成室内场景，但它们在编辑约束、物理不一致性、人力投入过大、仅限单个房间以及材料质量不佳等方面存在局限性。为了解决这些问题，我们提出了SceneLCM框架，该框架整合了大型语言模型（LLM）用于布局设计和潜在一致模型（LCM）用于场景优化的端到端方法。 我们的方法将场景生成分解成四个模块化管道： 1. **布局生成**：我们使用由LLM引导的3D空间推理技术，将文本描述转换为参数化的蓝图（即3D布局）。同时，通过LLM中介对话循环进行迭代验证机制，逐步细化布局参数。 2. **家具生成**：SceneLCM采用了一致性轨迹采样（CTS）方法，该方法由LCM指导的一致性蒸馏采样损失驱动，能够快速形成语义丰富且高质量的表示。此外，我们提供了两个理论依据来证明我们的CTS损失等同于一致性损失，并且其蒸馏误差被欧拉解算器截断误差所限定。 3. **环境优化**：我们使用多分辨率纹理字段编码场景外观，并通过CTS损失进行优化。为了保持跨几何结构的纹理连贯性，我们引入了一种法线感知交叉注意解码器，通过交叉关注不同实例中的锚定位置来预测RGB值。 4. **物理编辑支持**：SceneLCM通过整合物理模拟实现了持久的物理真实性，从而支持场景的物理编辑。广泛实验验证了SceneLCM在当前技术前沿上的优越性，并展示了其在各种应用领域的广泛应用潜力。
URL
https://arxiv.org/abs/2506.07091
PDF
-
AI-powered Contextual 3D Environment Generation: A Systematic Review
2025-06-05 15:56:28Miguel Silva, Alexandre Valle de CarvalhoAbstract
The generation of high-quality 3D environments is crucial for industries such as gaming, virtual reality, and cinema, yet remains resource-intensive due to the reliance on manual processes. This study performs a systematic review of existing generative AI techniques for 3D scene generation, analyzing their characteristics, strengths, limitations, and potential for improvement. By examining state-of-the-art approaches, it presents key challenges such as scene authenticity and the influence of textual inputs. Special attention is given to how AI can blend different stylistic domains while maintaining coherence, the impact of training data on output quality, and the limitations of current models. In addition, this review surveys existing evaluation metrics for assessing realism and explores how industry professionals incorporate AI into their workflows. The findings of this study aim to provide a comprehensive understanding of the current landscape and serve as a foundation for future research on AI-driven 3D content generation. Key findings include that advanced generative architectures enable high-quality 3D content creation at a high computational cost, effective multi-modal integration techniques like cross-attention and latent space alignment facilitate text-to-3D tasks, and the quality and diversity of training data combined with comprehensive evaluation metrics are critical to achieving scalable, robust 3D scene generation.
Abstract (translated)
高质量的三维环境生成对于游戏、虚拟现实和电影等行业至关重要，但因其依赖于手动过程而仍然非常耗费资源。本研究对现有的用于生成3D场景的生成式AI技术进行了系统性回顾，分析了这些技术的特点、优势、局限性和改进潜力。通过考察最前沿的方法，它揭示了一些关键挑战，如场景的真实性以及文本输入的影响。特别关注的是，如何利用AI在保持一致性的同时融合不同的风格领域，训练数据对输出质量的影响以及当前模型的限制。此外，本综述还调研了现有的评估真实性的指标，并探讨了行业专业人士如何将AI整合到他们的工作流程中。这项研究的发现旨在为当前的研究提供一个全面的理解基础，并作为未来关于基于人工智能的3D内容生成研究的基础。 关键发现包括：先进的生成架构能够在高计算成本下创建高质量的3D内容；有效的多模态集成技术，如跨注意力和潜在空间对齐，有助于文本到3D的任务；以及高质量且多样化的训练数据结合全面的评估指标对于实现可扩展、稳健的3D场景生成至关重要。
URL
https://arxiv.org/abs/2506.05449
PDF
-
Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation
2025-06-04 17:59:04Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson W. H. Lau, Wangmeng Zuo, Chunchao GuoAbstract
Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications.
Abstract (translated)
现实世界的应用，如视频游戏和虚拟现实，经常需要能够根据用户的自定义相机轨迹来模拟可以探索的三维场景。虽然从文本或图像生成3D物体已取得显著进展，但创建长距离、一致且可探索的3D场景仍然是一个复杂且具有挑战性的问题。在这项工作中，我们介绍了Voyager，这是一种新颖的视频扩散框架，可以从单张图片和用户定义的相机路径中生成世界一致性3D点云序列。与现有方法不同，Voyager实现了端到端的场景生成与重建，并在整个帧之间保持固有的连贯性，从而消除了对三维重建管线（例如运动结构或多视图立体匹配）的需求。 我们的方法集成了三个关键组件： 1. **世界一致性视频扩散**：一种统一架构，在给定现有世界观察的基础上联合生成一致的RGB和深度视频序列，以确保全局连贯性。 2. **长距离世界探索**：一个高效的世界缓存系统与点剔除结合，并通过平滑视频采样实现了自回归推理，能够进行上下文感知的一致迭代场景扩展。 3. **可扩展数据引擎**：一种视频重建管线，自动估计相机姿态并预测任意视频的度量深度，使大规模、多样化的训练数据整理成为可能，而无需手动标注三维信息。 通过这些设计，Voyager在视觉质量和几何精度上相对于现有方法有了明显的改进，并具有广泛的应用潜力。
URL
https://arxiv.org/abs/2506.04225
PDF
-
Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection
2025-06-04 16:55:35Boyong He, Yuxiang Ji, Qianwen Ye, Zhuoyue Tan, Liaoni WuAbstract
Domain generalization (DG) for object detection aims to enhance detectors' performance in unseen scenarios. This task remains challenging due to complex variations in real-world applications. Recently, diffusion models have demonstrated remarkable capabilities in diverse scene generation, which inspires us to explore their potential for improving DG tasks. Instead of generating images, our method extracts multi-step intermediate features during the diffusion process to obtain domain-invariant features for generalized detection. Furthermore, we propose an efficient knowledge transfer framework that enables detectors to inherit the generalization capabilities of diffusion models through feature and object-level alignment, without increasing inference time. We conduct extensive experiments on six challenging DG benchmarks. The results demonstrate that our method achieves substantial improvements of 14.0% mAP over existing DG approaches across different domains and corruption types. Notably, our method even outperforms most domain adaptation methods without accessing any target domain data. Moreover, the diffusion-guided detectors show consistent improvements of 15.9% mAP on average compared to the baseline. Our work aims to present an effective approach for domain-generalized detection and provide potential insights for robust visual recognition in real-world scenarios. The code is available at this https URL.
Abstract (translated)
领域泛化（Domain Generalization，DG）在目标检测中的应用旨在提高检测器在未见过场景下的表现。由于实际应用场景中存在复杂多变的情况，这一任务依然充满挑战。最近，扩散模型展示了其在生成多样化场景方面的卓越能力，这激发了我们将它们用于改善DG任务的潜力的研究兴趣。与直接生成图像不同，我们的方法通过在扩散过程中提取多步中间特征来获取跨域不变性特征，从而实现泛化检测。 此外，我们提出了一种高效的知识转移框架，使检测器能够继承扩散模型的泛化能力，而无需增加推理时间，并且该框架是通过特征级和对象级对齐完成的。我们在六个具有挑战性的DG基准数据集上进行了广泛的实验，结果显示我们的方法在不同域和损坏类型上比现有DG方法提高了14.0% mAP（平均精度）。值得注意的是，即使不访问目标域数据，我们提出的方法也超越了大多数领域适应方法的表现。 此外，由扩散模型指导的检测器相比基准线，在所有测试场景中平均mAP值提升达到了15.9%。我们的工作旨在提供一种有效的跨领域泛化检测解决方案，并为在实际场景中的稳健视觉识别提供潜在见解。代码可以在以下链接获取：[此链接](https://this https URL)。 请注意，最后一个URL应替换为您实际提供的代码仓库地址或其他相关资源的正确链接。
URL
https://arxiv.org/abs/2503.02101
PDF
-
FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
2025-06-03 12:01:41Tongyuan Bai, Wangyuanfan Bai, Dong Chen, Tieru Wu, Manyi Li, Rui MaAbstract
Controllability plays a crucial role in the practical applications of 3D indoor scene synthesis. Existing works either allow rough language-based control, that is convenient but lacks fine-grained scene customization, or employ graph based control, which offers better controllability but demands considerable knowledge for the cumbersome graph design process. To address these challenges, we present FreeScene, a user-friendly framework that enables both convenient and effective control for indoor scene this http URL, FreeScene supports free-form user inputs including text description and/or reference images, allowing users to express versatile design intentions. The user inputs are adequately analyzed and integrated into a graph representation by a VLM-based Graph Designer. We then propose MG-DiT, a Mixed Graph Diffusion Transformer, which performs graph-aware denoising to enhance scene generation. Our MG-DiT not only excels at preserving graph structure but also offers broad applicability to various tasks, including, but not limited to, text-to-scene, graph-to-scene, and rearrangement, all within a single model. Extensive experiments demonstrate that FreeScene provides an efficient and user-friendly solution that unifies text-based and graph based scene synthesis, outperforming state-of-the-art methods in terms of both generation quality and controllability in a range of applications.
Abstract (translated)
可控性在3D室内场景合成的实际应用中起着关键作用。现有研究要么提供粗糙的语言控制，虽然使用方便但缺乏精细的场景定制；要么采用基于图的控制方式，这种控制方式提供了更好的可控性，但需要用户具备复杂且繁琐的图形设计知识。为了解决这些问题，我们提出了一种名为FreeScene的框架，它既便捷又有效，能够使用户对室内场景进行细致的控制。 在FreeScene中，支持自由形式的用户输入，包括文本描述和/或参考图像，允许用户表达各种设计意图。通过基于视觉语言模型（VLM）的图设计器，用户的这些输入会被充分分析并整合到一个图形表示之中。然后我们提出了一种混合图扩散变压器(MG-DiT)，该变压器能够执行图感知去噪以增强场景生成过程。我们的MG-DiT不仅擅长保持图形结构，还具有广泛的应用性，包括但不限于文本到场景、图到场景以及重排任务，并且所有这些都在单一模型中实现。 通过广泛的实验验证，FreeScene提供了一种高效且用户友好的解决方案，能够统一基于文本和基于图的场景合成方法。在多种应用场合下，该方案不仅在生成质量上超越了当前最先进的方法，在可控性方面也表现出色。
URL
https://arxiv.org/abs/2506.02781
PDF
-
WorldExplorer: Towards Generating Fully Navigable 3D Scenes
2025-06-02 15:41:31Manuel-Andreas Schneider, Lukas H\"ollein, Matthias Nie{\ss}nerAbstract
Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce streched-out and noisy artifacts when moving beyond central or panoramic perspectives. To this end, we propose WorldExplorer, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. We initialize our scenes by creating multi-view consistent images corresponding to a 360 degree panorama. Then, we expand it by leveraging video diffusion models in an iterative scene generation pipeline. Concretely, we generate multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. Our novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. Finally, we fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. We believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments.
Abstract (translated)
从文本生成三维世界是计算机视觉领域一个备受期待的目标。现有的方法在场景内部探索方面受到限制，即超出中心或全景视角时会产生拉伸和噪点等不理想的效果。为此，我们提出了WorldExplorer这一新方法，它基于自回归视频轨迹生成技术，在广泛的视角范围内构建出完全可导航且具有连贯视觉质量的三维场景。我们的方法首先通过创建一系列多视图一致性图像（对应360度全景）来初始化场景。然后利用视频扩散模型在迭代场景生成流水线中扩展这些初始设置，具体来说，我们在预定义的短轨迹上生成多个视频，深入探索场景，包括围绕物体运动的情况。我们提出的新颖场景记忆机制让每个视频都基于最相关的先前视图条件化处理，同时一个碰撞检测机制防止了诸如移动进物体等退化的结果产生。最后，通过3D高斯点绘优化将所有生成的视角融合为统一的三维表示。 相比于之前的方法，WorldExplorer能够生成在大范围相机运动下依然保持稳定的高质量场景，首次实现了真实且不受限的探索。我们认为这标志着向生成沉浸式和真正可探索的虚拟三维环境迈出的重要一步。
URL
https://arxiv.org/abs/2506.01799
PDF
-
ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
2025-05-31 23:03:54Zeqi Gu, Yin Cui, Zhaoshuo Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yu Liu, Abe Davis, Yifan DingAbstract
Designing 3D scenes is traditionally a challenging task that demands both artistic expertise and proficiency with complex software. Recent advances in text-to-3D generation have greatly simplified this process by letting users create scenes based on simple text descriptions. However, as these methods generally require extra training or in-context learning, their performance is often hindered by the limited availability of high-quality 3D data. In contrast, modern text-to-image models learned from web-scale images can generate scenes with diverse, reliable spatial layouts and consistent, visually appealing styles. Our key insight is that instead of learning directly from 3D scenes, we can leverage generated 2D images as an intermediary to guide 3D synthesis. In light of this, we introduce ArtiScene, a training-free automated pipeline for scene design that integrates the flexibility of free-form text-to-image generation with the diversity and reliability of 2D intermediary layouts. First, we generate 2D images from a scene description, then extract the shape and appearance of objects to create 3D models. These models are assembled into the final scene using geometry, position, and pose information derived from the same intermediary image. Being generalizable to a wide range of scenes and styles, ArtiScene outperforms state-of-the-art benchmarks by a large margin in layout and aesthetic quality by quantitative metrics. It also averages a 74.89% winning rate in extensive user studies and 95.07% in GPT-4o evaluation. Project page: this https URL
Abstract (translated)
设计三维场景传统上是一项既需要艺术专长又需掌握复杂软件技能的挑战性任务。最近在文本到3D生成领域的进步通过让用户基于简单的文字描述来创建场景，大大简化了这一过程。然而，由于这些方法通常要求额外训练或上下文学习，因此受限于高质量三维数据有限可用性的性能问题仍然存在。相比之下，现代从网络规模图像中学习的文本到图像模型能够产生具有多样性和可靠性空间布局以及一致且视觉吸引人的风格的场景。 我们的关键见解是：与其直接从3D场景进行学习，不如利用生成的2D图像作为中间体来指导3D合成。基于此理念，我们介绍了ArtiScene——一个无需训练的自动化管线，用于场景设计，该管道将自由形式文本到图像生成的灵活性与2D中间布局的多样性和可靠性相结合。 首先，从场景描述中生成2D图像；然后提取对象的形状和外观以创建3D模型。这些模型利用来自同一中间图像的几何、位置和姿态信息进行最终场景组装。ArtiScene能够广泛适用于各种类型的场景和风格，并且在广泛的用户研究中获得了74.89%的胜率，在GPT-4o评估中得到了95.07%的好评。 通过定量指标，ArtiScene在布局和美学质量上大大优于最先进的基准测试。项目页面：[提供链接]
URL
https://arxiv.org/abs/2506.00742
PDF
-
SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation
2025-05-28 17:55:35Dekai Zhu, Yixuan Hu, Youquan Liu, Dongyue Lu, Lingdong Kong, Slobodan IlicAbstract
Leveraging recent diffusion models, LiDAR-based large-scale 3D scene generation has achieved great success. While recent voxel-based approaches can generate both geometric structures and semantic labels, existing range-view methods are limited to producing unlabeled LiDAR scenes. Relying on pretrained segmentation models to predict the semantic maps often results in suboptimal cross-modal consistency. To address this limitation while preserving the advantages of range-view representations, such as computational efficiency and simplified network design, we propose Spiral, a novel range-view LiDAR diffusion model that simultaneously generates depth, reflectance images, and semantic maps. Furthermore, we introduce novel semantic-aware metrics to evaluate the quality of the generated labeled range-view data. Experiments on the SemanticKITTI and nuScenes datasets demonstrate that Spiral achieves state-of-the-art performance with the smallest parameter size, outperforming two-step methods that combine the generative and segmentation models. Additionally, we validate that range images generated by Spiral can be effectively used for synthetic data augmentation in the downstream segmentation training, significantly reducing the labeling effort on LiDAR data.
Abstract (translated)
利用最近的扩散模型，基于LiDAR的大规模3D场景生成取得了显著的成功。尽管近期基于体素的方法可以同时生成几何结构和语义标签，现有的范围视图方法仅限于产生未标记的LiDAR场景。依赖预训练分割模型来预测语义地图通常会导致跨模态一致性较差的结果。为了在保留范围视图表示的优点（如计算效率和简化网络设计）的同时解决这一局限性，我们提出了Spiral，这是一种新颖的范围视图LiDAR扩散模型，可以同时生成深度图像、反射率图像和语义地图。此外，我们引入了新的基于语义的度量标准来评估生成的带标签的范围视图数据的质量。在SemanticKITTI和nuScenes数据集上的实验表明，Spiral实现了最先进的性能，并且参数规模最小，在结合生成模型和分割模型的两步方法中表现更佳。此外，我们验证了由Spiral生成的范围图像可以有效地用于下游分割训练中的合成数据增强，从而显著减少了对LiDAR数据的手动标注工作。
URL
https://arxiv.org/abs/2505.22643
PDF
-
Agentic 3D Scene Generation with Spatially Contextualized VLMs
2025-05-26 15:28:17Xinhang Liu, Yu-Wing Tai, Chi-Keung TangAbstract
Despite recent advances in multimodal content generation enabled by vision-language models (VLMs), their ability to reason about and generate structured 3D scenes remains largely underexplored. This limitation constrains their utility in spatially grounded tasks such as embodied AI, immersive simulations, and interactive 3D applications. We introduce a new paradigm that enables VLMs to generate, understand, and edit complex 3D environments by injecting a continually evolving spatial context. Constructed from multimodal input, this context consists of three components: a scene portrait that provides a high-level semantic blueprint, a semantically labeled point cloud capturing object-level geometry, and a scene hypergraph that encodes rich spatial relationships, including unary, binary, and higher-order constraints. Together, these components provide the VLM with a structured, geometry-aware working memory that integrates its inherent multimodal reasoning capabilities with structured 3D understanding for effective spatial reasoning. Building on this foundation, we develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context. The pipeline features high-quality asset generation with geometric restoration, environment setup with automatic verification, and ergonomic adjustment guided by the scene hypergraph. Experiments show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work. Further results demonstrate that injecting spatial context enables VLMs to perform downstream tasks such as interactive scene editing and path planning, suggesting strong potential for spatially intelligent systems in computer graphics, 3D vision, and embodied applications.
Abstract (translated)
尽管近期得益于视觉-语言模型（VLM）的进展，多模态内容生成得到了显著提升，但这些模型在理解和生成结构化的三维场景方面的能力仍然很大程度上未被探索。这一限制制约了它们在基于空间的任务中的实用性，如具身人工智能、沉浸式模拟和交互式的3D应用中。为此，我们引入了一个新的范例，使VLM能够通过注入不断演进的空间上下文来生成、理解和编辑复杂的三维环境。该上下文由多模态输入构建而成，并包含三个组成部分：场景肖像，提供高层次的语义蓝图；带有语义标签的点云，捕获对象级别的几何信息；以及场景超图，编码丰富的空间关系，包括一元、二元和高阶约束。这三部分共同为VLM提供了结构化且具有几何感知的工作记忆，将它的固有多模态推理能力与三维理解相结合，实现有效的空间推理。 在此基础上，我们开发了一个代理性3D场景生成流水线，在这个管道中，VLM可以迭代地读取并更新空间上下文。该流程的特点包括高质量的资产生成、带有自动验证的环境设置以及通过场景超图指导的人体工学调整。实验表明，我们的框架能够处理多样且具有挑战性的输入，达到了此前工作中未观察到的一般化水平。进一步的结果显示，注入空间上下文使VLM能够执行下游任务如交互式场景编辑和路径规划，这预示着在计算机图形、3D视觉以及具身应用中的智能空间系统有着强大的潜力。
URL
https://arxiv.org/abs/2505.20129
PDF
-
WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions
2025-05-23 17:59:24Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, Jiajun WuAbstract
WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: this https URL
Abstract (translated)
WonderPlay 是一个新颖的框架，它将物理模拟与视频生成相结合，可以从单一图像中生成条件化的动态3D场景。虽然先前的工作仅限于刚体或简单的弹性动力学，但 WonderPlay 特别设计了一个混合生成式仿真器来合成各种各样的 3D 动力学。该混合生成式仿真器首先使用物理求解器模拟粗糙的 3D 动力学，然后利用视频生成器在这些基础之上产生更精细、更具现实感的动作视频。接着，所生成的视频被用来更新动态的 3D 场景，形成一个闭环过程，在其中物理求解器与视频生成器之间相互作用。这种方案使得用户能够直观地控制场景，并且结合了基于物理模拟器的精确动力学和扩散基础视频生成器的表现力。 实验结果表明，WonderPlay 允许用户通过单一图像输入与包含布料、沙子、雪、液体、烟雾、弹性体以及刚性物体等不同内容的各种场景进行互动。代码将会公开发布。项目网站：[此链接](https://this.url/)
URL
https://arxiv.org/abs/2505.18151
PDF
-
CGS-GAN: 3D Consistent Gaussian Splatting GANs for High Resolution Human Head Synthesis
2025-05-23 07:56:25Florian Barthel, Wieland Morgenstern, Paul Hinzer, Anna Hilsmann, Peter EisertAbstract
Recently, 3D GANs based on 3D Gaussian splatting have been proposed for high quality synthesis of human heads. However, existing methods stabilize training and enhance rendering quality from steep viewpoints by conditioning the random latent vector on the current camera position. This compromises 3D consistency, as we observe significant identity changes when re-synthesizing the 3D head with each camera shift. Conversely, fixing the camera to a single viewpoint yields high-quality renderings for that perspective but results in poor performance for novel views. Removing view-conditioning typically destabilizes GAN training, often causing the training to collapse. In response to these challenges, we introduce CGS-GAN, a novel 3D Gaussian Splatting GAN framework that enables stable training and high-quality 3D-consistent synthesis of human heads without relying on view-conditioning. To ensure training stability, we introduce a multi-view regularization technique that enhances generator convergence with minimal computational overhead. Additionally, we adapt the conditional loss used in existing 3D Gaussian splatting GANs and propose a generator architecture designed to not only stabilize training but also facilitate efficient rendering and straightforward scaling, enabling output resolutions up to $2048^2$. To evaluate the capabilities of CGS-GAN, we curate a new dataset derived from FFHQ. This dataset enables very high resolutions, focuses on larger portions of the human head, reduces view-dependent artifacts for improved 3D consistency, and excludes images where subjects are obscured by hands or other objects. As a result, our approach achieves very high rendering quality, supported by competitive FID scores, while ensuring consistent 3D scene generation. Check our our project page here: this https URL
Abstract (translated)
最近，基于三维高斯点阵（3D Gaussian splatting）的三维生成对抗网络（3D GANs）被提出用于高质量的人脸合成。然而，现有的方法通过将随机潜在向量条件化于当前摄像机位置来稳定训练并提升从极端视角进行渲染的质量，这种方法会损害三维一致性，因为在重新合成人脸时随着相机的位置变化会产生明显的身份变化。相比之下，固定相机在一个单一的视角虽然可以为该视角提供高质量的渲染效果，但对于新的视图则表现不佳。移除视图条件化通常会导致GAN训练不稳定，经常使训练崩溃。 为了应对这些挑战，我们引入了CGS-GAN（Conditional Gaussian Splatting GAN），这是一种新型的三维高斯点阵生成对抗网络框架，它能够在不依赖于视图条件下实现稳定的训练和高质量、三维一致的人脸合成。为确保训练稳定性，我们提出了一种多视角正则化技术，该技术在几乎无额外计算成本的情况下增强了生成器的收敛性。此外，我们将现有的3D高斯点阵GAN中使用的条件损失进行了调整，并设计了一个新的生成器架构，不仅可以稳定训练过程，而且有助于高效的渲染和轻松扩展，从而能够输出高达$2048^2$分辨率的结果。 为了评估CGS-GAN的能力，我们基于FFHQ数据集构建了新的数据集。该数据集支持非常高的解析度，重点关注人脸的较大区域，并减少视图相关的伪影以改善三维一致性。同时排除了被手或其他物体遮挡的图像样本。最终结果表明，我们的方法在高质量渲染方面取得了卓越的成绩，这得到了竞争性的FID（Fréchet Inception Distance）分数的支持，同时也保证了一致的三维场景生成。 如需了解更多信息，请访问我们的项目页面：[此链接](https://this-url.com) （请将"this https URL"替换为您实际的网址）。
URL
https://arxiv.org/abs/2505.17590
PDF
-
Constructing a 3D Town from a Single Image
2025-05-21 17:10:47Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric WangAbstract
Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce 3DTown, a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that 3DTown outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach.
Abstract (translated)
获取详细的三维场景通常需要昂贵的设备、多视角数据或复杂的建模过程。因此，一种轻量级的方法——从单一顶视图图像生成复杂三维场景，在实际应用中扮演着重要角色。尽管最近的三维生成模型在物体级别的表现非常出色，但它们扩展到全场景生成时往往会导致不一致的几何结构、布局幻觉以及低质量的网格。为此，我们在本研究中引入了3DTown，这是一个无需训练框架，旨在从单一顶视图图像合成真实且连贯的三维场景。 我们的方法基于两个原则：区域化生成以提高二维到三维的一致性和分辨率，以及空间感知的三维修复填充来确保全局场景的一致性及高质量几何结构生成。具体而言，我们将输入图像分解为重叠的区域，并使用预训练的三维对象生成器生成每个区域；随后进行掩码修正流修复填充过程，以填补缺失的几何信息并保持结构连续性。 这种模块化设计允许我们克服分辨率瓶颈问题，并在无需三维监督或微调的情况下保存空间结构。通过在各种场景中进行广泛的实验表明，3DTown 在几何质量、空间一致性以及纹理保真度方面均优于当前最佳基线模型（包括 Trellis, Hunyuan3D-2 和 TripoSG）。 我们的研究成果展示了基于单一图像生成高质量三维城镇的可行性，并且采用了一种无需训练的原理性方法。
URL
https://arxiv.org/abs/2505.15765
PDF
-
3D Scene Generation: A Survey
2025-05-08 17:59:54Beichen Wen, Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei LiuAbstract
3D scene generation seeks to synthesize spatially structured, semantically meaningful, and photorealistic environments for applications such as immersive media, robotics, autonomous driving, and embodied AI. Early methods based on procedural rules offered scalability but limited diversity. Recent advances in deep generative models (e.g., GANs, diffusion models) and 3D representations (e.g., NeRF, 3D Gaussians) have enabled the learning of real-world scene distributions, improving fidelity, diversity, and view consistency. Recent advances like diffusion models bridge 3D scene synthesis and photorealism by reframing generation as image or video synthesis problems. This survey provides a systematic overview of state-of-the-art approaches, organizing them into four paradigms: procedural generation, neural 3D-based generation, image-based generation, and video-based generation. We analyze their technical foundations, trade-offs, and representative results, and review commonly used datasets, evaluation protocols, and downstream applications. We conclude by discussing key challenges in generation capacity, 3D representation, data and annotations, and evaluation, and outline promising directions including higher fidelity, physics-aware and interactive generation, and unified perception-generation models. This review organizes recent advances in 3D scene generation and highlights promising directions at the intersection of generative AI, 3D vision, and embodied intelligence. To track ongoing developments, we maintain an up-to-date project page: this https URL.
Abstract (translated)
三维场景生成旨在为沉浸式媒体、机器人技术、自动驾驶和具身人工智能等应用合成具有空间结构化、语义意义且逼真的环境。早期基于程序规则的方法虽然具备可扩展性，但多样性有限。近年来，深度生成模型（如GANs、扩散模型）以及3D表示方法（如NeRF、3D高斯分布）的进步使得能够学习真实场景的分布，从而提高了逼真度、多样性和视角一致性。最近的技术进步，比如扩散模型通过将生成问题重新定义为图像或视频合成问题的方式，成功地连接了三维场景生成与照片级真实性。本综述系统性地概述了当前最先进的方法，并将其分类为四大范式：程序化生成、基于神经3D的生成、基于图像的生成和基于视频的生成。我们分析了它们的技术基础、权衡以及代表性结果，还回顾了一些常用的数据库、评估协议及下游应用。最后，讨论了在生成能力、三维表示、数据与注释以及评估方面的关键挑战，并概述了一系列有前景的方向，包括更高的保真度、物理感知和互动生成，以及统一的感知-生成模型。这篇综述整理了最近在三维场景生成领域的进展，并突出了生成AI、三维视觉和具身智能交叉领域中的潜在发展方向。为了跟踪正在进行的发展，我们维护了一个最新的项目页面：[此URL](this https URL)。
URL
https://arxiv.org/abs/2505.05474
PDF
-
Steerable Scene Generation with Post Training and Inference-Time Search
2025-05-07 22:07:42Nicholas Pfaff, Hongkai Dai, Sergey Zakharov, Shun Iwase, Russ TedrakeAbstract
Training robots in simulation requires diverse 3D scenes that reflect the specific challenges of downstream tasks. However, scenes that satisfy strict task requirements, such as high-clutter environments with plausible spatial arrangement, are rare and costly to curate manually. Instead, we generate large-scale scene data using procedural models that approximate realistic environments for robotic manipulation, and adapt it to task-specific goals. We do this by training a unified diffusion-based generative model that predicts which objects to place from a fixed asset library, along with their SE(3) poses. This model serves as a flexible scene prior that can be adapted using reinforcement learning-based post training, conditional generation, or inference-time search, steering generation toward downstream objectives even when they differ from the original data distribution. Our method enables goal-directed scene synthesis that respects physical feasibility and scales across scene types. We introduce a novel MCTS-based inference-time search strategy for diffusion models, enforce feasibility via projection and simulation, and release a dataset of over 44 million SE(3) scenes spanning five diverse environments. Website with videos, code, data, and model weights: this https URL
Abstract (translated)
在仿真中训练机器人需要反映下游任务特定挑战的多样化三维场景。然而，满足严格任务要求（如具有合理空间布局的高杂乱环境）的场景非常罕见且难以手动整理。因此，我们利用过程模型生成大规模场景数据，这些模型可以近似现实环境中机器人的操作需求，并将这些数据调整以适应具体任务目标。为此，我们训练了一个统一的基于扩散的生成模型，该模型能预测从固定资产库中放置哪些物体及其 SE(3) 位姿（位置和姿态）。此模型作为一个灵活的场景先验知识可以利用强化学习后训练、条件生成或推理时间搜索进行调整，即使这些目标与原始数据分布不同也能引导生成朝向下游目标发展。我们的方法实现了尊重物理可行性的目标导向场景合成，并且能够跨各种场景类型扩展。 我们还引入了一种基于MCTS（蒙特卡罗树搜索）的推理时间搜索策略应用于扩散模型中，并通过投影和模拟来确保可行性，同时发布了一个包含超过4400万SE(3)场景的数据集，这些场景涵盖了五种多样的环境。网站提供了视频、代码、数据和模型权重：[此链接](https://example.com/)（请根据实际链接替换）。
URL
https://arxiv.org/abs/2505.04831
PDF
-
Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
2025-05-05 17:59:58Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, Zhaoshuo LiAbstract
Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
Abstract (translated)
从文本合成交互式3D场景对于游戏、虚拟现实和具身人工智能（Embodied AI）至关重要。然而，现有的方法面临着一些挑战。基于学习的方法依赖于小型的室内数据集，限制了场景多样性和布局复杂性；而大型语言模型（LLM）虽然可以利用多样的文本领域知识，但在空间真实感方面表现不佳，常常生成不符合常识的物体放置位置。我们的关键洞察是视觉感知能够弥合这一差距，通过提供现实的空间指导来弥补大型语言模型在此方面的不足。 为此，我们引入了Scenethesis框架，这是一个无需训练的代理框架，将基于LLM的场景规划与由视觉引导的布局精炼相结合。给定一个文本提示后，Scenethesis首先使用LLM生成粗略的布局草案；然后通过产生图像指引并提取场景结构来捕捉物体间的关联性以进行细化处理。接下来，优化模块迭代执行精确的姿态对齐和物理合理性检查，防止诸如物体穿透或不稳定等伪影现象的发生。最后，评判模块验证空间一致性。 全面实验表明，Scenethesis能够生成多样、真实且符合物理规律的3D交互场景，在虚拟内容创建、模拟环境以及具身人工智能研究方面具有重要的价值。
URL
https://arxiv.org/abs/2505.02836
PDF
-
MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation
2025-05-05 13:50:03Mingcheng Li, Xiaolu Hou, Ziyang Liu, Dingkang Yang, Ziyun Qian, Jiawei Chen, Jinjie Wei, Yue Jiang, Qingyao Xu, Lihua ZhangAbstract
Diffusion models have shown excellent performance in text-to-image generation. Nevertheless, existing methods often suffer from performance bottlenecks when handling complex prompts that involve multiple objects, characteristics, and relations. Therefore, we propose a Multi-agent Collaboration-based Compositional Diffusion (MCCD) for text-to-image generation for complex scenes. Specifically, we design a multi-agent collaboration-based scene parsing module that generates an agent system comprising multiple agents with distinct tasks, utilizing MLLMs to extract various scene elements effectively. In addition, Hierarchical Compositional diffusion utilizes a Gaussian mask and filtering to refine bounding box regions and enhance objects through region enhancement, resulting in the accurate and high-fidelity generation of complex scenes. Comprehensive experiments demonstrate that our MCCD significantly improves the performance of the baseline models in a training-free manner, providing a substantial advantage in complex scene generation.
Abstract (translated)
扩散模型在文本到图像生成方面表现出色。然而，现有的方法在处理涉及多个对象、特征和关系的复杂提示时常常会遇到性能瓶颈。因此，我们提出了一种基于多智能体协作的组合扩散（MCCD）方法，用于复杂的场景文本到图像生成。具体而言，我们设计了一个基于多智能体协作的场景解析模块，该模块生成一个由执行不同任务的多个代理组成的系统，并利用大规模语言模型有效地提取各种场景元素。此外，层次化组合扩散通过使用高斯掩码和过滤技术来细化边界框区域并增强对象，从而实现了复杂场景的精确且高质量生成。 全面的实验表明，在无需训练的情况下，我们的MCCD显著提高了基准模型在复杂场景生成方面的性能，并提供了明显的优势。
URL
https://arxiv.org/abs/2505.02648
PDF
-
DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion
2025-05-03 16:20:01Haoteng Li, Zhao Yang, Zezhong Qian, Gongpeng Zhao, Yuqi Huang, Jun Yu, Huazheng Zhou, Longjun LiuAbstract
Accurate and high-fidelity driving scene reconstruction relies on fully leveraging scene information as conditioning. However, existing approaches, which primarily use 3D bounding boxes and binary maps for foreground and background control, fall short in capturing the complexity of the scene and integrating multi-modal information. In this paper, we propose DualDiff, a dual-branch conditional diffusion model designed to enhance multi-view driving scene generation. We introduce Occupancy Ray Sampling (ORS), a semantic-rich 3D representation, alongside numerical driving scene representation, for comprehensive foreground and background control. To improve cross-modal information integration, we propose a Semantic Fusion Attention (SFA) mechanism that aligns and fuses features across modalities. Furthermore, we design a foreground-aware masked (FGM) loss to enhance the generation of tiny objects. DualDiff achieves state-of-the-art performance in FID score, as well as consistently better results in downstream BEV segmentation and 3D object detection tasks.
Abstract (translated)
准确且高保真的驾驶场景重建依赖于充分利用场景信息作为条件。然而，现有的方法主要依靠三维边界框和二值图来控制前景和背景，在捕捉场景复杂性和整合多模态信息方面存在不足。在本文中，我们提出了DualDiff，这是一种双分支条件扩散模型，旨在提升多视角驾驶场景生成的效果。我们引入了Occupancy Ray Sampling（ORS），一种语义丰富的三维表示方法，并结合数值驾驶场景表示，实现全面的前景和背景控制。为了改进跨模态信息整合，我们提出了一种语义融合注意力(SFA)机制，用于对齐并融合不同模态特征。此外，我们设计了一种基于前景感知的掩码（FGM）损失函数，以增强微小物体生成的效果。DualDiff在FID评分上达到了最先进的性能，并且在下游BEV分割和3D目标检测任务中也表现出一致更好的结果。
URL
https://arxiv.org/abs/2505.01857
PDF
-
HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation
2025-04-30 13:55:28Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li YuanAbstract
The rapid advancement of diffusion models holds the promise of revolutionizing the application of VR and AR technologies, which typically require scene-level 4D assets for user experience. Nonetheless, existing diffusion models predominantly concentrate on modeling static 3D scenes or object-level dynamics, constraining their capacity to provide truly immersive experiences. To address this issue, we propose HoloTime, a framework that integrates video diffusion models to generate panoramic videos from a single prompt or reference image, along with a 360-degree 4D scene reconstruction method that seamlessly transforms the generated panoramic video into 4D assets, enabling a fully immersive 4D experience for users. Specifically, to tame video diffusion models for generating high-fidelity panoramic videos, we introduce the 360World dataset, the first comprehensive collection of panoramic videos suitable for downstream 4D scene reconstruction tasks. With this curated dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion model that can convert panoramic images into high-quality panoramic videos. Following this, we present Panoramic Space-Time Reconstruction, which leverages a space-time depth estimation method to transform the generated panoramic videos into 4D point clouds, enabling the optimization of a holistic 4D Gaussian Splatting representation to reconstruct spatially and temporally consistent 4D scenes. To validate the efficacy of our method, we conducted a comparative analysis with existing approaches, revealing its superiority in both panoramic video generation and 4D scene reconstruction. This demonstrates our method's capability to create more engaging and realistic immersive environments, thereby enhancing user experiences in VR and AR applications.
Abstract (translated)
扩散模型的迅速发展预示着其在虚拟现实（VR）和增强现实（AR）技术应用中的革命性潜力，这些领域通常需要场景级别的4D资产来提升用户体验。然而，现有的扩散模型主要集中在静态3D场景或对象级别动态建模上，这限制了它们提供真正沉浸式体验的能力。为了解决这个问题，我们提出了一种名为HoloTime的框架，该框架集成了视频扩散模型，可以从单个提示或参考图像生成全景视频，并结合一种无缝将生成的全景视频转化为4D资产的方法，从而实现用户全方位的沉浸式4D体验。 具体而言，为了使视频扩散模型能够生成高质量的全景视频，我们引入了360World数据集，这是第一个针对下游4D场景重建任务而设计的全面全景视频集合。借助这一精心策划的数据集，我们提出了Panoramic Animator，这是一种两阶段的图像到视频扩散模型，可以将全景图像转换为高质量的全景视频。 在此之后，我们介绍了全景时空重构方法，这种方法利用时空深度估计技术将生成的全景视频转化为4D点云，并优化整体4D高斯散布表示法来重建在空间和时间上都一致的4D场景。为了验证我们的方法的有效性，我们进行了与现有方法的比较分析，结果表明它在全景视频生成和4D场景重构方面均表现出卓越性能。 这证明了我们的方法能够创建更吸引人且更具现实感的沉浸式环境，从而提升VR和AR应用中的用户体验。
URL
https://arxiv.org/abs/2504.21650
PDF
-
CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design
2025-04-28 04:35:04Weitao Feng, Hang Zhou, Jing Liao, Li Cheng, Wenbo ZhouAbstract
We present a novel approach for indoor scene synthesis, which learns to arrange decomposed cuboid primitives to represent 3D objects within a scene. Unlike conventional methods that use bounding boxes to determine the placement and scale of 3D objects, our approach leverages cuboids as a straightforward yet highly effective alternative for modeling objects. This allows for compact scene generation while minimizing object intersections. Our approach, coined CasaGPT for Cuboid Arrangement and Scene Assembly, employs an autoregressive model to sequentially arrange cuboids, producing physically plausible scenes. By applying rejection sampling during the fine-tuning stage to filter out scenes with object collisions, our model further reduces intersections and enhances scene quality. Additionally, we introduce a refined dataset, 3DFRONT-NC, which eliminates significant noise presented in the original dataset, 3D-FRONT. Extensive experiments on the 3D-FRONT dataset as well as our dataset demonstrate that our approach consistently outperforms the state-of-the-art methods, enhancing the realism of generated scenes, and providing a promising direction for 3D scene synthesis.
Abstract (translated)
我们提出了一种新颖的室内场景合成方法，该方法通过学习安排分解的立方体基元来表示场景中的三维对象。与传统的使用边界框确定三维物体放置和比例的方法不同，我们的方法利用立方体作为建模物体的直接且高度有效的方式。这使得在减少物体相交的同时能够生成紧凑的场景。我们称之为CasaGPT（用于立方体排列和场景组装）的方法采用自回归模型顺序安排立方体，从而生成物理上合理的场景。通过在微调阶段应用拒绝采样来过滤掉有对象碰撞的场景，我们的模型进一步减少了相交并提高了场景质量。此外，我们还引入了一个改进的数据集3DFRONT-NC，该数据集消除了原始数据集3D-FRONT中存在的大量噪声。在3D-FRONT和我们自己的数据集上的广泛实验表明，我们的方法始终优于最先进的方法，增强了生成场景的现实感，并为三维场景合成提供了有前景的方向。
URL
https://arxiv.org/abs/2504.19478
PDF
-
WORLDMEM: Long-term Consistent World Simulation with Memory
2025-04-16 17:59:30Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, Xingang PanAbstract
World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.
Abstract (translated)
世界模拟由于其能够建模虚拟环境并预测行动后果而日益流行。然而，有限的时间上下文窗口经常导致在保持长期一致性方面出现问题，特别是在维持3D空间一致性上。为此，我们提出了WorldMem框架，该框架通过引入一个存储记忆单元的记忆库来增强场景生成能力，这些记忆单元存储了记忆帧和状态（如姿态和时间戳）。通过采用有效的记忆注意力机制从这些记忆帧中提取相关信息，我们的方法能够在显著的视角或时间间隔下准确重构先前观察到的场景。此外，通过在状态中加入时间戳，我们的框架不仅能够模拟静态世界，还能捕捉其随时间的变化发展过程，从而支持感知和与虚拟世界的交互。我们在虚拟和真实场景中的大量实验验证了我们这种方法的有效性。
URL
https://arxiv.org/abs/2504.12369
PDF
