Scene_Generation
-
BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving
2025-07-01 12:10:11Zeming Chen, Hang ZhaoAbstract
Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at: this https URL.
Abstract (translated)
在自动驾驶中，多视角图像生成要求不同相机视图之间具有连贯的三维场景理解。大多数现有的方法将这个问题视为二维图像集生成任务，并没有明确地进行三维建模。然而，我们认为一个结构化的表示对于场景生成至关重要，特别是在自动驾驶应用中。本文提出了BEV-VAE（Bird's Eye View Variational Autoencoder），以实现一致且可控的视图合成。BEV-VAE首先训练一个多视角图像变分自编码器，以形成紧凑而统一的鸟瞰图隐式空间，然后使用一个隐式扩散变压器来生成场景。BEV-VAE支持给定相机配置以及可选的三维布局的情况下进行任意视图生成。在nuScenes和Argoverse 2（AV2）数据集上的实验表明，在3D一致重建和生成方面具有出色表现。代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2507.00707
PDF
-
InfGen: Scenario Generation as Next Token Group Prediction
2025-06-29 16:18:32Zhenghao Peng, Yuxin Liu, Bolei ZhouAbstract
Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. We propose InfGen, a scenario generation framework that outputs agent states and trajectories in an autoregressive manner. InfGen represents the entire scene as a sequence of tokens, including traffic light signals, agent states, and motion vectors, and uses a transformer model to simulate traffic over time. This design enables InfGen to continuously insert new agents into traffic, supporting infinite scene generation. Experiments demonstrate that InfGen produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in InfGen-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at this https URL.
Abstract (translated)
现实而互动的交通模拟对于自动驾驶系统的训练和评估至关重要。然而，大多数现有的数据驱动仿真方法依赖于静态初始化或日志回放数据，这限制了它们在长时间范围内建模动态变化且不断发展的代理群体的能力。我们提出了InfGen这一场景生成框架，它能够以自回归的方式输出代理的状态和轨迹。InfGen将整个场景表示为一个包括交通信号灯、代理状态和运动向量的标记序列，并采用Transformer模型来模拟随着时间推移的变化中的交通状况。这种设计使得InfGen能够在交通中持续插入新的代理，支持无限场景生成。实验表明，InfGen能够产生现实性强、多样化且适应性好的交通行为。此外，在InfGen生成的情景中训练的强化学习策略在鲁棒性和泛化能力上表现更佳，验证了它作为自动驾驶高保真模拟环境的有效性。 更多信息可在此网址获取：[请在此处插入链接]
URL
https://arxiv.org/abs/2506.23316
PDF
-
RoomCraft: Controllable and Complete 3D Indoor Scene Generation
2025-06-27 15:03:17Mengqi Zhou, Xipeng Wang, Yuxi Wang, Zhaoxiang ZhangAbstract
Generating realistic 3D indoor scenes from user inputs remains a challenging problem in computer vision and graphics, requiring careful balance of geometric consistency, spatial relationships, and visual realism. While neural generation methods often produce repetitive elements due to limited global spatial reasoning, procedural approaches can leverage constraints for controllable generation but struggle with multi-constraint scenarios. When constraints become numerous, object collisions frequently occur, forcing the removal of furniture items and compromising layout completeness. To address these limitations, we propose RoomCraft, a multi-stage pipeline that converts real images, sketches, or text descriptions into coherent 3D indoor scenes. Our approach combines a scene generation pipeline with a constraint-driven optimization framework. The pipeline first extracts high-level scene information from user inputs and organizes it into a structured format containing room type, furniture items, and spatial relations. It then constructs a spatial relationship network to represent furniture arrangements and generates an optimized placement sequence using a heuristic-based depth-first search (HDFS) algorithm to ensure layout coherence. To handle complex multi-constraint scenarios, we introduce a unified constraint representation that processes both formal specifications and natural language inputs, enabling flexible constraint-oriented adjustments through a comprehensive action space design. Additionally, we propose a Conflict-Aware Positioning Strategy (CAPS) that dynamically adjusts placement weights to minimize furniture collisions and ensure layout completeness. Extensive experiments demonstrate that RoomCraft significantly outperforms existing methods in generating realistic, semantically coherent, and visually appealing room layouts across diverse input modalities.
Abstract (translated)
从用户输入生成逼真的3D室内场景在计算机视觉和图形学领域仍然是一个挑战性问题，需要精细地平衡几何一致性、空间关系以及视觉真实感。虽然神经网络生成方法常因全球空间推理能力有限而产生重复元素，程序化方法可以通过利用约束实现可控生成，但在面对多约束场景时却往往力不从心。当约束数量增多时，物体碰撞频繁发生，导致移除家具物品并损害布局完整性。为了克服这些限制，我们提出了RoomCraft，这是一种将真实图像、草图或文本描述转换为连贯3D室内场景的多阶段流水线方法。我们的方法结合了场景生成管道和基于约束驱动优化框架。 该管道首先从用户输入中提取高层次场景信息，并将其组织成包含房间类型、家具项目和空间关系的结构化格式。然后，它构建一个空间关系网络来表示家具布局安排，并使用启发式深度优先搜索（HDFS）算法生成最佳放置顺序以确保布局一致性。为了处理复杂的多约束场景，我们引入了一种统一的约束表示方法，可以同时处理正式规范和自然语言输入，并通过全面的动作设计实现灵活的基于约束调整。此外，我们还提出了一种冲突感知定位策略(CAPS)，该策略动态调整放置权重以尽量减少家具碰撞并确保布局完整性。 广泛的实验表明，RoomCraft在各种输入模式下显著优于现有方法，在生成逼真、语义连贯且视觉吸引人的房间布局方面表现出色。
URL
https://arxiv.org/abs/2506.22291
PDF
-
SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model
2025-06-27 07:35:04Shuhan Tan, John Lambert, Hong Jeon, Sakshum Kulshrestha, Yijing Bai, Jing Luo, Dragomir Anguelov, Mingxing Tan, Chiyu Max JiangAbstract
The goal of traffic simulation is to augment a potentially limited amount of manually-driven miles that is available for testing and validation, with a much larger amount of simulated synthetic miles. The culmination of this vision would be a generative simulated city, where given a map of the city and an autonomous vehicle (AV) software stack, the simulator can seamlessly simulate the trip from point A to point B by populating the city around the AV and controlling all aspects of the scene, from animating the dynamic agents (e.g., vehicles, pedestrians) to controlling the traffic light states. We refer to this vision as CitySim, which requires an agglomeration of simulation technologies: scene generation to populate the initial scene, agent behavior modeling to animate the scene, occlusion reasoning, dynamic scene generation to seamlessly spawn and remove agents, and environment simulation for factors such as traffic lights. While some key technologies have been separately studied in various works, others such as dynamic scene generation and environment simulation have received less attention in the research community. We propose SceneDiffuser++, the first end-to-end generative world model trained on a single loss function capable of point A-to-B simulation on a city scale integrating all the requirements above. We demonstrate the city-scale traffic simulation capability of SceneDiffuser++ and study its superior realism under long simulation conditions. We evaluate the simulation quality on an augmented version of the Waymo Open Motion Dataset (WOMD) with larger map regions to support trip-level simulation.
Abstract (translated)
交通模拟的目标是通过大量的仿真合成里程来增强有限的手动驾驶测试和验证里程。这一愿景的最终目标是一个生成式的虚拟城市，即给定一个城市的地图和自动驾驶车辆（AV）软件堆栈后，该模拟器能够无缝地从A点到B点进行旅程模拟，并且可以通过填充城市环境并控制场景的所有方面来实现这一点，包括动画化动态代理（例如，车辆、行人）以及管理交通灯状态等。我们将这一愿景称为CitySim，这需要一系列的仿真技术：用于初始化场景生成的技术，用于动画化场景的行为建模技术，遮挡推理，无缝启动和移除代理的动态场景生成技术，以及包括交通灯在内的环境模拟技术。尽管一些关键技术已经在各种研究中分别进行了单独的研究，但其他技术如动态场景生成和环境模拟在学术界却受到了较少的关注。 我们提出了SceneDiffuser++，这是第一个基于单个损失函数训练完成的端到端生成式世界模型，能够在城市规模上实现从A点到B点的仿真，并集成了上述所有需求。我们展示了SceneDiffuser++的城市级交通仿真的能力以及其在长时间模拟条件下的优越现实性。我们在Waymo开放运动数据集（WOMD）的一个增强版本上评估了该模拟的质量，此版本包含了更大的地图区域以支持行程级别的模拟。
URL
https://arxiv.org/abs/2506.21976
PDF
-
XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
2025-06-26 16:04:16Bowen Chen, Mengyi Zhao, Haomiao Sun, Li Chen, Xu Wang, Kang Du, Xinglong WuAbstract
Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.
Abstract (translated)
在从文本生成图像的过程中，实现对主题身份和语义属性（如姿态、风格、光照）的精细控制，特别是在涉及多个主体时，通常会削弱扩散变压器（DiT）的可编辑性和连贯性。许多方法引入了伪影或遭受属性纠缠的问题。为了解决这些挑战，我们提出了一种新的多主题可控生成模型XVerse。通过将参考图像转换为用于特定标记文本流调节的偏移量，XVerse允许对特定主体进行精确且独立的控制，而不破坏图像潜在特征。因此，XVerse提供了高保真度、可编辑的多主体图像合成，并具有强大的个体主体特性和语义属性控制能力。这一进展显著提高了个性化和复杂场景生成的能力。
URL
https://arxiv.org/abs/2506.21416
PDF
-
WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration
2025-06-25 16:28:40Chaojun Ni, Jie Li, Haoyun Li, Hengyu Liu, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Boyuan Wang, Chenxin Li, Guan Huang, Wenjun MeiAbstract
Interactive 3D scene generation from a single image has gained significant attention due to its potential to create immersive virtual worlds. However, a key challenge in current 3D generation methods is the limited explorability, which cannot render high-quality images during larger maneuvers beyond the original viewpoint, particularly when attempting to move forward into unseen areas. To address this challenge, we propose WonderFree, the first model that enables users to interactively generate 3D worlds with the freedom to explore from arbitrary angles and directions. Specifically, we decouple this challenge into two key subproblems: novel view quality, which addresses visual artifacts and floating issues in novel views, and cross-view consistency, which ensures spatial consistency across different viewpoints. To enhance rendering quality in novel views, we introduce WorldRestorer, a data-driven video restoration model designed to eliminate floaters and artifacts. In addition, a data collection pipeline is presented to automatically gather training data for WorldRestorer, ensuring it can handle scenes with varying styles needed for 3D scene generation. Furthermore, to improve cross-view consistency, we propose ConsistView, a multi-view joint restoration mechanism that simultaneously restores multiple perspectives while maintaining spatiotemporal coherence. Experimental results demonstrate that WonderFree not only enhances rendering quality across diverse viewpoints but also significantly improves global coherence and consistency. These improvements are confirmed by CLIP-based metrics and a user study showing a 77.20% preference for WonderFree over WonderWorld enabling a seamless and immersive 3D exploration experience. The code, model, and data will be publicly available.
Abstract (translated)
从单张图像生成交互式3D场景因其在创建沉浸式虚拟世界方面的潜力而备受关注。然而，目前的3D生成方法面临的一个关键挑战是探索性有限，在超出原始视点进行更大范围操作（尤其是尝试进入未见区域时）无法渲染高质量图片。为了解决这一难题，我们提出了WonderFree模型，这是首个允许用户以任意角度和方向互动地生成3D世界的模型。具体来说，我们将此挑战分解成两个关键子问题：新颖视角质量和跨视图一致性。前者关注于解决新颖视角中的视觉伪影和漂浮问题；后者则确保不同视角间的空间一致性。 为了提升新视角下的渲染质量，我们引入了WorldRestorer，这是一个基于数据驱动的视频修复模型，旨在消除漂浮物体及其它视觉伪影。此外，我们还提出了一种自动收集训练数据的数据采集管道，为WorldRestorer提供涵盖各种风格场景所需的数据，以适应3D场景生成的需求。 为了提高跨视图的一致性，我们提出了ConsistView，这是一种多视角联合修复机制，在同时恢复多个视角的同时保持时空一致性。实验结果表明，WonderFree不仅在不同视角下提升了渲染质量，而且还显著提高了全局一致性和连贯性。这些改进通过CLIP（一种基于对比学习的预训练模型）基准测试和用户研究得到了验证，结果显示有77.20%的用户更偏好使用WonderFree而非WonderWorld来获得无缝且沉浸式的3D探索体验。 该方法的相关代码、模型及数据将公开发布。
URL
https://arxiv.org/abs/2506.20590
PDF
-
DreamAnywhere: Object-Centric Panoramic 3D Scene Generation
2025-06-25 12:30:41Edoardo Alberto Dominici, Jozef Hladky, Floor Verhoeven, Lukas Radl, Thomas Deixelberger, Stefan Ainetter, Philipp Drescher, Stefan Hauswiesner, Arno Coomans, Giacomo Nazzaro, Konstantinos Vardis, Markus SteinbergerAbstract
Recent advances in text-to-3D scene generation have demonstrated significant potential to transform content creation across multiple industries. Although the research community has made impressive progress in addressing the challenges of this complex task, existing methods often generate environments that are only front-facing, lack visual fidelity, exhibit limited scene understanding, and are typically fine-tuned for either indoor or outdoor settings. In this work, we address these issues and propose DreamAnywhere, a modular system for the fast generation and prototyping of 3D scenes. Our system synthesizes a 360° panoramic image from text, decomposes it into background and objects, constructs a complete 3D representation through hybrid inpainting, and lifts object masks to detailed 3D objects that are placed in the virtual environment. DreamAnywhere supports immersive navigation and intuitive object-level editing, making it ideal for scene exploration, visual mock-ups, and rapid prototyping -- all with minimal manual modeling. These features make our system particularly suitable for low-budget movie production, enabling quick iteration on scene layout and visual tone without the overhead of traditional 3D workflows. Our modular pipeline is highly customizable as it allows components to be replaced independently. Compared to current state-of-the-art text and image-based 3D scene generation approaches, DreamAnywhere shows significant improvements in coherence in novel view synthesis and achieves competitive image quality, demonstrating its effectiveness across diverse and challenging scenarios. A comprehensive user study demonstrates a clear preference for our method over existing approaches, validating both its technical robustness and practical usefulness.
Abstract (translated)
近期在文本到3D场景生成领域的进展展示了其有潜力彻底改变多个行业的内容创作。尽管研究社区在这项复杂任务上取得了令人瞩目的进步，但现有方法通常只能生成面向前方的环境，并且缺乏视觉真实感、场景理解能力有限，而且通常是为室内或室外环境进行微调的。在本工作中，我们解决了这些问题并提出了DreamAnywhere系统——这是一个用于快速生成和原型设计3D场景的模块化系统。 我们的系统从文本中合成一个360°全景图像，将其分解成背景和物体，并通过混合填充技术构建完整的3D表示。此外，它还能够将对象掩模提升为详细且放置在虚拟环境中的3D对象。DreamAnywhere支持沉浸式导航和直观的对象级编辑功能，非常适合场景探索、视觉模拟以及快速原型设计——所有这些都可以用最少的手动建模完成。 该系统的这些特点特别适合低预算电影制作，因为它允许创作者迅速迭代场景布局和视觉风格，而无需传统3D工作流程带来的额外负担。我们的模块化管道高度可定制，因为它的组件可以独立替换。与目前最先进的基于文本和图像的3D场景生成方法相比，DreamAnywhere在新视角合成的一致性和图像质量方面显示出显著改进，证明了它在各种具有挑战性的场景中的有效性。 一项全面的用户研究显示，相对于现有方法，用户更倾向于使用我们的方法，这验证了其技术稳定性和实际应用价值。
URL
https://arxiv.org/abs/2506.20367
PDF
-
From 2D to 3D Cognition: A Brief Survey of General World Models
2025-06-25 05:05:09Ningwei Xie, Zizi Tian, Lei Yang, Xiao-Ping Zhang, Meng Guo, Jie LiAbstract
World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.
Abstract (translated)
世界模型在人工通用智能（AGI）的发展中获得了越来越多的关注，它们作为计算框架用于学习外部世界的表示并预测未来的状态。早期的努力主要集中在二维视觉感知和模拟上，而最近基于三维感知的生成型世界模型则展示了合成几何一致且可互动的3D环境的能力，标志着向三维空间认知的重大转变。尽管这一领域取得了迅速的进步，但缺乏系统性的分析来分类新兴技术，并明确它们在推进三维认知世界模型中的作用。本综述旨在通过引入概念框架，提供一个结构化和前瞻性的关于从二维感知到三维认知的世界模型的审查，以满足这一需求。 在这个框架内，我们强调了两项关键的技术驱动力：三维表示的进步和世界知识的整合，作为基础支柱。在此基础上，我们将剖析三个核心的认知能力，这些能力支撑着3D世界的建模：3D物理场景生成、3D空间推理以及3D空间互动。此外，我们将进一步探讨在现实应用中部署这些能力的情况，包括具身人工智能、自动驾驶汽车、数字孪生及游戏/虚拟现实等。 最后，我们识别了跨数据、模型和部署层面的挑战，并概述了推进更强大且通用化的三维世界模型的未来方向。
URL
https://arxiv.org/abs/2506.20134
PDF
-
CoCo4D: Comprehensive and Complex 4D Scene Generation
2025-06-24 17:05:44Junwei Zhou, Xueting Li, Lu Qi, Ming-Hsuan YangAbstract
Existing 4D synthesis methods primarily focus on object-level generation or dynamic scene synthesis with limited novel views, restricting their ability to generate multi-view consistent and immersive dynamic 4D scenes. To address these constraints, we propose a framework (dubbed as CoCo4D) for generating detailed dynamic 4D scenes from text prompts, with the option to include images. Our method leverages the crucial observation that articulated motion typically characterizes foreground objects, whereas background alterations are less pronounced. Consequently, CoCo4D divides 4D scene synthesis into two responsibilities: modeling the dynamic foreground and creating the evolving background, both directed by a reference motion sequence. Given a text prompt and an optional reference image, CoCo4D first generates an initial motion sequence utilizing video diffusion models. This motion sequence then guides the synthesis of both the dynamic foreground object and the background using a novel progressive outpainting scheme. To ensure seamless integration of the moving foreground object within the dynamic background, CoCo4D optimizes a parametric trajectory for the foreground, resulting in realistic and coherent blending. Extensive experiments show that CoCo4D achieves comparable or superior performance in 4D scene generation compared to existing methods, demonstrating its effectiveness and efficiency. More results are presented on our website this https URL.
Abstract (translated)
现有的4D合成方法主要集中在对象级别的生成或具有有限新颖视角的动态场景合成上，这限制了它们生成多视角一致和沉浸式的动态4D场景的能力。为了解决这些局限性，我们提出了一种框架（称为CoCo4D），该框架可以根据文本提示生成详细的动态4D场景，并可选择性地包含图像。我们的方法基于一个关键观察：连贯运动通常表征前景对象的特性，而背景的变化则相对较少。因此，CoCo4D将4D场景合成分为两个任务：建模动态前景和创造不断变化的背景，两者都由参考运动序列指导。 给定文本提示及可选参考图像后，CoCo4D首先使用视频扩散模型生成初始运动序列。接着，该运动序列指导动态前景对象和背景的合成过程，采用了一种新颖的渐进式扩展方案（progressive outpainting scheme）。为了确保移动前景物体在动态背景下无缝融合，CoCo4D优化了前景路径的参数化轨迹，从而实现了现实感强且连贯的合成效果。 广泛的实验表明，与现有方法相比，CoCo4D在生成4D场景方面取得了相当或更好的性能表现，证明了其有效性和效率。更多结果可在我们的网站上查看：[此URL]（请将"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2506.19798
PDF
-
PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Scenes
2025-06-23 20:47:18Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas GeigerAbstract
Large-scale 3D semantic scene generation has predominantly relied on voxel-based representations, which are memory-intensive, bound by fixed resolutions, and challenging to edit. In contrast, primitives represent semantic entities using compact, coarse 3D structures that are easy to manipulate and compose, making them an ideal representation for this task. In this paper, we introduce PrITTI, a latent diffusion-based framework that leverages primitives as the main foundational elements for generating compositional, controllable, and editable 3D semantic scene layouts. Our method adopts a hybrid representation, modeling ground surfaces in a rasterized format while encoding objects as vectorized 3D primitives. This decomposition is also reflected in a structured latent representation that enables flexible scene manipulation of ground and object components. To overcome the orientation ambiguities in conventional encoding methods, we introduce a stable Cholesky-based parameterization that jointly encodes object size and orientation. Experiments on the KITTI-360 dataset show that PrITTI outperforms a voxel-based baseline in generation quality, while reducing memory requirements by up to $3\times$. In addition, PrITTI enables direct instance-level manipulation of objects in the scene and supports a range of downstream applications, including scene inpainting, outpainting, and photo-realistic street-view synthesis.
Abstract (translated)
大规模的三维语义场景生成主要依赖于基于体素（voxel）的表示方法，这种方法内存消耗大、分辨率固定且难以编辑。相比之下，原语（primitives）使用紧凑而粗糙的3D结构来表示语义实体，易于操作和组合，成为这一任务的理想选择。在本文中，我们介绍了一种名为PrITTI的新框架，该框架采用基于潜在扩散的方法，并以原语作为生成组成化、可控性和可编辑性三维语义场景布局的主要基础元素。我们的方法采用了混合表示方式，在栅格格式下建模地面表面的同时将物体编码为向量化的3D原语。这种分解也反映在一种结构化的潜在表示中，该表示能够灵活地操作场景中的地面和对象组件。 为了克服传统编码方法中存在的方位模糊性问题，我们引入了一种基于稳定的乔累斯基（Cholesky）参数化的方法，这种方法同时编码物体的大小和方向。实验结果表明，在KITTI-360数据集上的生成质量方面，PrITTI优于基于体素的基础线模型，并且将内存需求减少了最多三倍。此外，PrITTI支持场景中对象实例级别的直接操作，并适用于包括场景修复、扩展以及合成照片级真实街道视图在内的多种下游应用。
URL
https://arxiv.org/abs/2506.19117
PDF
-
VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
2025-06-23 17:59:56Runjia Li, Philip Torr, Andrea Vedaldi, Tomas JakabAbstract
We propose a novel memory mechanism to build video generators that can explore environments interactively. Similar results have previously been achieved by out-painting 2D views of the scene while incrementally reconstructing its 3D geometry, which quickly accumulates errors, or by video generators with a short context window, which struggle to maintain scene coherence over the long term. To address these limitations, we introduce Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by indexing them geometrically based on the 3D surface elements (surfels) they have observed. VMem enables the efficient retrieval of the most relevant past views when generating new ones. By focusing only on these relevant views, our method produces consistent explorations of imagined environments at a fraction of the computational cost of using all past views as context. We evaluate our approach on challenging long-term scene synthesis benchmarks and demonstrate superior performance compared to existing methods in maintaining scene coherence and camera control.
Abstract (translated)
我们提出了一种新颖的记忆机制，用于构建能够进行交互式环境探索的视频生成器。此前曾通过扩展场景的二维视图并逐步重建其三维几何结构来实现类似结果，但这种方法会迅速积累误差，或者使用短上下文窗口的视频生成器难以在长期内保持场景的一致性。为了解决这些限制，我们引入了基于3D表面元素（surfels）几何索引过去视图的视图记忆机制（VMem）。通过这种方式，当生成新视图时，可以高效地检索最相关的过去视图。我们的方法仅关注这些相关视图，在保持计算成本较低的同时，能够产生连贯性较强的想象环境探索结果。 我们在具有挑战性的长期场景合成基准测试中评估了这种方法，并展示了在维持场景一致性和相机控制方面优于现有方法的性能。
URL
https://arxiv.org/abs/2506.18903
PDF
-
TAMMs: Temporal-Aware Multimodal Model for Satellite Image Change Understanding and Forecasting
2025-06-23 17:26:16Zhongbin Guo, Yuhao Wang, Ping Jian, Xinyue Chen, Wei Peng, Ertai EAbstract
Satellite image time-series analysis demands fine-grained spatial-temporal reasoning, which remains a challenge for existing multimodal large language models (MLLMs). In this work, we study the capabilities of MLLMs on a novel task that jointly targets temporal change understanding and future scene generation, aiming to assess their potential for modeling complex multimodal dynamics over time. We propose TAMMs, a Temporal-Aware Multimodal Model for satellite image change understanding and forecasting, which enhances frozen MLLMs with lightweight temporal modules for structured sequence encoding and contextual prompting. To guide future image generation, TAMMs introduces a Semantic-Fused Control Injection (SFCI) mechanism that adaptively combines high-level semantic reasoning and structural priors within an enhanced ControlNet. This dual-path conditioning enables temporally consistent and semantically grounded image synthesis. Experiments demonstrate that TAMMs outperforms strong MLLM baselines in both temporal change understanding and future image forecasting tasks, highlighting how carefully designed temporal reasoning and semantic fusion can unlock the full potential of MLLMs for spatio-temporal understanding.
Abstract (translated)
卫星图像时间序列分析需要精细的时空推理，这对现有的多模态大型语言模型（MLLM）来说仍然是一个挑战。在本项工作中，我们研究了这些模型在一个新颖的任务上的能力，该任务旨在同时理解时间变化并生成未来场景，以评估它们建模复杂多模态动态变化随时间演变的能力。 为此，我们提出了TAMMs，即时空感知的多模态模型（Temporal-Aware Multimodal Model），用于卫星图像变化理解和预测。通过为冻结的MLLM添加轻量级的时间模块，该模型能够进行结构化序列编码和上下文提示增强。为了指导未来图像生成，TAMMs引入了语义融合控制注入机制（Semantic-Fused Control Injection, SFCI）。这种机制在改进后的ControlNet中适应性地结合高级别的语义推理和结构性先验知识。双路径条件处理使得图像合成既具有时间一致性又基于语义。 实验结果表明，TAMMs在时间和变化理解以及未来图像预测任务上都超过了强大的MLLM基准模型，这突显了精心设计的时间推理和语义融合如何解锁MLLM在时空理解方面的全部潜力。
URL
https://arxiv.org/abs/2506.18862
PDF
-
Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation
2025-06-20 17:59:21Xiuyu Yang, Shuhan Tan, Philipp Kr\"ahenb\"uhlAbstract
An ideal traffic simulator replicates the realistic long-term point-to-point trip that a self-driving system experiences during deployment. Prior models and benchmarks focus on closed-loop motion simulation for initial agents in a scene. This is problematic for long-term simulation. Agents enter and exit the scene as the ego vehicle enters new regions. We propose InfGen, a unified next-token prediction model that performs interleaved closed-loop motion simulation and scene generation. InfGen automatically switches between closed-loop motion simulation and scene generation mode. It enables stable long-term rollout simulation. InfGen performs at the state-of-the-art in short-term (9s) traffic simulation, and significantly outperforms all other methods in long-term (30s) simulation. The code and model of InfGen will be released at this https URL
Abstract (translated)
一个理想的交通模拟器能够复制自主驾驶系统在部署期间经历的真实长期点到点的旅程。之前的模型和基准测试主要集中在场景中初始代理的闭环运动仿真上，这不利于长期仿真的实施。随着主车辆进入新的区域，代理会不断进出场景，这就造成了问题。 我们提出了一种名为InfGen的统一模型，它可以执行交错进行的闭环运动模拟与场景生成任务。InfGen能够自动在闭环运动模拟和场景生成模式之间切换，从而实现稳定的长时间仿真流程。该模型在短期（9秒）交通模拟方面达到了当前最佳水平，并且在长期（30秒）模拟中显著超越了所有其他方法。 InfGen的代码和模型将在以下链接发布：[请参见原文链接以获取正确的URL]
URL
https://arxiv.org/abs/2506.17213
PDF
-
DreamCube: 3D Panorama Generation via Multi-plane Synchronization
2025-06-20 17:55:06Yukun Huang, Yanning Zhou, Jianan Wang, Kaiyi Huang, Xihui LiuAbstract
3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.
Abstract (translated)
360度全景图合成是一项既具前景又充满挑战的任务，它要求生成的全方位内容具有高质量且多样的视觉外观和几何结构。现有方法通过利用预训练2D基础模型中的丰富图像先验来克服3D全景数据稀缺的问题，但3D全景与2D单一视图之间的不兼容性限制了这些方法的有效性。在本工作中，我们展示了通过将2D基础模型的操作符应用于多平面同步技术，可以无缝地扩展它们的能力以适用于全方位领域。基于这一设计，我们进一步提出了DreamCube，这是一种用于生成3D全景图的多平面RGB-D扩散模型，它最大化利用了2D基础模型先验知识来实现多样化的外观和准确的几何结构，并同时保持多视角一致性。广泛的实验展示了我们的方法在全景图像生成、全景深度估计以及三维场景生成方面的有效性。
URL
https://arxiv.org/abs/2506.17206
PDF
-
4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation
2025-06-18 23:44:59Chaoyang Wang, Ashkan Mirzaei, Vidit Goel, Willi Menapace, Aliaksandr Siarohin, Avalon Vinella, Michael Vasilkovsky, Ivan Skorokhodov, Vladislav Shakhrai, Sergey Korolev, Sergey Tulyakov, Peter WonkaAbstract
We propose the first framework capable of computing a 4D spatio-temporal grid of video frames and 3D Gaussian particles for each time step using a feed-forward architecture. Our architecture has two main components, a 4D video model and a 4D reconstruction model. In the first part, we analyze current 4D video diffusion architectures that perform spatial and temporal attention either sequentially or in parallel within a two-stream design. We highlight the limitations of existing approaches and introduce a novel fused architecture that performs spatial and temporal attention within a single layer. The key to our method is a sparse attention pattern, where tokens attend to others in the same frame, at the same timestamp, or from the same viewpoint. In the second part, we extend existing 3D reconstruction algorithms by introducing a Gaussian head, a camera token replacement algorithm, and additional dynamic layers and training. Overall, we establish a new state of the art for 4D generation, improving both visual quality and reconstruction capability.
Abstract (translated)
我们提出了一种框架，该框架能够使用前馈架构计算视频帧的四维时空网格和每个时间步的三维高斯粒子。我们的架构主要包含两个部分：一个四维视频模型和一个四维重建模型。 在第一部分中，我们分析了当前的四维视频扩散架构，这些架构在其双流设计中要么顺序地、要么并行地执行空间和时间注意力操作。我们指出现有方法的局限性，并引入了一种新的融合架构，在单层内同时执行空间和时间注意力操作。我们的方法的关键在于一种稀疏注意模式，其中标记物（tokens）可以注意到同一帧内的其他标记物、相同时间戳上的或从相同视角来的标记物。 在第二部分中，我们通过引入高斯头（Gaussian head）、相机标记替换算法以及额外的动态层和训练扩展了现有的三维重建算法。总体而言，我们为四维生成建立了新的业界标准，在视觉质量和重建能力方面都有所提升。
URL
https://arxiv.org/abs/2506.18839
PDF
-
ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies
2025-06-17 08:50:05Jinyan Yuan, Bangbang Yang, Keke Wang, Panwang Pan, Lin Ma, Xuehai Zhang, Xiao Liu, Zhaopeng Cui, Yuewen MaAbstract
Automatic creation of 3D scenes for immersive VR presence has been a significant research focus for decades. However, existing methods often rely on either high-poly mesh modeling with post-hoc simplification or massive 3D Gaussians, resulting in a complex pipeline or limited visual realism. In this paper, we demonstrate that such exhaustive modeling is unnecessary for achieving compelling immersive experience. We introduce ImmerseGen, a novel agent-guided framework for compact and photorealistic world modeling. ImmerseGen represents scenes as hierarchical compositions of lightweight geometric proxies, i.e., simplified terrain and billboard meshes, and generates photorealistic appearance by synthesizing RGBA textures onto these proxies. Specifically, we propose terrain-conditioned texturing for user-centric base world synthesis, and RGBA asset texturing for midground and foreground this http URL reformulation offers several advantages: (i) it simplifies modeling by enabling agents to guide generative models in producing coherent textures that integrate seamlessly with the scene; (ii) it bypasses complex geometry creation and decimation by directly synthesizing photorealistic textures on proxies, preserving visual quality without degradation; (iii) it enables compact representations suitable for real-time rendering on mobile VR headsets. To automate scene creation from text prompts, we introduce VLM-based modeling agents enhanced with semantic grid-based analysis for improved spatial reasoning and accurate asset placement. ImmerseGen further enriches scenes with dynamic effects and ambient audio to support multisensory immersion. Experiments on scene generation and live VR showcases demonstrate that ImmerseGen achieves superior photorealism, spatial coherence and rendering efficiency compared to prior methods. Project webpage: this https URL.
Abstract (translated)
几十年来，自动创建用于沉浸式VR体验的3D场景一直是重要的研究焦点。然而，现有的方法通常依赖于高多边形网格建模后再进行简化处理或使用大量的三维高斯模型，这导致了复杂的流程或是有限的真实感视觉效果。在本文中，我们展示了为了实现令人信服的沉浸式体验，并不需要这种详尽的建模工作。我们引入了ImmerseGen，这是一个新的代理引导框架，用于紧凑且逼真的世界建模。 ImmerseGen将场景表示为轻量级几何代理（即简化的地形和海报网格）的层次组合，并通过合成RGBA纹理在这些代理上生成逼真的外观效果。具体而言，我们提出了基于地形条件的纹理处理方法来合成用户为中心的基本世界的组成元素，以及用于中景和前景元素的RGBA资产纹理化。这种重新构想提供了几个优势：(i) 它简化了建模过程，通过让代理指导生成模型生产与场景无缝集成的连贯纹理；(ii) 无需复杂的几何创建和减面处理，直接在代理上合成逼真的纹理可以保持视觉质量而不退化；(iii) 支持适合移动VR头显实时渲染的紧凑表示形式。 为了从文本提示中自动创建场景，我们引入了增强有语义网格分析功能的VLM建模代理，以改进空间推理和准确的资产放置。ImmerseGen进一步通过动态效果和环境音频来丰富场景，支持多感官沉浸体验。实验结果表明，在场景生成和现场VR演示中，与先前方法相比，ImmerseGen在逼真度、空间一致性以及渲染效率方面都表现出色。 项目网页：[请参阅原文提供的链接]
URL
https://arxiv.org/abs/2506.14315
PDF
-
X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
2025-06-16 14:43:18Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee LeeAbstract
Diffusion models are advancing autonomous driving by enabling realistic data synthesis, predictive end-to-end planning, and closed-loop simulation, with a primary focus on temporally consistent generation. However, the generation of large-scale 3D scenes that require spatial coherence remains underexplored. In this paper, we propose X-Scene, a novel framework for large-scale driving scene generation that achieves both geometric intricacy and appearance fidelity, while offering flexible controllability. Specifically, X-Scene supports multi-granular control, including low-level conditions such as user-provided or text-driven layout for detailed scene composition and high-level semantic guidance such as user-intent and LLM-enriched text prompts for efficient customization. To enhance geometrical and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and the corresponding multiview images, while ensuring alignment between modalities. Additionally, we extend the generated local region into a large-scale scene through consistency-aware scene outpainting, which extrapolates new occupancy and images conditioned on the previously generated area, enhancing spatial continuity and preserving visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as scene exploration. Comprehensive experiments demonstrate that X-Scene significantly advances controllability and fidelity for large-scale driving scene generation, empowering data generation and simulation for autonomous driving.
Abstract (translated)
扩散模型通过实现现实数据合成、预测端到端规划和闭环模拟，在推动自动驾驶技术进步方面发挥了重要作用，尤其是在时间上的一致性生成方面。然而，对于需要空间连贯性的大规模3D场景的生成研究仍较为有限。为此，本文提出了一种名为X-Scene的新框架，用于进行大规模驾驶场景生成，该框架能够同时实现几何复杂性和外观真实性，并提供灵活的可控性。 具体而言，X-Scene支持多粒度控制，包括低级条件（如用户提供的或文本驱动的布局），以构成详细的场景组合；以及高级语义指导（如用户的意图和增强型LLM文本提示）用于高效定制。为了提升几何学和视觉真实感，我们引入了一个统一的管道，在该管道中顺序生成3D语义占用图和对应的多视角图像，并确保不同模式之间的一致性。此外，通过一致性感知场景扩增技术，我们将生成的局部区域扩展为大规模场景，以增强空间连续性和保持视觉连贯性。 最终生成的场景被提升至高质量的3DGS表示形式，支持如场景探索等多样化的应用需求。全面的实验表明，X-Scene显著提升了大规模驾驶场景生成中的可控性和真实感，从而增强了自动驾驶数据生成和模拟的能力。
URL
https://arxiv.org/abs/2506.13558
PDF
-
EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence
2025-06-12 11:43:50Wang Xinjie, Liu Liu, Cao Yu, Wu Ruiqi, Qin Wenkang, Wang Dehui, Sui Wei, Su ZhizhongAbstract
Constructing a physically realistic and accurately scaled simulated 3D world is crucial for the training and evaluation of embodied intelligence tasks. The diversity, realism, low cost accessibility and affordability of 3D data assets are critical for achieving generalization and scalability in embodied AI. However, most current embodied intelligence tasks still rely heavily on traditional 3D computer graphics assets manually created and annotated, which suffer from high production costs and limited realism. These limitations significantly hinder the scalability of data driven approaches. We present EmbodiedGen, a foundational platform for interactive 3D world generation. It enables the scalable generation of high-quality, controllable and photorealistic 3D assets with accurate physical properties and real-world scale in the Unified Robotics Description Format (URDF) at low cost. These assets can be directly imported into various physics simulation engines for fine-grained physical control, supporting downstream tasks in training and evaluation. EmbodiedGen is an easy-to-use, full-featured toolkit composed of six key modules: Image-to-3D, Text-to-3D, Texture Generation, Articulated Object Generation, Scene Generation and Layout Generation. EmbodiedGen generates diverse and interactive 3D worlds composed of generative 3D assets, leveraging generative AI to address the challenges of generalization and evaluation to the needs of embodied intelligence related research. Code is available at this https URL.
Abstract (translated)
构建一个物理上真实且比例准确的三维模拟世界对于具身智能任务的训练和评估至关重要。三维数据资产的多样性、现实性、低成本获取性和可负担性是实现具身人工智能中的泛化和规模扩展的关键因素。然而，目前大多数具身智能任务仍然严重依赖于传统的由人工创建和标注的3D计算机图形资源，这些资源面临着高昂的生产成本和有限的真实感问题。这些问题显著限制了数据驱动方法的可扩展性。 我们提出了一种名为EmbodiedGen的基础平台，该平台用于生成交互式三维世界。它能够在低成本的情况下，大规模地生成高质量、可控且高度逼真的3D资产，并具备准确的物理特性和实际世界的规模（在统一机器人描述格式URDF中）。这些资源可以直接导入到各种物理模拟引擎中以实现细微程度的物理控制，支持下游任务中的训练和评估工作。EmbodiedGen是一个易于使用的全功能工具包，由六个关键模块组成：Image-to-3D、Text-to-3D、Texture Generation（纹理生成）、Articulated Object Generation（连杆对象生成）、Scene Generation（场景生成）和Layout Generation（布局生成）。通过利用生成式AI技术，EmbodiedGen能够创建包含生成式3D资产的多样化且可交互的三维世界，以解决具身智能相关研究中泛化与评估方面的挑战。该代码可在提供的URL处获取。
URL
https://arxiv.org/abs/2506.10600
PDF
-
PIG: Physically-based Multi-Material Interaction with 3D Gaussians
2025-06-09 11:25:21Zeyu Xiao, Zhenyi Wu, Mingyang Sun, Qipeng Yan, Yufan Guo, Zhuoer Liang, Lihua ZhangAbstract
3D Gaussian Splatting has achieved remarkable success in reconstructing both static and dynamic 3D scenes. However, in a scene represented by 3D Gaussian primitives, interactions between objects suffer from inaccurate 3D segmentation, imprecise deformation among different materials, and severe rendering artifacts. To address these challenges, we introduce PIG: Physically-Based Multi-Material Interaction with 3D Gaussians, a novel approach that combines 3D object segmentation with the simulation of interacting objects in high precision. Firstly, our method facilitates fast and accurate mapping from 2D pixels to 3D Gaussians, enabling precise 3D object-level segmentation. Secondly, we assign unique physical properties to correspondingly segmented objects within the scene for multi-material coupled interactions. Finally, we have successfully embedded constraint scales into deformation gradients, specifically clamping the scaling and rotation properties of the Gaussian primitives to eliminate artifacts and achieve geometric fidelity and visual consistency. Experimental results demonstrate that our method not only outperforms the state-of-the-art (SOTA) in terms of visual quality, but also opens up new directions and pipelines for the field of physically realistic scene generation.
Abstract (translated)
3D高斯点阵（Gaussian Splatting）在重建静态和动态的三维场景方面取得了显著的成功。然而，在由3D高斯原语表示的场景中，物体之间的交互受到不准确的三维分割、不同材料间变形的不精确性以及严重的渲染伪影的影响。为了应对这些挑战，我们引入了PIG：基于物理的多材料相互作用与3D高斯方法（Physically-Based Multi-Material Interaction with 3D Gaussians），这是一种结合了快速且精准地从2D像素映射到3D高斯点阵，并在物体级进行精确三维分割的新方法，同时模拟场景内交互对象之间的复杂物理现象。具体而言： 首先，我们的方法支持将二维像素快速、准确地映射为3D高斯，从而实现物体级别的精细三维分割。 其次，在场景中我们为不同材料的对应分割对象分配特定的物理属性，以实现多材料耦合互动。 最后，我们将约束尺度成功嵌入到变形梯度中，特别针对高斯原语的缩放和旋转特性进行了限制，这有助于消除伪影，并达到几何精确性和视觉一致性的目标。 实验结果表明，我们的方法不仅在视觉质量上超过了最先进的技术（SOTA），而且还为物理现实场景生成领域开辟了新的方向和技术途径。
URL
https://arxiv.org/abs/2506.07657
PDF
-
Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
2025-06-09 07:20:49Xiangyu Guo, Zhanqian Wu, Kaixin Xiong, Ziyang Xu, Lijun Zhou, Gangwei Xu, Shaoqing Xu, Haiyang Sun, Bing Wang, Guang Chen, Hangjun Ye, Wenyu Liu, Xinggang WangAbstract
We present Genesis, a unified framework for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. Genesis employs a two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and a BEV-aware LiDAR generator with NeRF-based rendering and adaptive sampling. Both modalities are directly coupled through a shared latent space, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level supervision. Extensive experiments on the nuScenes benchmark demonstrate that Genesis achieves state-of-the-art performance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantic fidelity and practical utility of the generated data.
Abstract (translated)
我们介绍了Genesis，这是一个统一的框架，用于生成多视角驾驶视频和LiDAR序列，并确保它们在时空上和跨模态上的连贯性。Genesis采用两阶段架构，结合了基于DiT（Diffusion Transformer）的视频扩散模型与3D-VAE编码器，以及使用NeRF渲染和自适应采样的BEV感知LiDAR生成器。两种模式通过共享潜在空间直接耦合在一起，从而实现在视觉和几何领域的一致演变。为了用结构化的语义引导生成过程，我们引入了DataCrafter，这是一个基于视觉-语言模型的图像描述模块，能够提供场景级和实例级监督。在nuScenes基准测试上的大量实验表明，Genesis在视频和LiDAR指标（FVD 16.95、FID 4.24、Chamfer 0.611）上均达到了最先进的性能，并且增强了下游任务如分割和3D检测的实用性，验证了生成数据在语义保真度和实际应用中的价值。
URL
https://arxiv.org/abs/2506.07497
PDF
