Scene_Generation
-
FAIRGAMER: Evaluating Biases in the Application of Large Language Models to Video Games
2025-08-25 09:26:19Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang YaoAbstract
Leveraging their advanced capabilities, Large Language Models (LLMs) demonstrate vast application potential in video games--from dynamic scene generation and intelligent NPC interactions to adaptive opponents--replacing or enhancing traditional game mechanics. However, LLMs' trustworthiness in this application has not been sufficiently explored. In this paper, we reveal that the models' inherent social biases can directly damage game balance in real-world gaming environments. To this end, we present FairGamer, the first bias evaluation Benchmark for LLMs in video game scenarios, featuring six tasks and a novel metrics ${D_lstd}$. It covers three key scenarios in games where LLMs' social biases are particularly likely to manifest: Serving as Non-Player Characters, Interacting as Competitive Opponents, and Generating Game Scenes. FairGamer utilizes both reality-grounded and fully fictional game content, covering a variety of video game genres. Experiments reveal: (1) Decision biases directly cause game balance degradation, with Grok-3 (average ${D_lstd}$ score=0.431) exhibiting the most severe degradation; (2) LLMs demonstrate isomorphic social/cultural biases toward both real and virtual world content, suggesting their biases nature may stem from inherent model characteristics. These findings expose critical reliability gaps in LLMs' gaming applications. Our code and data are available at anonymous GitHub this https URL .
Abstract (translated)
利用其先进的能力，大型语言模型（LLMs）在视频游戏中的应用潜力巨大——从动态场景生成、智能NPC互动到适应性对手，这些都可能替代或增强传统的游戏机制。然而，人们对LLM在这种应用场景中的可信度尚未进行充分探讨。在这篇论文中，我们揭示了模型内在的社会偏见可以直接损害实际游戏环境中的平衡性。为此，我们介绍了FairGamer——首个用于评估视频游戏中LLMs社会偏见的基准测试工具，该工具包括六个任务和一种新的度量指标${D_lstd}$。它涵盖了三个在游戏场景中LLM的社会偏见特别容易表现出来的关键领域：作为非玩家角色、充当竞争对手以及生成游戏场景。FairGamer同时使用了基于现实和完全虚构的游戏内容，涵盖多种视频游戏类型。 实验结果表明： 1. 决策偏差直接导致游戏平衡的恶化，其中Grok-3（平均${D_lstd}$分数=0.431）表现最差。 2. LLM对于真实世界和虚拟世界的主题均展示出同构的社会/文化偏见，这表明它们的偏见可能源自模型本身的特性。这些发现揭示了LLMs在游戏应用中的关键可靠性差距。 我们的代码和数据可在匿名GitHub（此URL不可直接链接，请访问论文获取具体网址）上获得。
URL
https://arxiv.org/abs/2508.17825
PDF
-
SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
2025-08-21 17:59:16Yanxu Meng, Haoning Wu, Ya Zhang, Weidi XieAbstract
3D content generation has recently attracted significant research interest due to its applications in VR/AR and embodied AI. In this work, we address the challenging task of synthesizing multiple 3D assets within a single scene image. Concretely, our contributions are fourfold: (i) we present SceneGen, a novel framework that takes a scene image and corresponding object masks as input, simultaneously producing multiple 3D assets with geometry and texture. Notably, SceneGen operates with no need for optimization or asset retrieval; (ii) we introduce a novel feature aggregation module that integrates local and global scene information from visual and geometric encoders within the feature extraction module. Coupled with a position head, this enables the generation of 3D assets and their relative spatial positions in a single feedforward pass; (iii) we demonstrate SceneGen's direct extensibility to multi-image input scenarios. Despite being trained solely on single-image inputs, our architectural design enables improved generation performance with multi-image inputs; and (iv) extensive quantitative and qualitative evaluations confirm the efficiency and robust generation abilities of our approach. We believe this paradigm offers a novel solution for high-quality 3D content generation, potentially advancing its practical applications in downstream tasks. The code and model will be publicly available at: this https URL.
Abstract (translated)
最近，由于在虚拟现实（VR）、增强现实（AR）和具身人工智能中的应用，三维内容生成引起了广泛的研究兴趣。在这项工作中，我们解决了一个具有挑战性的任务：从单个场景图像中合成多个三维资产。具体来说，我们的贡献有四个方面： (i) 我们提出了SceneGen，一个新颖的框架，它接受场景图像及其对应对象掩码作为输入，并同时产生带有几何形状和纹理的多个三维资产。值得注意的是，SceneGen无需优化或资产检索即可运行； (ii) 我们引入了一个新颖的功能聚合模块，在特征提取模块中整合了来自视觉和几何编码器的局部和全局场景信息。与位置头相结合，这使得在一个前馈传递过程中生成三维资产及其相对空间位置成为可能； (iii) 我们展示了SceneGen在多图像输入场景中的直接扩展性。尽管仅使用单个图像输入进行训练，但我们的架构设计允许使用多图像输入来提高生成性能； (iv) 全面的定量和定性评估确认了我们方法的有效性和鲁棒生成能力。 我们认为这一范式为高质量三维内容生成提供了新颖解决方案，并有可能推进其在下游任务中的实际应用。代码和模型将在以下网址公开发布：[此URL]。
URL
https://arxiv.org/abs/2508.15769
PDF
-
MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion
2025-08-21 02:16:15Xuyang Chen, Zhijun Zhai, Kaixuan Zhou, Zengmao Wang, Jianan He, Dong Wang, Yanfeng Zhang, mingwei Sun, R\"udiger Westermann, Konrad Schindler, Liqiu MengAbstract
Mesh models have become increasingly accessible for numerous cities; however, the lack of realistic textures restricts their application in virtual urban navigation and autonomous driving. To address this, this paper proposes MeSS (Meshbased Scene Synthesis) for generating high-quality, styleconsistent outdoor scenes with city mesh models serving as the geometric prior. While image and video diffusion models can leverage spatial layouts (such as depth maps or HD maps) as control conditions to generate street-level perspective views, they are not directly applicable to 3D scene generation. Video diffusion models excel at synthesizing consistent view sequences that depict scenes but often struggle to adhere to predefined camera paths or align accurately with rendered control videos. In contrast, image diffusion models, though unable to guarantee cross-view visual consistency, can produce more geometry-aligned results when combined with ControlNet. Building on this insight, our approach enhances image diffusion models by improving cross-view consistency. The pipeline comprises three key stages: first, we generate geometrically consistent sparse views using Cascaded Outpainting ControlNets; second, we propagate denser intermediate views via a component dubbed AGInpaint; and third, we globally eliminate visual inconsistencies (e.g., varying exposure) using the GCAlign module. Concurrently with generation, a 3D Gaussian Splatting (3DGS) scene is reconstructed by initializing Gaussian balls on the mesh surface. Our method outperforms existing approaches in both geometric alignment and generation quality. Once synthesized, the scene can be rendered in diverse styles through relighting and style transfer techniques.
Abstract (translated)
网格模型（Mesh models）对于许多城市来说已经变得越来越容易获取；然而，缺乏逼真的纹理限制了它们在虚拟城市导航和自动驾驶中的应用。为了应对这一问题，本文提出了 MeSS (基于网格的场景合成)，旨在利用城市的网格模型作为几何先验条件来生成高质量、风格一致的户外场景。虽然图像和视频扩散模型可以借助空间布局（如深度图或高清地图）作为控制条件来生成街道视角视图，但它们并不直接适用于3D场景生成。视频扩散模型擅长于合成描绘场景的一致性视图序列，但在遵循预定义的相机路径或与渲染的控制视频对齐方面常常遇到困难。相比之下，尽管图像扩散模型无法保证跨视角视觉一致性，但如果结合使用ControlNet，则可以产生更具几何一致性的结果。基于这一洞察，我们的方法通过改进图像扩散模型之间的视角一致性来增强它们的功能。 该流程包含三个关键阶段：首先，我们利用级联Outpainting ControlNets生成几何上一致的稀疏视图；其次，我们将更密集的中间视图通过名为AGInpaint的组件进行传播；第三，我们使用GCAlign模块全局消除视觉不一致性（例如不同的曝光度）。与此同时，在生成过程中，通过在网格表面上初始化高斯球来重建3D高斯点阵（3DGS）场景。我们的方法在几何对齐和生成质量方面都优于现有的方法。一旦合成完成，可以通过重新照明和风格转移技术以多种样式渲染场景。 这种综合方法不仅提升了基于图像扩散模型的三维场景生成能力，还为未来在虚拟城市导航和自动驾驶领域中的应用铺平了道路，使得利用现有网格数据创建逼真的虚拟环境成为可能。
URL
https://arxiv.org/abs/2508.15169
PDF
-
TiP4GEN: Text to Immersive Panorama 4D Scene Generation
2025-08-17 16:02:24Ke Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao WeiAbstract
With the rapid advancement and widespread adoption of VR/AR technologies, there is a growing demand for the creation of high-quality, immersive dynamic scenes. However, existing generation works predominantly concentrate on the creation of static scenes or narrow perspective-view dynamic scenes, falling short of delivering a truly 360-degree immersive experience from any viewpoint. In this paper, we introduce \textbf{TiP4GEN}, an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and synthesizes motion-rich, geometry-consistent panoramic 4D scenes. TiP4GEN integrates panorama video generation and dynamic scene reconstruction to create 360-degree immersive virtual environments. For video generation, we introduce a \textbf{Dual-branch Generation Model} consisting of a panorama branch and a perspective branch, responsible for global and local view generation, respectively. A bidirectional cross-attention mechanism facilitates comprehensive information exchange between the branches. For scene reconstruction, we propose a \textbf{Geometry-aligned Reconstruction Model} based on 3D Gaussian Splatting. By aligning spatial-temporal point clouds using metric depth maps and initializing scene cameras with estimated poses, our method ensures geometric consistency and temporal coherence for the reconstructed scenes. Extensive experiments demonstrate the effectiveness of our proposed designs and the superiority of TiP4GEN in generating visually compelling and motion-coherent dynamic panoramic scenes. Our project page is at this https URL.
Abstract (translated)
随着虚拟现实（VR）和增强现实（AR）技术的迅速发展及其广泛应用，对高质量、沉浸式动态场景的需求日益增长。然而，现有的生成工作主要集中在静态场景或窄视角动态场景的创建上，无法提供从任意视点出发的真实360度沉浸式体验。在本文中，我们介绍了**TiP4GEN**，这是一个先进的文本到动态全景场景生成框架，能够实现精细的内容控制，并合成富于动作、几何一致性的全景4D场景。TiP4GEN结合了全景视频生成和动态场景重建技术，以创建360度的沉浸式虚拟环境。 对于视频生成部分，我们提出了一种**双分支生成模型**，包括一个用于全局视图生成的全景支路和一个用于局部视图生成的透视支路。双向交叉注意机制促进了两个分支之间全面的信息交换。 在场景重建方面，我们提出了基于三维高斯点云的**几何对齐重构模型**。通过使用度量深度地图来对齐时空点云，并利用估计的姿态初始化场景摄像机，我们的方法确保了重建场景的几何一致性和时间连贯性。 广泛的实验验证了我们所提出的方案的有效性和TiP4GEN在生成视觉吸引力强、动作协调的动态全景场景方面的优越性。我们的项目页面位于此[URL]（请将此处的"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2508.12415
PDF
-
ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
2025-08-15 12:06:55Jingyu Li, Bozhou Zhang, Xin Jin, Jiankang Deng, Xiatian Zhu, Li ZhangAbstract
Autonomous driving requires rich contextual comprehension and precise predictive reasoning to navigate dynamic and complex environments safely. Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge. VLMs provide interpretability and robust action prediction through their ability to understand multi-modal context, while DWMs excel in generating detailed and plausible future driving scenarios essential for proactive planning. Integrating VLMs with DWMs is an intuitive, promising, yet understudied strategy to exploit the complementary strengths of accurate behavioral prediction and realistic scene generation. Nevertheless, this integration presents notable challenges, particularly in effectively connecting action-level decisions with high-fidelity pixel-level predictions and maintaining computational efficiency. In this paper, we propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer to form a unified imagination-and-planning loop. The driving agent predicts initial driving trajectories based on multi-modal inputs, guiding the scene imaginer to generate corresponding future scenarios. These imagined scenarios are subsequently utilized to iteratively refine the driving agent's planning decisions. To address efficiency and predictive accuracy challenges inherent in this integration, we introduce an early stopping mechanism and a trajectory selection strategy. Extensive experimental validation on the nuScenes and NAVSIM datasets demonstrates the robustness and superiority of ImagiDrive over previous alternatives under both open-loop and closed-loop conditions.
Abstract (translated)
自动驾驶需要丰富的语境理解和精确的预测推理，以安全地导航动态和复杂的环境。视觉语言模型（VLMs）和驾驶世界模型（DWMs）分别在解决这一挑战的不同方面中独立涌现为强有力的解决方案。VLMs通过其理解多模态上下文的能力提供解释性和稳健的动作预测，而DWMs则擅长生成详细且可信的未来驾驶场景，这对于主动规划至关重要。将VLM与DWM结合是一种直观、有前景但研究不足的战略，旨在利用准确的行为预测和逼真场景生成的互补优势。然而，这种整合带来了显著挑战，特别是在有效地连接行动级决策与高保真的像素级预测以及保持计算效率方面。 在本文中，我们提出了ImagiDrive——一种新颖的端到端自动驾驶框架，它将基于VLM的驾驶代理与基于DWM的场景想象器相结合，形成统一的想象和规划循环。该驾驶代理根据多模态输入预测初始行驶轨迹，并指导场景想象器生成相应的未来情景。这些被设想的情景随后用于迭代优化驾驶代理的规划决策。 为了应对这种整合中固有的效率和预测准确性挑战，我们引入了一个早期停止机制和一种轨迹选择策略。在nuScenes和NAVSIM数据集上的广泛实验验证表明，在开环和闭环条件下，ImagiDrive比以前的方法具有更强健性和优越性。
URL
https://arxiv.org/abs/2508.11428
PDF
-
HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
2025-08-07 23:23:07Zixuan Bian, Ruohan Ren, Yue Yang, Chris Callison-BurchAbstract
3D scene generation plays a crucial role in gaming, artistic creation, virtual reality and many other domains. However, current 3D scene design still relies heavily on extensive manual effort from creators, and existing automated methods struggle to generate open-domain scenes or support flexible editing. As a result, generating 3D worlds directly from text has garnered increasing attention. In this paper, we introduce HOLODECK 2.0, an advanced vision-language-guided framework for 3D world generation with support for interactive scene editing based on human feedback. HOLODECK 2.0 can generate diverse and stylistically rich 3D scenes (e.g., realistic, cartoon, anime, and cyberpunk styles) that exhibit high semantic fidelity to fine-grained input descriptions, suitable for both indoor and open-domain environments. HOLODECK 2.0 leverages vision-language models (VLMs) to identify and parse the objects required in a scene and generates corresponding high-quality assets via state-of-the-art 3D generative models. It then iteratively applies spatial constraints derived from the VLMs to achieve semantically coherent and physically plausible layouts. Human evaluations and CLIP-based assessments demonstrate that HOLODECK 2.0 effectively generates high-quality scenes closely aligned with detailed textual descriptions, consistently outperforming baselines across indoor and open-domain scenarios. Additionally, we provide editing capabilities that flexibly adapt to human feedback, supporting layout refinement and style-consistent object edits. Finally, we present a practical application of HOLODECK 2.0 in procedural game modeling, generating visually rich and immersive environments, potentially boosting efficiency.
Abstract (translated)
3D场景生成在游戏、艺术创作、虚拟现实等多个领域中扮演着至关重要的角色。然而，当前的3D场景设计仍然高度依赖于创作者大量的手工劳动，并且现有的自动化方法难以生成开放领域的场景或支持灵活编辑。因此，从文本直接生成3D世界引起了越来越多的关注。在这篇论文中，我们介绍了HOLODECK 2.0，这是一个先进的基于视觉-语言指导的框架，用于生成具有互动场景编辑功能（基于人类反馈）的3D世界。HOLODECK 2.0能够生成多样化且风格丰富的3D场景（例如：写实、卡通、动漫和赛博朋克风格），这些场景与细粒度输入描述的高度语义一致，适用于室内及开放领域环境。通过利用视觉-语言模型(VLMs)，HOLODECK 2.0可以识别并解析场景所需的对象，并借助最先进的3D生成模型来生成相应的高质量资产。然后，它会迭代应用由VLMs推导的空间约束，以实现语义连贯且物理上合理的布局。人类评估和CLIP（对比语言-图像预训练）评估表明，HOLODECK 2.0能够有效生成与详细文本描述高度一致的高质量场景，在室内及开放领域场景中持续优于基准方法。此外，我们提供了灵活适应人类反馈的编辑功能，支持布局调整和风格一致性对象编辑。最后，我们将展示HOLODECK 2.0在程序化游戏建模中的实际应用，生成视觉丰富且沉浸式的环境，有望提高效率。
URL
https://arxiv.org/abs/2508.05899
PDF
-
IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Control
2025-08-06 07:19:16Lijuan Liu, Wenfa Li, Dongbo Zhang, Shuo Wang, Shaohui JiaoAbstract
We present IDC-Net (Image-Depth Consistency Network), a novel framework designed to generate RGB-D video sequences under explicit camera trajectory control. Unlike approaches that treat RGB and depth generation separately, IDC-Net jointly synthesizes both RGB images and corresponding depth maps within a unified geometry-aware diffusion model. The joint learning framework strengthens spatial and geometric alignment across frames, enabling more precise camera control in the generated sequences. To support the training of this camera-conditioned model and ensure high geometric fidelity, we construct a camera-image-depth consistent dataset with metric-aligned RGB videos, depth maps, and accurate camera poses, which provides precise geometric supervision with notably improved inter-frame geometric consistency. Moreover, we introduce a geometry-aware transformer block that enables fine-grained camera control, enhancing control over the generated sequences. Extensive experiments show that IDC-Net achieves improvements over state-of-the-art approaches in both visual quality and geometric consistency of generated scene sequences. Notably, the generated RGB-D sequences can be directly feed for downstream 3D Scene reconstruction tasks without extra post-processing steps, showcasing the practical benefits of our joint learning framework. See more at this https URL.
Abstract (translated)
我们提出了IDC-Net（图像深度一致性网络），这是一种新颖的框架，旨在根据明确的相机轨迹控制生成RGB-D视频序列。与将RGB和深度生成分开处理的方法不同，IDC-Net在一个统一的几何感知扩散模型中联合合成RGB图像及其对应的深度图，从而加强了帧间的空间和几何对齐，使得在生成序列中可以实现更精确的相机控制。 为了支持该条件化相机模型的训练，并确保高几何保真度，我们构建了一个包含度量一致的RGB视频、深度图及准确相机姿态的相机-图像-深度一致性数据集。该数据集提供了精准的几何监督，并且显著提高了帧间几何一致性。此外，我们还引入了一种具有几何感知能力的变压器模块，使细粒度的相机控制成为可能，从而增强对生成序列的控制。 大量的实验表明，在视觉质量和生成场景序列中的几何一致性的方面，IDC-Net相比最先进的方法实现了改进。值得注意的是，生成的RGB-D序列可以直接用于下游3D场景重建任务而无需额外的后处理步骤，展示了我们联合学习框架的实际益处。有关更多信息，请访问此链接：[https URL]（请注意这里的URL应替换为实际提供的链接）。
URL
https://arxiv.org/abs/2508.04147
PDF
-
La La LiDAR: Large-Scale Layout Generation from LiDAR Data
2025-08-05 17:59:55Youquan Liu, Lingdong Kong, Weidong Yang, Xin Li, Ao Liang, Runnan Chen, Ben Fei, Tongliang LiuAbstract
Controllable generation of realistic LiDAR scenes is crucial for applications such as autonomous driving and robotics. While recent diffusion-based models achieve high-fidelity LiDAR generation, they lack explicit control over foreground objects and spatial relationships, limiting their usefulness for scenario simulation and safety validation. To address these limitations, we propose Large-scale Layout-guided LiDAR generation model ("La La LiDAR"), a novel layout-guided generative framework that introduces semantic-enhanced scene graph diffusion with relation-aware contextual conditioning for structured LiDAR layout generation, followed by foreground-aware control injection for complete scene generation. This enables customizable control over object placement while ensuring spatial and semantic consistency. To support our structured LiDAR generation, we introduce Waymo-SG and nuScenes-SG, two large-scale LiDAR scene graph datasets, along with new evaluation metrics for layout synthesis. Extensive experiments demonstrate that La La LiDAR achieves state-of-the-art performance in both LiDAR generation and downstream perception tasks, establishing a new benchmark for controllable 3D scene generation.
Abstract (translated)
可控生成逼真的激光雷达场景对于自动驾驶和机器人技术等应用至关重要。尽管最近基于扩散模型的方案在高保真激光雷达生成方面取得了成功，但这些模型缺乏对前景物体及其空间关系的显式控制，这限制了它们在场景模拟和安全验证中的实用性。为了解决这些问题，我们提出了一种新的布局引导生成框架——大规模布局引导激光雷达生成模型（"La La LiDAR"），该模型引入了语义增强的场景图扩散，并结合了感知上下文关系的情境条件，用于结构化的激光雷达布局生成。随后通过前景感知控制注入来完成完整场景的生成，从而在确保空间和语义一致性的同时实现对象放置的自定义化控制。 为了支持我们的结构化激光雷达生成工作，我们还引入了Waymo-SG和nuScenes-SG两个大规模激光雷达场景图数据集，并提出了新的布局合成评估指标。大量的实验表明，La La LiDAR在激光雷达生成以及下游感知任务中均达到了最先进的性能水平，为可控的3D场景生成设定了一个新的基准标准。
URL
https://arxiv.org/abs/2508.03691
PDF
-
Graph-Guided Dual-Level Augmentation for 3D Scene Segmentation
2025-07-30 13:25:36Hongbin Lin, Yifan Jiang, Juangui Xu, Jesse Jiaxi Xu, Yi Lu, Zhengyu Hu, Ying-Cong Chen, Hao WangAbstract
3D point cloud segmentation aims to assign semantic labels to individual points in a scene for fine-grained spatial understanding. Existing methods typically adopt data augmentation to alleviate the burden of large-scale annotation. However, most augmentation strategies only focus on local transformations or semantic recomposition, lacking the consideration of global structural dependencies within scenes. To address this limitation, we propose a graph-guided data augmentation framework with dual-level constraints for realistic 3D scene synthesis. Our method learns object relationship statistics from real-world data to construct guiding graphs for scene generation. Local-level constraints enforce geometric plausibility and semantic consistency between objects, while global-level constraints maintain the topological structure of the scene by aligning the generated layout with the guiding graph. Extensive experiments on indoor and outdoor datasets demonstrate that our framework generates diverse and high-quality augmented scenes, leading to consistent improvements in point cloud segmentation performance across various models.
Abstract (translated)
3D点云分割的目标是为场景中的单个点分配语义标签，以实现精细的三维空间理解。现有的方法通常采用数据增强来减轻大规模标注的工作量。然而，大多数增强策略仅关注局部变换或语义重组，缺乏对场景内全局结构依赖性的考虑。为了克服这一限制，我们提出了一种基于图引导的数据增强框架，该框架具有双层约束机制，用于生成现实的三维场景。我们的方法从真实世界数据中学习对象关系统计信息，并以此构建指导图以进行场景生成。局部级别的约束强制执行几何合理性和对象之间的语义一致性，而全局级别的约束通过使生成的布局与指导图对齐来保持场景的拓扑结构。在室内和室外数据集上进行了广泛的实验，结果表明我们的框架能够生成多样化且高质量的数据增强场景，并能在各种模型中显著提升点云分割性能。
URL
https://arxiv.org/abs/2507.22668
PDF
-
TopoLiDM: Topology-Aware LiDAR Diffusion Models for Interpretable and Realistic LiDAR Point Cloud Generation
2025-07-30 08:02:42Jiuming Liu, Zheng Huang, Mengmeng Liu, Tianchen Deng, Francesco Nex, Hao Cheng, Hesheng WangAbstract
LiDAR scene generation is critical for mitigating real-world LiDAR data collection costs and enhancing the robustness of downstream perception tasks in autonomous driving. However, existing methods commonly struggle to capture geometric realism and global topological consistency. Recent LiDAR Diffusion Models (LiDMs) predominantly embed LiDAR points into the latent space for improved generation efficiency, which limits their interpretable ability to model detailed geometric structures and preserve global topological consistency. To address these challenges, we propose TopoLiDM, a novel framework that integrates graph neural networks (GNNs) with diffusion models under topological regularization for high-fidelity LiDAR generation. Our approach first trains a topological-preserving VAE to extract latent graph representations by graph construction and multiple graph convolutional layers. Then we freeze the VAE and generate novel latent topological graphs through the latent diffusion models. We also introduce 0-dimensional persistent homology (PH) constraints, ensuring the generated LiDAR scenes adhere to real-world global topological structures. Extensive experiments on the KITTI-360 dataset demonstrate TopoLiDM's superiority over state-of-the-art methods, achieving improvements of 22.6% lower Frechet Range Image Distance (FRID) and 9.2% lower Minimum Matching Distance (MMD). Notably, our model also enables fast generation speed with an average inference time of 1.68 samples/s, showcasing its scalability for real-world applications. We will release the related codes at this https URL.
Abstract (translated)
LiDAR场景生成对于降低真实世界中的激光雷达数据采集成本以及增强自动驾驶下游感知任务的鲁棒性至关重要。然而，现有方法通常难以捕捉几何真实性及全局拓扑一致性。最近提出的激光雷达扩散模型（LiDMs）主要通过将激光雷达点嵌入到潜在空间来提高生成效率，这限制了它们在建模详细几何结构和保持全局拓扑一致方面的解释能力。为了解决这些挑战，我们提出了一种新框架TopoLiDM，该框架整合了图神经网络（GNNs）与扩散模型，并通过拓扑正则化实现高保真度的激光雷达生成。 我们的方法首先训练一个保持拓扑结构的变分自编码器（VAE），通过构建图和多个图卷积层来提取潜在图表示。然后，我们将该VAE冻结并利用潜扩散模型生成新的潜在拓扑图。此外，我们还引入了零维持久同调（PH）约束，确保所生成的激光雷达场景符合真实世界的全局拓扑结构。 在KITTI-360数据集上进行的大量实验表明，TopoLiDM相比现有最先进方法有明显优势，在Frechet范围图像距离（FRID）和最小匹配距离（MMD）方面分别取得了22.6%和9.2%的改进。值得注意的是，我们的模型还具备快速生成速度，平均推断时间为每秒1.68个样本，展示了其在实际应用中的可扩展性。 相关代码将在以下网址发布：[链接] (请提供URL)。
URL
https://arxiv.org/abs/2507.22454
PDF
-
ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment
2025-07-25 08:21:12Chong Xia, Shengjun Zhang, Fangfu Liu, Chang Liu, Khodchaphun Hirunyaratsameewong, Yueqi DuanAbstract
Perpetual 3D scene generation aims to produce long-range and coherent 3D view sequences, which is applicable for long-term video synthesis and 3D scene reconstruction. Existing methods follow a "navigate-and-imagine" fashion and rely on outpainting for successive view expansion. However, the generated view sequences suffer from semantic drift issue derived from the accumulated deviation of the outpainting module. To tackle this challenge, we propose ScenePainter, a new framework for semantically consistent 3D scene generation, which aligns the outpainter's scene-specific prior with the comprehension of the current scene. To be specific, we introduce a hierarchical graph structure dubbed SceneConceptGraph to construct relations among multi-level scene concepts, which directs the outpainter for consistent novel views and can be dynamically refined to enhance diversity. Extensive experiments demonstrate that our framework overcomes the semantic drift issue and generates more consistent and immersive 3D view sequences. Project Page: this https URL.
Abstract (translated)
持续的3D场景生成旨在生成长跨度且连贯的3D视图序列，这适用于长期视频合成和3D场景重建。现有方法遵循“导航与想象”的方式，并依赖于扩展（outpainting）来实现连续视图的扩展。然而，所生成的视图序列会因累积误差导致语义漂移问题。为解决这一挑战，我们提出了ScenePainter，这是一个新的框架，用于生成语义上一致的3D场景，它通过将扩展器的特定于场景的先验知识与当前场景的理解进行对齐来实现这一目标。 具体而言，我们引入了一种层次化的图结构，名为SceneConceptGraph，用以构建多级场景概念之间的关系。这种图结构可以指导生成器产生一致的新视图，并且可以通过动态细化增强多样性。 广泛的实验表明，我们的框架能够克服语义漂移问题，并生成更为一致和沉浸式的3D视图序列。 项目页面：[此链接](this https URL)。
URL
https://arxiv.org/abs/2507.19058
PDF
-
Enhancing Scene Transition Awareness in Video Generation via Post-Training
2025-07-24 02:50:26Hanwen Shen, Jiajie Lu, Yupeng Cao, Xiaonan YangAbstract
Recent advances in AI-generated video have shown strong performance on \emph{text-to-video} tasks, particularly for short clips depicting a single scene. However, current models struggle to generate longer videos with coherent scene transitions, primarily because they cannot infer when a transition is needed from the prompt. Most open-source models are trained on datasets consisting of single-scene video clips, which limits their capacity to learn and respond to prompts requiring multiple scenes. Developing scene transition awareness is essential for multi-scene generation, as it allows models to identify and segment videos into distinct clips by accurately detecting transitions. To address this, we propose the \textbf{Transition-Aware Video} (TAV) dataset, which consists of preprocessed video clips with multiple scene transitions. Our experiment shows that post-training on the \textbf{TAV} dataset improves prompt-based scene transition understanding, narrows the gap between required and generated scenes, and maintains image quality.
Abstract (translated)
最近在AI生成视频领域的进展显示，在将文本转换为短视频的任务中表现出色，尤其是对于描绘单一场景的短片段。然而，当前的模型难以生成包含连贯场景过渡的较长视频，因为它们无法从提示中推断出何时需要进行场景切换。大多数开源模型都是基于由单个场景组成的视频剪辑的数据集训练出来的，这限制了它们对要求多场景响应的提示的学习和应对能力。为了提高多场景生成的效果，发展场景转换意识至关重要，它使模型能够准确地检测并分割视频中的不同片段。为了解决这个问题，我们提出了**过渡感知视频（Transition-Aware Video, TAV）数据集**，该数据集包含具有多个场景转换的预处理视频剪辑。我们的实验表明，在TAV数据集上进行后期训练可以改善基于提示的场景转换理解能力，缩小所需场景与生成场景之间的差距，并保持图像质量。
URL
https://arxiv.org/abs/2507.18046
PDF
-
Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization
2025-07-20 06:59:42Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
Abstract (translated)
近年来，3D生成技术在学术界和工业界都取得了重大进展。然而，从单张RGB图像生成3D场景仍然是一个巨大的挑战，因为现有方法常常难以同时保证多对象场景中的物体生成质量和场景一致性。为了克服这些限制，我们提出了一种新颖的三阶段框架，用于通过单一图像引导模型生成和空间布局优化来创建具有显式几何表示和高质量纹理细节的3D场景。 我们的方法首先进行图像实例分割与修复阶段，在此过程中恢复输入图片中被遮挡物体缺失的详细信息，从而实现前景3D资产的完整生成。随后，我们通过构建伪立体视角来进行相机参数估计和场景深度推断，以此捕捉参考图的空间几何结构，并采用模型选择策略确保先前步骤产生的3D资产与输入图像的最佳对齐。最后，通过对模型进行参数化处理并最小化3D和2D空间中点云之间的Chamfer距离，我们的方法优化布局参数，从而生成一个显式的3D场景表示，保持精确地与指导输入图一致。 在多对象场景图像集上的广泛实验表明，我们所提出的方法不仅在单个生成的3D模型的几何准确性和纹理保真度方面超越了最先进的技术，还在场景布局合成方面具有显著优势。
URL
https://arxiv.org/abs/2507.14841
PDF
-
DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation
2025-07-18 14:45:54Haoran Li, Yuli Tian, Kun Lan, Yong Liao, Lin Wang, Pan Hui, Peng Yuan ZhouAbstract
Generating 3D scenes from natural language holds great promise for applications in gaming, film, and design. However, existing methods struggle with automation, 3D consistency, and fine-grained control. We present DreamScene, an end-to-end framework for high-quality and editable 3D scene generation from text or dialogue. DreamScene begins with a scene planning module, where a GPT-4 agent infers object semantics and spatial constraints to construct a hybrid graph. A graph-based placement algorithm then produces a structured, collision-free layout. Based on this layout, Formation Pattern Sampling (FPS) generates object geometry using multi-timestep sampling and reconstructive optimization, enabling fast and realistic synthesis. To ensure global consistent, DreamScene employs a progressive camera sampling strategy tailored to both indoor and outdoor settings. Finally, the system supports fine-grained scene editing, including object movement, appearance changes, and 4D dynamic motion. Experiments demonstrate that DreamScene surpasses prior methods in quality, consistency, and flexibility, offering a practical solution for open-domain 3D content creation. Code and demos are available at this https URL.
Abstract (translated)
从自然语言生成三维场景在游戏、电影和设计等领域具有巨大的潜力。然而，现有的方法在自动化、三维一致性以及精细控制方面存在挑战。我们提出了DreamScene，这是一个端到端的框架，用于根据文本或对话生成高质量且可编辑的三维场景。DreamScene 以场景规划模块开始，其中GPT-4代理推断出对象语义和空间约束，并构建了一个混合图。随后，基于图的放置算法产生一个结构化且无碰撞的布局。在此布局的基础上，形成模式采样（FPS）通过多时间步采样和重构优化生成物体几何形状，从而实现快速而逼真的合成。为了确保全局一致性，DreamScene 采用了一种针对室内和室外场景量身定制的渐进式相机采样策略。最后，该系统支持精细的场景编辑功能，包括对象移动、外观变化以及四维动态运动。实验表明，DreamScene 在质量、一致性和灵活性方面超越了先前的方法，为开放领域的三维内容创作提供了一个实用解决方案。代码和演示可在[此链接](https://this-url.com)获取。
URL
https://arxiv.org/abs/2507.13985
PDF
-
World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving
2025-07-17 03:34:54Yanchen Guan, Haicheng Liao, Chengyue Wang, Xingcheng Liu, Jiaxun Zhang, Zhenning LiAbstract
Reliable anticipation of traffic accidents is essential for advancing autonomous driving systems. However, this objective is limited by two fundamental challenges: the scarcity of diverse, high-quality training data and the frequent absence of crucial object-level cues due to environmental disruptions or sensor deficiencies. To tackle these issues, we propose a comprehensive framework combining generative scene augmentation with adaptive temporal reasoning. Specifically, we develop a video generation pipeline that utilizes a world model guided by domain-informed prompts to create high-resolution, statistically consistent driving scenarios, particularly enriching the coverage of edge cases and complex interactions. In parallel, we construct a dynamic prediction model that encodes spatio-temporal relationships through strengthened graph convolutions and dilated temporal operators, effectively addressing data incompleteness and transient visual noise. Furthermore, we release a new benchmark dataset designed to better capture diverse real-world driving risks. Extensive experiments on public and newly released datasets confirm that our framework enhances both the accuracy and lead time of accident anticipation, offering a robust solution to current data and modeling limitations in safety-critical autonomous driving applications.
Abstract (translated)
可靠地预测交通事故对于推进自动驾驶系统的发展至关重要。然而，这一目标受到了两个基本挑战的限制：高质量多样化训练数据的稀缺性以及由于环境干扰或传感器缺陷而导致的重要对象级线索经常缺失的问题。为了解决这些问题，我们提出了一种结合生成场景增强和自适应时间推理的全面框架。 具体而言，我们开发了一个视频生成管道，该管道利用由领域信息提示指导的世界模型来创建高分辨率且统计上一致的驾驶场景，并特别丰富了边缘案例和复杂交互的情况。同时，我们构建了一个动态预测模型，通过加强的图卷积和膨胀的时间操作符编码空间-时间关系，有效地解决了数据不完整性和短暂视觉噪声的问题。 此外，我们发布了一套新的基准数据集，旨在更好地捕捉各种现实世界驾驶风险。在公开数据集和新发布的数据集上的广泛实验表明，我们的框架增强了事故预测的准确性和提前期，为安全关键性的自动驾驶应用提供了应对当前数据和建模限制的有效解决方案。
URL
https://arxiv.org/abs/2507.12762
PDF
-
Foundation Model Driven Robotics: A Comprehensive Review
2025-07-14 09:13:07Muhammad Tayyab Khan, Ammar WaheedAbstract
The rapid emergence of foundation models, particularly Large Language Models (LLMs) and Vision-Language Models (VLMs), has introduced a transformative paradigm in robotics. These models offer powerful capabilities in semantic understanding, high-level reasoning, and cross-modal generalization, enabling significant advances in perception, planning, control, and human-robot interaction. This critical review provides a structured synthesis of recent developments, categorizing applications across simulation-driven design, open-world execution, sim-to-real transfer, and adaptable robotics. Unlike existing surveys that emphasize isolated capabilities, this work highlights integrated, system-level strategies and evaluates their practical feasibility in real-world environments. Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust. The review concludes with a roadmap for future research aimed at bridging semantic reasoning and physical intelligence through more robust, interpretable, and embodied models.
Abstract (translated)
基础模型（尤其是大型语言模型和视觉-语言模型）的迅速发展为机器人技术带来了变革性的范式。这些模型提供了强大的语义理解、高级推理和跨模态泛化能力，从而在感知、规划、控制以及人机交互方面取得了重大进展。这篇批判性综述系统地总结了最近的发展，并根据模拟驱动设计、开放环境执行、从模拟到现实的迁移以及适应性机器人等应用对这些发展进行了分类。与强调孤立功能的现有调查不同，这项工作突出了集成的、系统级策略，并评估了它们在实际环境中的可行性。 文中还讨论了一些关键的技术趋势，如程序化场景生成、政策泛化和多模态推理，同时指出了核心瓶颈问题，包括有限的身体化能力、缺乏多模态数据、安全风险以及计算限制。通过这一视角，本文不仅识别出基于基础模型的机器人架构的优势和局限性，还强调了实时操作、实体化、弹性和信任方面的开放挑战。 综述最后提出了一条未来研究路线图，旨在通过更强大、可解释且具身体化的模型来弥合语义推理与物理智能之间的差距。
URL
https://arxiv.org/abs/2507.10087
PDF
-
$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
2025-07-12 05:14:39Zhimin Liao, Ping Wei, Ruijie Zhang, Shuaijia Chen, Haoxuan Wang, Ziyang RenAbstract
Forecasting the evolution of 3D scenes and generating unseen scenarios via occupancy-based world models offers substantial potential for addressing corner cases in autonomous driving systems. While tokenization has revolutionized image and video generation, efficiently tokenizing complex 3D scenes remains a critical challenge for 3D world models. To address this, we propose $I^{2}$-World, an efficient framework for 4D occupancy forecasting. Our method decouples scene tokenization into intra-scene and inter-scene tokenizers. The intra-scene tokenizer employs a multi-scale residual quantization strategy to hierarchically compress 3D scenes while preserving spatial details. The inter-scene tokenizer residually aggregates temporal dependencies across timesteps. This dual design preserves the compactness of 3D tokenizers while retaining the dynamic expressiveness of 4D tokenizers. Unlike decoder-only GPT-style autoregressive models, $I^{2}$-World adopts an encoder-decoder architecture. The encoder aggregates spatial context from the current scene and predicts a transformation matrix to enable high-level control over scene generation. The decoder, conditioned on this matrix and historical tokens, ensures temporal consistency during generation. Experiments demonstrate that $I^{2}$-World achieves state-of-the-art performance, outperforming existing methods by 25.1\% in mIoU and 36.9\% in IoU for 4D occupancy forecasting while exhibiting exceptional computational efficiency: it requires merely 2.9 GB of training memory and achieves real-time inference at 37.0 FPS. Our code is available on this https URL.
Abstract (translated)
基于占用的模型预测三维场景的发展并生成未见过的情景，为解决自动驾驶系统中的边缘情况提供了巨大的潜力。虽然标记化技术在图像和视频生成领域带来了革命性的变革，但高效地对复杂的三维场景进行标记仍然是三维世界模型面临的一个关键挑战。为此，我们提出了$I^{2}$-World，这是一个用于四维占用预测的高效框架。我们的方法将场景标记分解为内部场景标记器（intra-scene tokenizer）和跨场景标记器（inter-scene tokenizer）。 **内部场景标记器**采用多尺度残差量化策略来分层压缩三维场景，同时保留空间细节；而**跨场景标记器**则通过逐时间步的方式聚合时序依赖性。这种双管齐下的设计在保持3D标记器的紧凑性的同时，还保留了4D标记器的动态表达能力。 与解码器独占的GPT风格的自回归模型不同，$I^{2}$-World采用了一种编码器-解码器架构。编码器从当前场景中聚合空间上下文，并预测一个转换矩阵以实现对场景生成的高级控制；而解码器则根据该矩阵和历史标记，在生成过程中确保时间一致性。 实验表明，$I^{2}$-World在四维占用预测方面取得了最先进的性能，与现有方法相比，其mIoU（平均交并比）提高了25.1%，IoU（交并比）提高了36.9%。此外，它还展示了卓越的计算效率：仅需2.9 GB的训练内存，并在实时推理中达到每秒37.0帧的速度。 我们的代码可以在以下链接获取: [请在此处插入URL]。
URL
https://arxiv.org/abs/2507.09144
PDF
-
Acquiring and Adapting Priors for Novel Tasks via Neural Meta-Architectures
2025-07-07 18:48:09Sudarshan BabuAbstract
The ability to transfer knowledge from prior experiences to novel tasks stands as a pivotal capability of intelligent agents, including both humans and computational models. This principle forms the basis of transfer learning, where large pre-trained neural networks are fine-tuned to adapt to downstream tasks. Transfer learning has demonstrated tremendous success, both in terms of task adaptation speed and performance. However there are several domains where, due to lack of data, training such large pre-trained models or foundational models is not a possibility - computational chemistry, computational immunology, and medical imaging are examples. To address these challenges, our work focuses on designing architectures to enable efficient acquisition of priors when large amounts of data are unavailable. In particular, we demonstrate that we can use neural memory to enable adaptation on non-stationary distributions with only a few samples. Then we demonstrate that our hypernetwork designs (a network that generates another network) can acquire more generalizable priors than standard networks when trained with Model Agnostic Meta-Learning (MAML). Subsequently, we apply hypernetworks to 3D scene generation, demonstrating that they can acquire priors efficiently on just a handful of training scenes, thereby leading to faster text-to-3D generation. We then extend our hypernetwork framework to perform 3D segmentation on novel scenes with limited data by efficiently transferring priors from earlier viewed scenes. Finally, we repurpose an existing molecular generative method as a pre-training framework that facilitates improved molecular property prediction, addressing critical challenges in computational immunology
Abstract (translated)
将先前经验中的知识转移到新任务的能力是智能代理（包括人类和计算模型）的一个关键能力。这一原则构成了迁移学习的基础，在这种学习中，大型预训练神经网络被微调以适应下游任务。迁移学习在任务适应速度和性能方面都取得了巨大的成功。然而，在某些领域，由于数据缺乏，训练如此大规模的预训练模型或基础模型是不可能的——计算化学、计算免疫学和医学成像就是例子。为了解决这些挑战，我们的工作重点是设计架构，以便在大量数据不可用的情况下高效地获取先验知识。特别地，我们展示了如何使用神经记忆来适应非平稳分布，并且仅需少量样本即可实现这一目标。然后，我们证明了当使用模型无关元学习（MAML）训练时，我们的超网络设计（生成另一个网络的网络）可以比标准网络获得更通用的先验知识。随后，我们将超网络应用于三维场景生成，展示了它们只需少量训练场景就能高效地获取先验知识，从而加快从文本到3D生成的速度。最后，我们重新利用现有的分子生成方法作为预训练框架，以改善分子性质预测，解决了计算免疫学中的关键挑战。
URL
https://arxiv.org/abs/2507.10446
PDF
-
Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion
2025-07-06 14:30:08Tongyan Hua, Lutao Jiang, Ying-Cong Chen, Wufan ZhaoAbstract
Recent advancements in generative models have enabled 3D urban scene generation from satellite imagery, unlocking promising applications in gaming, digital twins, and beyond. However, most existing methods rely heavily on neural rendering techniques, which hinder their ability to produce detailed 3D structures on a broader scale, largely due to the inherent structural ambiguity derived from relatively limited 2D observations. To address this challenge, we propose Sat2City, a novel framework that synergizes the representational capacity of sparse voxel grids with latent diffusion models, tailored specifically for our novel 3D city dataset. Our approach is enabled by three key components: (1) A cascaded latent diffusion framework that progressively recovers 3D city structures from satellite imagery, (2) a Re-Hash operation at its Variational Autoencoder (VAE) bottleneck to compute multi-scale feature grids for stable appearance optimization and (3) an inverse sampling strategy enabling implicit supervision for smooth appearance this http URL overcome the challenge of collecting real-world city-scale 3D models with high-quality geometry and appearance, we introduce a dataset of synthesized large-scale 3D cities paired with satellite-view height maps. Validated on this dataset, our framework generates detailed 3D structures from a single satellite image, achieving superior fidelity compared to existing city generation models.
Abstract (translated)
最近在生成模型领域的进展使得从卫星图像生成三维城市场景成为可能，这为游戏、数字孪生等领域开启了新的应用前景。然而，大多数现有方法严重依赖于神经渲染技术，这种技术限制了它们在更广泛范围内产生详细三维结构的能力，主要是由于相对有限的二维观测导致的基本结构模糊性。为了应对这一挑战，我们提出了Sat2City，这是一种新颖的框架，它将稀疏体素网格的表现能力与潜在扩散模型相结合，专为我们的新型3D城市数据集设计。 该方法由三个关键组件组成： 1. 一个级联式潜在扩散框架，逐步从卫星图像中恢复三维城市结构。 2. 在其变分自编码器（VAE）瓶颈处进行的Re-Hash操作，用于计算多尺度特征网格以实现稳定的外观优化。 3. 一种反向采样策略，通过隐性监督使外观平滑化。 为了克服收集具有高质量几何和外观的真实世界城市规模三维模型这一挑战，我们引入了一个由合成大规模3D城市及其卫星视图高度地图组成的新型数据集。在该数据集上验证后，我们的框架能够从单张卫星图像生成详细的三维结构，并且与现有城市生成模型相比，在保真度方面表现更佳。
URL
https://arxiv.org/abs/2507.04403
PDF
-
Voyaging into Unbounded Dynamic Scenes from a Single View
2025-07-05 22:49:25Fengrui Tian, Tianjiao Ding, Jinqi Luo, Hancheng Min, Ren\'e VidalAbstract
This paper studies the problem of generating an unbounded dynamic scene from a single view, which has wide applications in augmented/virtual reality and robotics. Since the scene is changing over time, different generated views need to be consistent with the underlying 3D motions. While previous works learn such consistency by training from multiple views, the generated scene regions are bounded to be close to the training views with limited camera movements. To address this issue, we propose DynamicVoyager that reformulates the dynamic scene generation as a scene outpainting process for new dynamic content. As 2D outpainting models can hardly generate 3D consistent motions from only 2D pixels at a single view, we consider pixels as rays to enrich the pixel input with the ray context, so that the 3D motion consistency can be learned from the ray information. More specifically, we first map the single-view video input to a dynamic point cloud with the estimated video depths. Then we render the partial video at a novel view and outpaint the video with ray contexts from the point cloud to generate 3D consistent motions. We employ the outpainted video to update the point cloud, which is used for scene outpainting from future novel views. Experiments show that our model is able to generate unbounded scenes with consistent motions along fly-through cameras, and the generated contents can be controlled with scene prompts.
Abstract (translated)
本文研究了从单视图生成无边界动态场景的问题，该问题在增强/虚拟现实和机器人技术中具有广泛的应用。由于场景随时间变化，不同生成的视图需要与底层3D运动保持一致。虽然先前的工作通过多视角训练来学习这种一致性，但生成的场景区域被限制在靠近训练视图且相机移动有限的位置上。为了解决这个问题，我们提出了DynamicVoyager，它将动态场景生成重新表述为新动态内容的场景外推过程（scene outpainting）。由于2D外推模型难以仅从单视图的2D像素中生成3D一致的运动，我们将像素视为射线，并通过添加射线上下文来丰富像素输入，从而可以从射线信息中学到3D运动一致性。具体来说，我们首先将单视角视频输入映射为动态点云，使用估计的视频深度信息进行映射。然后，在新视图中渲染部分视频，并利用点云中的射线上下文外推视频以生成一致的3D运动。我们采用外推后的视频来更新点云，并用于从未来的新的视图中进行场景外推。实验表明，我们的模型能够生成沿飞行穿越相机路径具有连续动作的无边界场景，并且可以使用场景提示控制生成的内容。
URL
https://arxiv.org/abs/2507.04183
PDF
