Scene_Generation
-
SonoWorld: From One Image to a 3D Audio-Visual Scene
2026-03-30 17:57:47Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan GaoAbstract
Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.28757
PDF
-
GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation
2026-03-27 17:58:05Nicolas von L\"utzow, Barbara R\"ossle, Katharina Schmid, Matthias Nie{\ss}nerAbstract
Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.
Abstract (translated)
近期3D生成建模领域的前沿进展大多基于扩散或流匹配（flow-matching）框架。本文则探索了一种完全自回归的替代方案，并提出了GaussianGPT——一种基于Transformer的模型，通过下一个token预测直接生成3D高斯元，从而实现完整的3D场景生成。我们首先使用带有向量量化的稀疏3D卷积自编码器，将高斯基元压缩为离散的潜在网格。所得token被序列化，并采用具备三维旋转位置编码的因果Transformer进行建模，从而支持空间结构与外观的逐序生成。与需要整体优化场景的扩散方法不同，我们的方法以逐步构建的方式生成场景，自然支持场景补全、扩展绘制、通过温度参数实现可控采样，以及灵活的生成范围。该方法在利用自回归建模的组合归纳偏置与可扩展性的同时，操作于与现代神经渲染管线兼容的显式表示，使自回归Transformer成为可控且具备上下文感知能力的3D生成的一种有效补充范式。
URL
https://arxiv.org/abs/2603.26661
PDF
-
I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation
2026-03-24 16:45:40Jia Li, Han Yan, Yihang Chen, Siqi Li, Xibin Song, Yifu Wang, Jianfei Cai, Tien-Tsin Wong, Pan JiAbstract
Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.
Abstract (translated)
尽管视频生成技术已取得显著进展，但在重访先前探索区域时维持长期场景一致性仍具挑战性。现有方案要么依赖显式构建3D几何结构（易产生误差累积与尺度模糊），要么采用简单的相机视场检索（在复杂遮挡场景下通常失效）。为突破这些局限，本文提出I3DM——一种用于生成一致视频场景的新型隐式3D感知记忆机制，该机制无需显式3D重建。其核心在于3D感知记忆检索策略：通过利用预训练前馈 novel view synthesis（FF-NVS）模型的中间特征对视角相关性进行评分，从而在高度遮挡场景中实现鲁棒检索。此外，为充分利用检索到的历史帧，我们设计了3D对齐记忆注入模块。该模块可将历史内容隐式扭曲至目标视角，并自适应地基于可靠扭曲区域进行生成条件控制，从而提升重访一致性与相机控制精度。大量实验表明，本方法超越现有最优方案，在重访一致性、生成保真度及相机控制精度上均取得更优表现。
URL
https://arxiv.org/abs/2603.23413
PDF
-
WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion
2026-03-24 09:10:08Manuel-Andreas Schneider, Angela DaiAbstract
Recent progress in image and video synthesis has inspired their use in advancing 3D scene generation. However, we observe that text-to-image and -video approaches struggle to maintain scene- and object-level consistency beyond a limited environment scale due to the absence of explicit geometry. We thus present a geometry-first approach that decouples this complex problem of large-scale 3D scene synthesis into its structural composition, represented as a mesh scaffold, and realistic appearance synthesis, which leverages powerful image synthesis models conditioned on the mesh scaffold. From an input text description, we first construct a mesh capturing the environment's geometry (walls, floors, etc.), and then use image synthesis, segmentation and object reconstruction to populate the mesh structure with objects in realistic layouts. This mesh scaffold is then rendered to condition image synthesis, providing a structural backbone for consistent appearance generation. This enables scalable, arbitrarily-sized 3D scenes of high object richness and diversity, combining robust 3D consistency with photorealistic detail. We believe this marks a significant step toward generating truly environment-scale, immersive 3D worlds.
Abstract (translated)
图像与视频合成的最新进展推动了其在3D场景生成领域的应用。然而，我们观察到由于缺乏显式几何信息，现有的文生图与文生视频方法在环境规模超出一定范围后，难以保持场景级与对象级的一致性。因此，我们提出一种几何优先的方法，将大规模3D场景合成这一复杂问题解耦为结构组成（以网格支架表示）与真实外观合成（利用基于网格支架条件的强大图像合成模型）。从输入的文本描述出发，我们首先构建捕捉环境几何结构（墙壁、地板等）的网格，随后通过图像合成、分割与对象重建技术，在网格结构中填充布局真实的对象。该网格支架随后被渲染以条件化图像合成，为一致的外观生成提供结构骨架。这使得我们能够生成可扩展、任意尺寸且对象丰富多样的3D场景，在保持稳健3D一致性的同时实现照片级细节。我们相信这标志着向生成真正环境规模、沉浸式3D世界迈出了重要一步。
URL
https://arxiv.org/abs/2603.22972
PDF
-
FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow
2026-03-20 03:15:42Zhifei Yang, Guangyao Zhai, Keyang Lu, YuYang Yin, Chao Zhang, Zhen Xiao, Jieyi Long, Nassir Navab, Yikai WangAbstract
Scene generation has extensive industrial applications, demanding both high realism and precise control over geometry and appearance. Language-driven retrieval methods compose plausible scenes from a large object database, but overlook object-level control and often fail to enforce scene-level style coherence. Graph-based formulations offer higher controllability over objects and inform holistic consistency by explicitly modeling relations, yet existing methods struggle to produce high-fidelity textured results, thereby limiting their practical utility. We present FlowScene, a tri-branch scene generative model conditioned on multimodal graphs that collaboratively generates scene layouts, object shapes, and object textures. At its core lies a tight-coupled rectified flow model that exchanges object information during generation, enabling collaborative reasoning across the graph. This enables fine-grained control of objects' shapes, textures, and relations while enforcing scene-level style coherence across structure and appearance. Extensive experiments show that FlowScene outperforms both language-conditioned and graph-conditioned baselines in terms of generation realism, style consistency, and alignment with human preferences.
Abstract (translated)
场景生成具有广泛的工业应用，既要求高真实感，又需对几何与外观进行精确控制。基于语言检索的方法虽能从大型物体数据库中组合出合理场景，却忽略了物体级控制，且常难以保证场景级风格一致性。基于图的方法通过显式建模关系，在物体控制与整体一致性方面更具优势，但现有方法难以生成高保真纹理结果，限制了其实用价值。本文提出FlowScene，一个基于多模态图条件的三分支场景生成模型，可协同生成场景布局、物体形状与物体纹理。其核心是一个紧密耦合的整流流模型，在生成过程中交换物体信息，实现跨图的协同推理。该方法能在精细控制物体形状、纹理及关系的同时，确保结构与外观层面的场景级风格一致性。大量实验表明，FlowScene在生成真实感、风格一致性及符合人类偏好方面，均优于语言条件与图条件基线方法。
URL
https://arxiv.org/abs/2603.19598
PDF
-
Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors
2026-03-19 11:33:27Jiatong Xia, Zicheng Duan, Anton van den Hengel, Lingqiao LiuAbstract
Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input this http URL practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.18782
PDF
-
OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder
2026-03-17 03:43:37Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Tongliang Liu, Mingming Gong, Jiawang BianAbstract
Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16099
PDF
-
Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
2026-03-16 07:59:39Xingtai Gui, Meijie Zhang, Tianyi Yan, Wencheng Han, Jiahao Gong, Feiyang Tan, Cheng-zhong Xu, Jianbing ShenAbstract
End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14948
PDF
-
Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation
2026-03-14 11:46:27Stefan Ainetter, Thomas Deixelberger, Edoardo A. Dominici, Philipp Drescher, Konstantinos Vardis, Markus SteinbergerAbstract
We present GuidedSceneGen, a text-to-3D generation framework that produces metrically accurate, globally consistent, and semantically interpretable indoor scenes. Unlike prior text-driven methods that often suffer from geometric drift or scale ambiguity, our approach maintains an absolute world coordinate frame throughout the entire generation process. Starting from a textual scene description, we predict a global 3D layout encoding both semantic and geometric structure, which serves as a guiding proxy for downstream stages. A semantics- and depth-conditioned panoramic diffusion model then synthesizes 360° imagery aligned with the global layout, substantially improving spatial coherence. To explore unobserved regions, we employ a video diffusion model guided by optimized camera trajectories that balances coverage and collision avoidance, achieving up to 10x faster sampling compared to exhaustive path exploration. The generated views are fused using 3D Gaussian Splatting, yielding a consistent and fully navigable 3D scene in absolute scale. GuidedSceneGen enables accurate transfer of object poses and semantic labels from layout to reconstruction, and supports progressive scene expansion without re-alignment. Quantitative results and a user study demonstrate greater 3D consistency and layout plausibility compared to recent panoramic text-to-3D baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13910
PDF
-
SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
2026-03-12 17:55:07Jun Luo, Jiaxiang Tang, Ruijie Lu, Gang ZengAbstract
Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at this https URL
Abstract (translated)
从文本生成三维场景是数字内容创作中非常有价值的需求。然而，现有的方法大多局限于特定领域或者依赖于预定义的空间关系，这限制了它们在不受限、开放式词汇表3D场景合成中的应用能力。在这篇论文中，我们介绍了SceneAssistant，这是一个专为开放词汇表的3D场景生成设计的视觉反馈驱动代理。我们的框架结合了现代三维对象生成模型以及Vision-Language Models（VLMs）的空间推理和规划功能。为了支持开放式词汇表的场景组合，我们为VLM提供了全面的基础操作集（例如：缩放、旋转、聚焦等）。在每个交互步骤中，当接收到渲染后的视觉反馈时，VLM会相应地采取行动，并迭代优化场景以实现更连贯的空间排列和更好地与输入文本对齐。实验结果表明，我们的方法可以生成多样化的、开放式词汇表的以及高质量的3D场景。无论是定性分析还是定量的人类评估都证明了我们这种方法优于现有技术。此外，我们的方法允许用户根据自然语言命令指示代理编辑现有的场景。我们的代码可以在提供的网址上获取。 这种翻译旨在概述该论文的主要贡献和技术细节，并且提供了一个对非专业读者友好的版本。
URL
https://arxiv.org/abs/2603.12238
PDF
-
MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks
2026-03-12 05:22:42Lirong Che, Shuo Wen, Shan Huang, Chuang Wang, Yuzhe Yang, Gregory Dudek, Xueqian Wang, Jian SuAbstract
Real-world robotic tasks are long-horizon and often span multiple floors, demanding rich spatial reasoning. However, existing embodied benchmarks are largely confined to single-floor in-house environments, failing to reflect the complexity of real-world tasks. We introduce MANSION, the first language-driven framework for generating building-scale, multi-floor 3D environments. Being aware of vertical structural constraints, MANSION generates realistic, navigable whole-building structures with diverse, human-friendly scenes, enabling the development and evaluation of cross-floor long-horizon tasks. Building on this framework, we release MansionWorld, a dataset of over 1,000 diverse buildings ranging from hospitals to offices, alongside a Task-Semantic Scene Editing Agent that customizes these environments using open-vocabulary commands to meet specific user needs. Benchmarking reveals that state-of-the-art agents degrade sharply in our settings, establishing MANSION as a critical testbed for the next generation of spatial reasoning and planning.
Abstract (translated)
现实中的机器人任务通常涉及较长的时间跨度，并且往往横跨多层楼，需要复杂的空间推理能力。然而，现有的嵌入式基准测试大多局限于单层室内环境，无法反映实际任务的复杂性。为此，我们引入了MANSION，这是一个首屈一指的语言驱动框架，用于生成建筑规模、多楼层的3D环境。考虑到垂直结构约束，MANSION能够生成现实主义强且可导航的整体建筑结构，并包含多样化的人类友好型场景，从而支持跨楼层长时任务的发展和评估。 基于此框架，我们发布了MansionWorld数据集，其中包含了超过1000个多样化的建筑物模型，从医院到办公室应有尽有。此外，我们还提供了一个任务语义场景编辑代理，它能够通过开放式词汇命令自定义这些环境以满足特定用户需求。基准测试表明，在我们的设置中，最先进的代理性能明显下降，这证明MANSION已成为下一代空间推理和规划技术的重要测试平台。
URL
https://arxiv.org/abs/2603.11554
PDF
-
Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image
Abstract
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: this https URL.
Abstract (translated)
当前的图像到三维场景生成方法通过耗时的迭代布局优化或刚性的联合对象-布局生成来构建3D场景。此外，大多数方法依赖于视野有限的透视图图像，这阻碍了创建完整的360度环境。为了解决这些限制，我们设计了一种面向全景图像的有效前馈框架Pano3DComposer。为了将物体生成与布局估计解耦，我们提出了一个即插即用的对象-世界变换预测器。该模块可以将现成的图像到三维模型产生的局部坐标中的三维对象转换为世界坐标系。为此，我们将VGGT架构通过使用目标物裁剪、多视角物体渲染以及相机参数调整为Alignment-VGGT来预测转换。我们利用伪几何监督训练预测器以解决生成和地面真实物体之间的形状差异问题。 对于来自未知领域的输入图像，我们进一步为Pano3DComposer引入了一种粗到细（C2F）对齐机制，该机制通过反馈场景渲染迭代地细化了几何一致性。我们的方法在合成数据集和现实世界数据集中实现了比现有技术更优越的几何精度，在将图片或文本转换成三维模型任务上表现尤为出色。它可以在NVIDIA RTX 4090 GPU上大约20秒内生成一个高保真的3D场景。 项目页面：[这里插入链接，原文中的链接无法直接在此显示]
URL
https://arxiv.org/abs/2603.05908
PDF
-
Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning
2026-03-04 08:22:22Zhengjian Yao, Yongzhi Li, Xinyuan Gao, Quan Chen, Peng Jiang, Yanye LuAbstract
We present "Narrative Weaver", a novel framework that addresses a fundamental challenge in generative AI: achieving multi-modal controllable, long-range, and consistent visual content generation. While existing models excel at generating high-fidelity short-form visual content, they struggle to maintain narrative coherence and visual consistency across extended sequences - a critical limitation for real-world applications such as filmmaking and e-commerce advertising. Narrative Weaver introduces the first holistic solution that seamlessly integrates three essential capabilities: fine-grained control, automatic narrative planning, and long-range coherence. Our architecture combines a Multimodal Large Language Model (MLLM) for high-level narrative planning with a novel fine-grained control module featuring a dynamic Memory Bank that prevents visual drift. To enable practical deployment, we develop a progressive, multi-stage training strategy that efficiently leverages existing pre-trained models, achieving state-of-the-art performance even with limited training data. Recognizing the absence of suitable evaluation benchmarks, we construct and release the E-commerce Advertising Video Storyboard Dataset (EAVSD) - the first comprehensive dataset for this task, containing over 330K high-quality images with rich narrative annotations. Through extensive experiments across three distinct scenarios (controllable multi-scene generation, autonomous storytelling, and e-commerce advertising), we demonstrate our method's superiority while opening new possibilities for AI-driven content creation.
Abstract (translated)
我们介绍了“叙事织工”(Narrative Weaver)，这是一个新颖的框架，旨在解决生成式人工智能中的一个基本挑战：实现多模态可控、长跨度和视觉内容的一致性生成。虽然现有的模型在生成高保真度的短格式视觉内容方面表现出色，但在扩展序列中保持叙述连贯性和视觉一致性方面存在困难——这对于电影制作和电子商务广告等现实世界应用而言是一个关键限制。“叙事织工”引入了第一个全面解决方案，无缝整合三种基本能力：细粒度控制、自动叙述规划以及长跨度的一致性。我们的架构结合了一种多模态大型语言模型（MLLM）用于高层次的叙述规划，并配备了一个具有动态记忆库的新颖细粒度控制模块，以防止视觉漂移。 为了实现实际部署，“叙事织工”开发了一种渐进式的、多阶段训练策略，高效利用现有的预训练模型，在数据有限的情况下也能达到最先进的性能。由于缺乏合适的评估基准，“我们构建并发布了电子商务广告视频故事版数据集（EAVSD）——这是该任务的第一个综合性数据集，包含超过33万张高质量图像及丰富的叙述注释。 通过在三种不同的场景中进行广泛的实验（可控多场景生成、自主叙事和电子商务广告），我们展示了“叙事织工”的优越性能，并开辟了人工智能驱动内容创作的新可能。
URL
https://arxiv.org/abs/2603.06688
PDF
-
WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories
2026-03-02 16:36:56Yisu Zhang, Chenjie Cao, Tengfei Wang, Xuhui Zuo, Junta Wu, Jianke Zhu, Chunchao GuoAbstract
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Abstract (translated)
最近，基础视频扩散模型（VDMs）取得了显著进展。尽管这些模型生成的视频具有出色的视觉质量，但由于相机控制能力有限以及从不同摄像机轨迹观看时内容的一致性较差，重建连贯的三维场景仍然面临挑战。 本文提出了一个新框架WorldStereo，该框架通过两个专门的几何记忆模块将基于相机指导的视频生成与3D重建连接起来。具体而言，全局几何记忆模块实现了精确的相机控制，并且通过增量更新点云来注入粗略的结构先验知识；而空间立体记忆则通过使用三维对应关系约束模型的关注力接收域，使模型能够从内存库中关注细粒度的细节。 这些组件使得WorldStereo能够在精确控制摄像机的情况下生成多视角一致性的视频，并促进了高质量的3D重建。此外，基于灵活控制分支的WorldStereo表现出令人印象深刻的效率，在无需联合训练的前提下受益于分布匹配蒸馏VDM骨干网络。通过针对基于相机指导的视频生成和3D重建基准进行广泛的实验，展示了我们方法的有效性。 值得注意的是，结果显示WorldStereo充当了一个强大的世界模型，能够处理各种场景生成任务（无论从透视图还是全景图像开始），并以高保真度获得三维结果。相关模型将对外发布。
URL
https://arxiv.org/abs/2603.02049
PDF
-
DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
2026-03-02 09:12:40Enhui Ma, Jiahuan Zhang, Guantian Zheng, Tao Tang, Shengbo Eben Li, Yuhang Lu, Xia Zhou, Xueyang Zhang, Yifei Zhan, Kun Zhan, Zhihui Hao, Xianpeng Lang, Kaicheng YuAbstract
Multimodal Large Language Models (MLLMs) are rapidly becoming the intelligence brain of end-to-end autonomous driving systems. A key challenge is to assess whether MLLMs can truly understand and follow complex real-world traffic rules. However, existing benchmarks mainly focus on single-rule scenarios like traffic sign recognition, neglecting the complexity of multi-rule concurrency and conflicts in real driving. Consequently, models perform well on simple tasks but often fail or violate rules in real world complex situations. To bridge this gap, we propose DriveCombo, a text and vision-based benchmark for compositional traffic rule reasoning. Inspired by human drivers' cognitive development, we propose a systematic Five-Level Cognitive Ladder that evaluates reasoning from single-rule understanding to multi-rule integration and conflict resolution, enabling quantitative assessment across cognitive stages. We further propose a Rule2Scene Agent that maps language-based traffic rules to dynamic driving scenes through rule crafting and scene generation, enabling scene-level traffic rule visual reasoning. Evaluations of 14 mainstream MLLMs reveal performance drops as task complexity grows, particularly during rule conflicts. After splitting the dataset and fine-tuning on the training set, we further observe substantial improvements in both traffic rule reasoning and downstream planning capabilities. These results highlight the effectiveness of DriveCombo in advancing compliant and intelligent autonomous driving systems.
Abstract (translated)
多模态大型语言模型（MLLMs）正在迅速成为端到端自动驾驶系统的智能核心。一个关键挑战在于评估这些模型是否能够真正理解并遵守复杂的现实世界交通规则。然而，现有的基准测试主要集中在单一规则场景上，如交通标志识别，忽视了实际驾驶中多重规则共存和冲突的复杂性。因此，虽然模型在简单任务上的表现良好，但在真实世界的复杂情况下往往会出现失误或违反规则的现象。 为了弥合这一差距，我们提出了DriveCombo，这是一个基于文本和视觉的基准测试平台，旨在评估组合型交通规则推理能力。受人类驾驶员认知发展过程启发，我们提出了一套五级认知阶梯系统评价从单一规则理解到多重规则整合与冲突解决的能力，并能够跨认知阶段进行量化评估。此外，我们还提出了Rule2Scene Agent这一概念，通过制定规则和生成场景将基于语言的交通规则映射为动态驾驶场景，从而实现基于场景级别的视觉推理。 对14种主流MLLMs的评测显示，在任务复杂度增加时性能显著下降，特别是在处理规则冲突的情况下。进一步地，在分割数据集并在训练集上进行微调后，我们观察到了交通规则推理和下游规划能力方面的重要改进。这些结果凸显了DriveCombo在推动合规且智能的自动驾驶系统发展方面的有效性。
URL
https://arxiv.org/abs/2603.01637
PDF
-
SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation
2026-02-26 09:19:59Ling Wang, Hao-Xiang Guo, Xinzhou Wang, Fuchun Sun, Kai Sun, Pengkun Liu, Hang Xiao, Zhong Wang, Guangyuan Fu, Eric Li, Yang Liu, Yikai WangAbstract
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at this https URL.
Abstract (translated)
我们介绍了一种名为SceneTransporter的端到端框架，用于从单张图像生成结构化的3D场景。尽管现有方法能够生成部分级别的3D对象，但它们往往无法在开放世界场景中将这些部分组织成不同的实例。通过偏差修正聚类探针，我们揭示了一个关键见解：这种失败的原因在于模型内部分配机制缺乏结构性约束。基于这一发现，我们将结构化3D场景的生成任务重新定义为一个全局相关性分配问题。为此，SceneTransporter在组成式DiT模型去噪循环中制定了并解决了熵最优传输（OT）目标。这种设定施加了两个强大的结构约束：首先，由此产生的运输计划通过控制跨注意力机制来确保图像补丁与部分级别3D潜在特征之间一对一的专有路由，从而防止纠缠；其次，竞争性的运输过程促使相似补丁分组，并且该过程进一步通过基于边的成本进行正则化，以形成连贯的对象并防止碎片化。广泛的实验表明，在开放世界场景生成方面，SceneTransporter超越了现有方法，显著提高了实例级别的连贯性和几何准确性。代码和模型将在[这个链接](https://this-url.com)公开提供。
URL
https://arxiv.org/abs/2602.22785
PDF
-
DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation
2026-02-26 02:42:14Zhechao Wang, Yiming Zeng, Lufan Ma, Zeqing Fu, Chen Bai, Ziyao Lin, Cheng LuAbstract
Synthesis of diverse driving scenes serves as a crucial data augmentation technique for validating the robustness and generalizability of autonomous driving systems. Current methods aggregate high-definition (HD) maps and 3D bounding boxes as geometric conditions in diffusion models for conditional scene generation. However, implicit inter-condition dependency causes generation failures when control conditions change independently. Additionally, these methods suffer from insufficient details in both semantic and structural aspects. Specifically, brief and view-invariant captions restrict semantic contexts, resulting in weak background modeling. Meanwhile, the standard denoising loss with uniform spatial weighting neglects foreground structural details, causing visual distortions and blurriness. To address these challenges, we propose DrivePTS, which incorporates three key innovations. Firstly, our framework adopts a progressive learning strategy to mitigate inter-dependency between geometric conditions, reinforced by an explicit mutual information constraint. Secondly, a Vision-Language Model is utilized to generate multi-view hierarchical descriptions across six semantic aspects, providing fine-grained textual guidance. Thirdly, a frequency-guided structure loss is introduced to strengthen the model's sensitivity to high-frequency elements, improving foreground structural fidelity. Extensive experiments demonstrate that our DrivePTS achieves state-of-the-art fidelity and controllability in generating diverse driving scenes. Notably, DrivePTS successfully generates rare scenes where prior methods fail, highlighting its strong generalization ability.
Abstract (translated)
合成多样化的驾驶场景是验证自动驾驶系统鲁棒性和泛化能力的关键数据增强技术。目前的方法在扩散模型中聚合高精度地图和3D边界框作为几何条件来生成条件场景。然而，隐含的跨条件依赖性导致当控制条件独立变化时产生生成失败的问题。此外，这些方法在语义和结构细节方面存在不足。具体来说，简短且视图不变的描述限制了语义上下文，从而弱化了背景建模效果。同时，标准的空间均匀加权去噪损失忽略了前景结构细节，导致视觉失真和模糊。 为了解决这些问题，我们提出了DrivePTS方法，该方法集成了三个关键创新点。首先，我们的框架采用了一种渐进式学习策略来缓解几何条件之间的相互依赖性，并通过显式的互信息约束加以强化。其次，利用视觉-语言模型生成六个语义方面的多视角层次化描述，提供精细的文本指导。第三，引入了基于频率引导的结构损失以增强模型对高频元素的敏感度，提高前景结构的一致性和真实性。 广泛的实验表明，我们的DrivePTS在生成多样化驾驶场景时达到了最先进的保真度和可控性。特别值得注意的是，DrivePTS成功地生成了一些此前方法无法处理的罕见场景，这突显了其强大的泛化能力。
URL
https://arxiv.org/abs/2602.22549
PDF
-
HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
2026-02-24 20:03:47Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You, Manmohan Chandraker, Ziyu JiangAbstract
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: this https URL .
Abstract (translated)
可控的驾驶场景生成对于现实主义和可扩展性的自主驾驶仿真至关重要，然而现有的方法难以同时实现照片级真实感与精确控制。我们引入了HorizonForge，这是一个统一框架，它将场景重建为可编辑的高斯斑点（Gaussian Splats）和网格（Meshes），从而能够进行细致入微的三维操作，并通过语言驱动的方式插入车辆。这些修改通过一个噪声感知视频扩散过程来渲染，在该过程中强制执行空间和时间一致性，无需每个轨迹优化即可在一次前馈传递中生成多样化的场景变化。 为了标准化评估，我们进一步提出了HorizonSuite，这是一个全面的基准测试平台，涵盖了自我主体级别（ego-level）和代理级别(agent-level)的任务编辑任务，例如轨迹修改和物体操作。广泛的实验表明，高斯网格表示法比其他三维表示方法具有显著更高的保真度，并且来自视频扩散的时间先验对于连贯合成是必不可少的。 结合这些发现，HorizonForge确立了一个简单但强大的范式，用于实现逼真的、可控制的驾驶仿真，在用户偏好和FID（Fréchet Inception Distance）指标上分别比现有最佳方法高出83.4%和25.19%。项目页面：[此链接](https://this-url.com)。 请注意，最后一个句子中的链接需要您替换为实际项目网页的URL。
URL
https://arxiv.org/abs/2602.21333
PDF
-
One2Scene: Geometric Consistent Explorable 3D Scene Generation from a Single Image
2026-02-23 12:15:54Pengfei Wang, Liyi Chen, Zhiyuan Ma, Yanjun Guo, Guowen Zhang, Lei ZhangAbstract
Generating explorable 3D scenes from a single image is a highly challenging problem in 3D vision. Existing methods struggle to support free exploration, often producing severe geometric distortions and noisy artifacts when the viewpoint moves far from the original perspective. We introduce \textbf{One2Scene}, an effective framework that decomposes this ill-posed problem into three tractable sub-tasks to enable immersive explorable scene generation. We first use a panorama generator to produce anchor views from a single input image as initialization. Then, we lift these 2D anchors into an explicit 3D geometric scaffold via a generalizable, feed-forward Gaussian Splatting network. Instead of treating the panorama as a single image for reconstruction, we project it into multiple sparse anchor views and reformulate the reconstruction task as multi-view stereo matching, which allows us to leverage robust geometric priors learned from large-scale multi-view datasets. A bidirectional feature fusion module is used to enforce cross-view consistency, yielding an efficient and geometrically reliable scaffold. Finally, the scaffold serves as a strong prior for a novel view generator to produce photorealistic and geometrically accurate views at arbitrary cameras. By explicitly conditioning on a 3D-consistent scaffold to perform reconstruction, One2Scene works stably under large camera motions, supporting immersive scene exploration. Extensive experiments show that One2Scene substantially outperforms state-of-the-art methods in panorama depth estimation, feed-forward 360° reconstruction, and explorable 3D scene generation. Code and models will be released.
Abstract (translated)
从单张图像生成可探索的三维场景是三维视觉中的一个高度挑战性的问题。现有方法在支持自由探索时往往难以克服，在视角远离原始视点时，会产生严重的几何畸变和噪声伪影。我们引入了**One2Scene**，这是一个有效的框架，它将这一病态问题分解为三个可处理的子任务，以实现沉浸式的可探索场景生成。 首先，使用全景生成器从单张输入图像中创建锚定视图作为初始化。然后，通过一个泛化的前向高斯散射网络将这些二维锚点提升到显式的三维几何支撑结构。不同于将全景视为单一图像进行重建，我们将它投影为多个稀疏的锚定视图，并重新定义了重建任务为多视角立体匹配问题，这使我们能够利用从大规模多视角数据集中学习到的强大几何先验知识。 双向特征融合模块被用来强制执行跨视图一致性，从而生成一个高效且几何上可靠的支撑结构。最后，该支撑结构作为先验条件用于新颖视角生成器中，以便在任意相机位置生成逼真且几何准确的视图。 通过显式地基于三维一致的支撑结构进行重建，One2Scene 在面对大范围相机运动时能稳定工作，支持沉浸式的场景探索。广泛的实验表明，One2Scene 在全景深度估计、前向360°重建和可探索三维场景生成方面显著优于现有的最先进方法。代码与模型将随后发布。
URL
https://arxiv.org/abs/2602.19766
PDF
-
SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
2026-02-21 16:08:17Mohammad Asim, Christopher Wewer, Jan Eric LenssenAbstract
We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.
Abstract (translated)
我们介绍了SceneTok，这是一种新颖的编码器，用于将场景视图集压缩并分解为一组无结构化的、可扩散的标记。现有的三维场景表示和生成方法通常使用3D数据结构或与视角对齐的字段。相比之下，我们首次提出了一种方法，可以将场景信息编码成一小组排列不变性标记，并且这些标记独立于空间网格。通过给定多个上下文视图，SceneTok由一个多视图编码器预测出场景标记，并利用轻量级校正流解码器将其渲染为新视角。 我们展示了这种压缩方法比其他表示方法强1-3个数量级的压缩能力，同时仍能达到业界领先的重建质量。此外，我们的表示可以从新的轨迹进行渲染，包括与输入轨迹偏离的情况，并且显示了解码器能够优雅地处理不确定性。最后，高度压缩的一组无结构潜在场景标记使得在5秒内简单高效地生成场景成为可能，实现了比以往范式更好的质量和速度权衡。
URL
https://arxiv.org/abs/2602.18882
PDF
