Scene_Generation
-
FilmSceneDesigner: Chaining Set Design for Procedural Film Scene Generation
2025-11-24 14:00:40Zhifeng Xie, Keyi Zhang, Yiye Yan, Yuling Guo, Fan Yang, Jiting Zhou, Mengtian LiAbstract
Film set design plays a pivotal role in cinematic storytelling and shaping the visual atmosphere. However, the traditional process depends on expert-driven manual modeling, which is labor-intensive and time-consuming. To address this issue, we introduce FilmSceneDesigner, an automated scene generation system that emulates professional film set design workflow. Given a natural language description, including scene type, historical period, and style, we design an agent-based chaining framework to generate structured parameters aligned with film set design workflow, guided by prompt strategies that ensure parameter accuracy and coherence. On the other hand, we propose a procedural generation pipeline which executes a series of dedicated functions with the structured parameters for floorplan and structure generation, material assignment, door and window placement, and object retrieval and layout, ultimately constructing a complete film scene from scratch. Moreover, to enhance cinematic realism and asset diversity, we construct SetDepot-Pro, a curated dataset of 6,862 film-specific 3D assets and 733 materials. Experimental results and human evaluations demonstrate that our system produces structurally sound scenes with strong cinematic fidelity, supporting downstream tasks such as virtual previs, construction drawing and mood board creation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19137
PDF
-
A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation
2025-11-24 11:32:15Wentao Qu, Guofeng Mei, Yang Wu, Yongshun Gong, Xiaoshui Huang, Liang XiaoAbstract
Text-to-LiDAR generation can customize 3D data with rich structures and diverse scenes for downstream tasks. However, the scarcity of Text-LiDAR pairs often causes insufficient training priors, generating overly smooth 3D scenes. Moreover, low-quality text descriptions may degrade generation quality and controllability. In this paper, we propose a Text-to-LiDAR Diffusion Model for scene generation, named T2LDM, with a Self-Conditioned Representation Guidance (SCRG). Specifically, SCRG, by aligning to the real representations, provides the soft supervision with reconstruction details for the Denoising Network (DN) in training, while decoupled in inference. In this way, T2LDM can perceive rich geometric structures from data distribution, generating detailed objects in scenes. Meanwhile, we construct a content-composable Text-LiDAR benchmark, T2nuScenes, along with a controllability metric. Based on this, we analyze the effects of different text prompts for LiDAR generation quality and controllability, providing practical prompt paradigms and insights. Furthermore, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, by learning a conditional encoder via frozen DN, T2LDM can support multiple conditional tasks, including Sparse-to-Dense, Dense-to-Sparse, and Semantic-to-LiDAR generation. Extensive experiments in unconditional and conditional generation demonstrate that T2LDM outperforms existing methods, achieving state-of-the-art scene generation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19004
PDF
-
Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
2025-11-24 04:02:48Keyang Lu, Sifan Zhou, Hongbin Xu, Gang Xu, Zhifei Yang, Yikai Wang, Zhen Xiao, Jieyi Long, Ming LiAbstract
Realistic 3D city generation is fundamental to a wide range of applications, including virtual reality and digital twins. However, most existing methods rely on training a single diffusion model, which limits their ability to generate personalized and boundless city-scale scenes. In this paper, we present Yo'City, a novel agentic framework that enables user-customized and infinitely expandable 3D city generation by leveraging the reasoning and compositional capabilities of off-the-shelf large models. Specifically, Yo'City first conceptualize the city through a top-down planning strategy that defines a hierarchical "City-District-Grid" structure. The Global Planner determines the overall layout and potential functional districts, while the Local Designer further refines each district with detailed grid-level descriptions. Subsequently, the grid-level 3D generation is achieved through a "produce-refine-evaluate" isometric image synthesis loop, followed by image-to-3D generation. To simulate continuous city evolution, Yo'City further introduces a user-interactive, relationship-guided expansion mechanism, which performs scene graph-based distance- and semantics-aware layout optimization, ensuring spatially coherent city growth. To comprehensively evaluate our method, we construct a diverse benchmark dataset and design six multi-dimensional metrics that assess generation quality from the perspectives of semantics, geometry, texture, and layout. Extensive experiments demonstrate that Yo'City consistently outperforms existing state-of-the-art methods across all evaluation aspects.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18734
PDF
-
GEN3D: Generating Domain-Free 3D Scenes from a Single Image
2025-11-18 09:40:43Yuxin Zhang, Ziyu Lu, Hongbo Duan, Keyu Fan, Pengting Luo, Peiyu Zhuang, Mengyu Yang, Houde LiuAbstract
Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. Additionally, 3D scene generation is vital for advancing embodied AI and world models, which depend on diverse, high-quality scenes for learning and evaluation. In this work, we propose Gen3d, a novel method for generation of high-quality, wide-scope, and generic 3D scenes from a single image. After the initial point cloud is created by lifting the RGBD image, Gen3d maintains and expands its world model. The 3D scene is finalized through optimizing a Gaussian splatting representation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in generating a world model and Synthesizing high-fidelity and consistent novel views.
Abstract (translated)
URL
https://arxiv.org/abs/2511.14291
PDF
-
DriveLiDAR4D: Sequential and Controllable LiDAR Scene Generation for Autonomous Driving
2025-11-17 12:43:26Kaiwen Cai, Xinze Liu, Xia Zhou, Hengtong Hu, Jie Xiang, Luyao Zhang, Xueyang Zhang, Kun Zhan, Yifei Zhan, Xianpeng LangAbstract
The generation of realistic LiDAR point clouds plays a crucial role in the development and evaluation of autonomous driving systems. Although recent methods for 3D LiDAR point cloud generation have shown significant improvements, they still face notable limitations, including the lack of sequential generation capabilities and the inability to produce accurately positioned foreground objects and realistic backgrounds. These shortcomings hinder their practical applicability. In this paper, we introduce DriveLiDAR4D, a novel LiDAR generation pipeline consisting of multimodal conditions and a novel sequential noise prediction model LiDAR4DNet, capable of producing temporally consistent LiDAR scenes with highly controllable foreground objects and realistic backgrounds. To the best of our knowledge, this is the first work to address the sequential generation of LiDAR scenes with full scene manipulation capability in an end-to-end manner. We evaluated DriveLiDAR4D on the nuScenes and KITTI datasets, where we achieved an FRD score of 743.13 and an FVD score of 16.96 on the nuScenes dataset, surpassing the current state-of-the-art (SOTA) method, UniScene, with an performance boost of 37.2% in FRD and 24.1% in FVD, respectively.
Abstract (translated)
URL
https://arxiv.org/abs/2511.13309
PDF
-
Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images
2025-11-10 15:44:48JiaKui Hu, Shanshan Zhao, Qing-Guo Chen, Xuerui Qiu, Jialun Liu, Zhao Xu, Weihua Luo, Kaifu Zhang, Yanye LuAbstract
This paper presents Omni-View, which extends the unified multimodal understanding and generation to 3D scenes based on multiview images, exploring the principle that "generation facilitates understanding". Consisting of understanding model, texture module, and geometry module, Omni-View jointly models scene understanding, novel view synthesis, and geometry estimation, enabling synergistic interaction between 3D scene understanding and generation tasks. By design, it leverages the spatiotemporal modeling capabilities of its texture module responsible for appearance synthesis, alongside the explicit geometric constraints provided by its dedicated geometry module, thereby enriching the model's holistic understanding of 3D scenes. Trained with a two-stage strategy, Omni-View achieves a state-of-the-art score of 55.4 on the VSI-Bench benchmark, outperforming existing specialized 3D understanding models, while simultaneously delivering strong performance in both novel view synthesis and 3D scene generation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.07222
PDF
-
Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
2025-11-01 11:16:25Panwang Pan, Chenguo Lin, Jingjing Zhao, Chenxin Li, Yuchen Lin, Haopeng Li, Honglei Yan, Kairun Wen, Yunlong Lin, Yixuan Yuan, Yadong MuAbstract
We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image. Our approach unifies the generative priors of video diffusion models with geometry and motion constraints learned from large-scale 4D datasets. Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion, all in a single forward pass, without test-time optimization or post-hoc refinement. At the core of our framework lies a video latent transformer, which augments video diffusion models to jointly capture spatio-temporal dependencies and predict time-varying 3D Gaussian primitives. Training is guided by objectives on appearance fidelity, geometric accuracy, and motion consistency, enabling Diff4Splat to synthesize high-quality 4D scenes in 30 seconds. We demonstrate the effectiveness of Diff4Splatacross video generation, novel view synthesis, and geometry extraction, where it matches or surpasses optimization-based methods for dynamic scene synthesis while being significantly more efficient.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00503
PDF
-
HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Composition
2025-10-31 03:50:47Jiacheng Hong, Kunzhen Wu, Mingrui Yu, Yichao Gu, Shengze Xue, Shuangjiu Xiao, Deli DongAbstract
Three-dimensional scene generation holds significant potential in gaming, film, and virtual reality. However, most existing methods adopt a single-step generation process, making it difficult to balance scene complexity with minimal user input. Inspired by the human cognitive process in scene modeling, which progresses from global to local, focuses on key elements, and completes the scene through semantic association, we propose HiGS, a hierarchical generative framework for multi-step associative semantic spatial composition. HiGS enables users to iteratively expand scenes by selecting key semantic objects, offering fine-grained control over regions of interest while the model completes peripheral areas automatically. To support structured and coherent generation, we introduce the Progressive Hierarchical Spatial-Semantic Graph (PHiSSG), which dynamically organizes spatial relationships and semantic dependencies across the evolving scene structure. PHiSSG ensures spatial and geometric consistency throughout the generation process by maintaining a one-to-one mapping between graph nodes and generated objects and supporting recursive layout optimization. Experiments demonstrate that HiGS outperforms single-stage methods in layout plausibility, style consistency, and user preference, offering a controllable and extensible paradigm for efficient 3D scene construction.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27148
PDF
-
OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes
2025-10-30 17:59:51Yukun Huang, Jiwen Yu, Yanning Zhou, Jianan Wang, Xintao Wang, Pengfei Wan, Xihui LiuAbstract
There are two prevalent ways to constructing 3D scenes: procedural generation and 2D lifting. Among them, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panoramic perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and ignore the perception of intrinsic properties, we present OmniX, a versatile and unified framework. Based on a lightweight and efficient cross-modal adapter structure, OmniX reuses 2D generative priors for a broad range of panoramic vision tasks, including panoramic perception, generation, and completion. Furthermore, we construct a large-scale synthetic panorama dataset containing high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness of our model in panoramic visual perception and graphics-ready 3D scene generation, opening new possibilities for immersive and physically realistic virtual world generation.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26800
PDF
-
Token Is All You Need: Cognitive Planning through Sparse Intent Alignment
2025-10-30 12:16:45Shiyao SangAbstract
We challenge the long-standing assumption that exhaustive scene modeling is required for high-performance end-to-end autonomous driving (E2EAD). Unlike world-model approaches that rely on computationally intensive future scene generation or vision-language-action (VLA) systems constrained by Markov assumptions, we show that a minimal set of semantically rich tokens is sufficient for effective planning. Experiments on the nuPlan benchmark (720 scenarios, over 11,000 samples) using perception-informed BEV representations yield three key findings: (1) even without future prediction, our sparse representation achieves 0.548 m ADE, comparable to or surpassing prior methods reporting around 0.75 m on nuScenes; (2) conditioning trajectory decoding on predicted future tokens reduces ADE to 0.479 m, a 12.6% improvement over current-state baselines; and (3) explicit reconstruction loss offers no benefit and may degrade performance under reliable perception inputs. Notably, we observe the emergence of temporal fuzziness, where the model adaptively attends to task-relevant semantics rather than aligning rigidly to fixed timestamps, providing a cognitive advantage for planning under uncertainty. Our "token is all you need" principle marks a paradigm shift from reconstructing the world to understanding it, laying a foundation for cognitively inspired systems that plan through imagination rather than reaction.
Abstract (translated)
URL
https://arxiv.org/abs/2511.05540
PDF
-
TRELLISWorld: Training-Free World Generation from Object Generators
2025-10-27 21:40:31Hanke Chen, Yuan Liu, Minchen LiAbstract
Text-driven 3D scene generation holds promise for a wide range of applications, from virtual prototyping to AR/VR and simulation. However, existing methods are often constrained to single-object generation, require domain-specific training, or lack support for full 360-degree viewability. In this work, we present a training-free approach to 3D scene synthesis by repurposing general-purpose text-to-3D object diffusion models as modular tile generators. We reformulate scene generation as a multi-tile denoising problem, where overlapping 3D regions are independently generated and seamlessly blended via weighted averaging. This enables scalable synthesis of large, coherent scenes while preserving local semantic control. Our method eliminates the need for scene-level datasets or retraining, relies on minimal heuristics, and inherits the generalization capabilities of object-level priors. We demonstrate that our approach supports diverse scene layouts, efficient generation, and flexible editing, establishing a simple yet powerful foundation for general-purpose, language-driven 3D scene construction.
Abstract (translated)
URL
https://arxiv.org/abs/2510.23880
PDF
-
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
2025-10-27 03:52:45Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai, Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, Wenjun ZengAbstract
Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. Repo: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.22973
PDF
-
WorldGrow: Generating Infinite 3D World
2025-10-24 17:39:52Sikuang Li, Chen Yang, Jiemin Fang, Taoran Yi, Jia Lu, Jiazhong Cen, Lingxi Xie, Wei Shen, Qi TianAbstract
We tackle the challenge of generating the infinitely extendable 3D world -- large, continuous environments with coherent geometry and realistic appearance. Existing methods face key challenges: 2D-lifting approaches suffer from geometric and appearance inconsistencies across views, 3D implicit representations are hard to scale up, and current 3D foundation models are mostly object-centric, limiting their applicability to scene-level generation. Our key insight is leveraging strong generation priors from pre-trained 3D models for structured scene block generation. To this end, we propose WorldGrow, a hierarchical framework for unbounded 3D scene synthesis. Our method features three core components: (1) a data curation pipeline that extracts high-quality scene blocks for training, making the 3D structured latent representations suitable for scene generation; (2) a 3D block inpainting mechanism that enables context-aware scene extension; and (3) a coarse-to-fine generation strategy that ensures both global layout plausibility and local geometric/textural fidelity. Evaluated on the large-scale 3D-FRONT dataset, WorldGrow achieves SOTA performance in geometry reconstruction, while uniquely supporting infinite scene generation with photorealistic and structurally consistent outputs. These results highlight its capability for constructing large-scale virtual environments and potential for building future world models.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21682
PDF
-
AutoScape: Geometry-Consistent Long-Horizon Scene Generation
2025-10-23 16:44:34Jiacheng Chen, Ziyu Jiang, Mingfu Liang, Bingbing Zhuang, Jong-Chyi Su, Sparsh Garg, Ying Wu, Manmohan ChandrakerAbstract
This paper proposes AutoScape, a long-horizon driving scene generation framework. At its core is a novel RGB-D diffusion model that iteratively generates sparse, geometrically consistent keyframes, serving as reliable anchors for the scene's appearance and geometry. To maintain long-range geometric consistency, the model 1) jointly handles image and depth in a shared latent space, 2) explicitly conditions on the existing scene geometry (i.e., rendered point clouds) from previously generated keyframes, and 3) steers the sampling process with a warp-consistent guidance. Given high-quality RGB-D keyframes, a video diffusion model then interpolates between them to produce dense and coherent video frames. AutoScape generates realistic and geometrically consistent driving videos of over 20 seconds, improving the long-horizon FID and FVD scores over the prior state-of-the-art by 48.6\% and 43.0\%, respectively.
Abstract (translated)
本文提出了AutoScape，这是一种长时域驾驶场景生成框架。其核心是一个新颖的RGB-D扩散模型，该模型通过迭代生成稀疏、几何一致的关键帧，这些关键帧作为场景外观和几何结构的可靠锚点。为了维持长距离几何一致性，模型采用了以下策略：1）在共享潜在空间中同时处理图像和深度信息；2）明确地以先前生成的关键帧中存在的场景几何（即渲染后的点云）为条件；3）使用一致变形引导来指导采样过程。基于高质量的RGB-D关键帧，随后的视频扩散模型会在这之间进行插值运算，从而产生密集且连贯的视频帧。AutoScape能够生成超过20秒的真实感和几何一致性良好的驾驶视频，在长时域FID（Fréchet Inception Distance）和FVD（Feature Voxel Distance）评分上分别比之前的最佳方法提高了48.6% 和 43.0%。
URL
https://arxiv.org/abs/2510.20726
PDF
-
Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
2025-10-22 18:16:32Jiashi Feng, Xiu Li, Jing Lin, Jiahang Liu, Gaohong Liu, Weiqiang Lou, Su Ma, Guang Shi, Qinlong Wang, Jun Wang, Zhongcong Xu, Xuanyu Yi, Zihao Yu, Jianfeng Zhang, Yifan Zhu, Rui Chen, Jinxin Chi, Zixian Du, Li Han, Lixin Huang, Kaihua Jiang, Yuhan Li, Guan Luo, Shuguang Wang, Qianyi Wu, Fan Yang, Junyang Zhang, Xuanmeng ZhangAbstract
Developing embodied AI agents requires scalable training environments that balance content diversity with physics accuracy. World simulators provide such environments but face distinct limitations: video-based methods generate diverse content but lack real-time physics feedback for interactive learning, while physics-based engines provide accurate dynamics but face scalability limitations from costly manual asset creation. We present Seed3D 1.0, a foundation model that generates simulation-ready 3D assets from single images, addressing the scalability challenge while maintaining physics rigor. Unlike existing 3D generation models, our system produces assets with accurate geometry, well-aligned textures, and realistic physically-based materials. These assets can be directly integrated into physics engines with minimal configuration, enabling deployment in robotic manipulation and simulation training. Beyond individual objects, the system scales to complete scene generation through assembling objects into coherent environments. By enabling scalable simulation-ready content creation, Seed3D 1.0 provides a foundation for advancing physics-based world simulators. Seed3D 1.0 is now available on this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.19944
PDF
-
DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion
2025-10-17 03:00:08Weijie Wang, Jiagang Zhu, Zeyu Zhang, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Chaojun Ni, Haoxiao Wang, Guan Huang, Xinze Chen, Yukun Zhou, Wenkang Qin, Duochao Shi, Haoyun Li, Guanghong Jia, Jiwen LuAbstract
We present DriveGen3D, a novel framework for generating high-quality and highly controllable dynamic 3D driving scenes that addresses critical limitations in existing methodologies. Current approaches to driving scene synthesis either suffer from prohibitive computational demands for extended temporal generation, focus exclusively on prolonged video synthesis without 3D representation, or restrict themselves to static single-scene reconstruction. Our work bridges this methodological gap by integrating accelerated long-term video generation with large-scale dynamic scene reconstruction through multimodal conditional control. DriveGen3D introduces a unified pipeline consisting of two specialized components: FastDrive-DiT, an efficient video diffusion transformer for high-resolution, temporally coherent video synthesis under text and Bird's-Eye-View (BEV) layout guidance; and FastRecon3D, a feed-forward reconstruction module that rapidly builds 3D Gaussian representations across time, ensuring spatial-temporal consistency. Together, these components enable real-time generation of extended driving videos (up to $424\times800$ at 12 FPS) and corresponding dynamic 3D scenes, achieving SSIM of 0.811 and PSNR of 22.84 on novel view synthesis, all while maintaining parameter efficiency.
Abstract (translated)
我们介绍了DriveGen3D，这是一个用于生成高质量、高度可控的动态3D驾驶场景的新颖框架，解决了现有方法中的关键限制。当前的驾驶场景合成方法要么在长时间生成时面临计算需求过高的问题，要么仅专注于没有3D表示的长时间视频合成都市仿真中常见的情况，或者局限于静态单一场景重建。 我们的工作通过结合加速长期视频生成和大规模动态场景重构来弥合这些方法论之间的差距，并且支持多模态条件控制。DriveGen3D引入了一个统一的工作流程，包括两个专门化的组件：FastDrive-DiT是一个高效的视频扩散变压器，在文本和鸟瞰图（BEV）布局引导下进行高分辨率、时间连贯的视频合成；以及FastRecon3D，这是一个前馈重构模块，可以快速构建时序中的三维高斯表示，并保证空间-时间一致性。这两个组件共同实现实时生成扩展驾驶视频（高达424x800像素，12 FPS）和相应的动态3D场景，在新视角合成中达到SSIM为0.811、PSNR为22.84的性能指标，同时保持参数效率。
URL
https://arxiv.org/abs/2510.15264
PDF
-
CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
2025-10-16 03:29:06Li Liang, Bo Miao, Xinyu Wang, Naveed Akhtar, Jordan Vice, Ajmal MianAbstract
Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large-scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo-labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at this https URL
Abstract (translated)
户外三维语义场景生成旨在为城市模拟和自动驾驶等应用创建现实且富含语义信息的环境。然而，这一领域的进步受到公开可用、标注完善的数据库缺失的限制。为此，我们推出了SketchSem3D，这是首个用于从抽象的手绘草图和卫星图像伪标签注释生成大规模三维户外语义场景的基准数据集。SketchSem3D包含两个子集：基于草图的SemanticKITTI和基于草图的KITTI-360（包括激光雷达体素及其对应的草图和标注过的卫星图像），以支持标准、严格且多样化的评估。此外，我们还提出了Cylinder Mamba Diffusion (CymbaDiff)，这一方法显著提升了户外三维场景生成的空间一致性。CymbaDiff施加了结构化空间顺序，并明确捕捉了圆柱连续性和垂直层次关系，在生成的场景中同时保留物理邻域关系和全局背景信息。在SketchSem3D上的广泛实验表明，CymbaDiff实现了更优的语义一致性和空间现实感，以及跨数据集的泛化能力。相关代码和数据集将在以下链接发布：[此网址] (请根据实际发布的地址替换该占位符)
URL
https://arxiv.org/abs/2510.13245
PDF
-
Hybrid Gaussian Splatting for Novel Urban View Synthesis
2025-10-14 09:09:13Mohamed Omran, Farhad Zanjani, Davide Abati, Jens Petersen, Amirhossein HabibianAbstract
This paper describes the Qualcomm AI Research solution to the RealADSim-NVS challenge, hosted at the RealADSim Workshop at ICCV 2025. The challenge concerns novel view synthesis in street scenes, and participants are required to generate, starting from car-centric frames captured during some training traversals, renders of the same urban environment as viewed from a different traversal (e.g. different street lane or car direction). Our solution is inspired by hybrid methods in scene generation and generative simulators merging gaussian splatting and diffusion models, and it is composed of two stages: First, we fit a 3D reconstruction of the scene and render novel views as seen from the target cameras. Then, we enhance the resulting frames with a dedicated single-step diffusion model. We discuss specific choices made in the initialization of gaussian primitives as well as the finetuning of the enhancer model and its training data curation. We report the performance of our model design and we ablate its components in terms of novel view quality as measured by PSNR, SSIM and LPIPS. On the public leaderboard reporting test results, our proposal reaches an aggregated score of 0.432, achieving the second place overall.
Abstract (translated)
这篇论文描述了高通AI研究为ICCV 2025举办的RealADSim Workshop中的RealADSim-NVS挑战提供的解决方案。该挑战涉及街景中视点新颖性合成问题，参赛者需要从训练过程中拍摄的以汽车为中心的画面出发，生成同一城市环境在不同视角下的渲染图（例如不同的街道车道或汽车方向）。我们的方案受到场景生成混合方法和结合高斯光斑技术和扩散模型的生成模拟器的启发，由两个阶段组成：首先，我们拟合出一个3D重建的场景，并从目标相机的角度重新渲染新颖视图；然后，我们使用专门设计的一步式扩散模型来增强所得帧。本文讨论了初始化高斯原语的具体选择以及细化增强模型及其训练数据整理方面的决策。我们报告了我们的模型的设计性能，并根据PSNR（峰值信噪比）、SSIM（结构相似性指数）和LPIPS（感知线性像差感知感知图像相似度）来衡量新颖视图质量，对其组件进行了消融研究。在公布的结果排行榜上，以测试结果为基础，我们方案的综合得分为0.432，取得了总成绩第二名的成绩。
URL
https://arxiv.org/abs/2510.12308
PDF
-
IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation
2025-10-14 03:02:33Wenxu Zhou, Kaixuan Nie, Hang Du, Dong Yin, Wei Huang, Siqiang Guo, Xiaobo Zhang, Pengbo HuAbstract
In this study, we present IL3D, a large-scale dataset meticulously designed for large language model (LLM)-driven 3D scene generation, addressing the pressing demand for diverse, high-quality training data in indoor layout design. Comprising 27,816 indoor layouts across 18 prevalent room types and a library of 29,215 high-fidelity 3D object assets, IL3D is enriched with instance-level natural language annotations to support robust multimodal learning for vision-language tasks. We establish rigorous benchmarks to evaluate LLM-driven scene generation. Experimental results show that supervised fine-tuning (SFT) of LLMs on IL3D significantly improves generalization and surpasses the performance of SFT on other datasets. IL3D offers flexible multimodal data export capabilities, including point clouds, 3D bounding boxes, multiview images, depth maps, normal maps, and semantic masks, enabling seamless adaptation to various visual tasks. As a versatile and robust resource, IL3D significantly advances research in 3D scene generation and embodied intelligence, by providing high-fidelity scene data to support environment perception tasks of embodied agents.
Abstract (translated)
在这项研究中，我们介绍了IL3D，这是一个为大规模语言模型（LLM）驱动的三维场景生成精心设计的大规模数据集，旨在满足室内布局设计领域对多样化、高质量训练数据迫切需求。IL3D包含27,816个室内布局，涵盖了18种常见的房间类型，并拥有一个由29,215项高保真3D物体资产组成的库。此外，该数据集通过实例级别的自然语言注释进行了丰富，以支持视觉-语言任务中的稳健多模态学习。我们建立了严格的基准测试来评估LLM驱动的场景生成能力。实验结果显示，在IL3D上对LLMs进行监督微调（SFT）显著提高了泛化性能，并且优于在其他数据集上的SFT表现。 IL3D提供了灵活的多模态数据导出功能，包括点云、三维边界框、多视角图像、深度图、法线图和语义掩码，从而可以无缝地适应各种视觉任务。作为一个多功能且稳健的资源，IL3D通过提供高保真场景数据来支持具身智能体的环境感知任务，在三维场景生成和具身智能的研究方面取得了重大进展。
URL
https://arxiv.org/abs/2510.12095
PDF
-
EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making
2025-10-14 02:26:52Zixing Lei, Sheng Yin, Yichen Xiong, Yuanzhuo Ding, Wenhao Huang, Yuxi Wei, Qingyao Xu, Yiming Li, Weixin Li, Yunhong Wang, Siheng ChenAbstract
Embodied decision-making enables agents to translate high-level goals into executable actions through continuous interactions within the physical world, forming a cornerstone of general-purpose embodied intelligence. Large language models (LLMs), with their general decision-making capabilities, offer a promising path to realize this potential; however, LLMs trained solely on language lack exposure to physical environments, limiting their true embodied understanding. To bridge this gap, we propose the concept of a training ground: a comprehensive infrastructure that provides task and scene simulation, embodied interaction, and feedback signals, offering a one-stop solution for LLM acquire genuine embodied decision-making skills. In this work, we present EmboMatrix, the first training ground of its kind, providing massive and diverse tasks with efficient simulation and precise rewards. EmboMatrix incorporates a series of novel techniques: a multi-agent data engine for large-scale task and scene generation, a distributed heterogeneous-hardware system for scalable simulation, and a multi-level reward architecture for precise supervision. Leveraging EmboMatrix, we cultivate EmboBrain, an LLM whose embodied decision-making abilities emerge from extensive embodied interactions. Experiments show that EmboBrain-7B surpasses the 671B DeepSeek-R1 baseline by 9.5\% on two challenging embodied decision-making benchmarks, demonstrating the power of interactive, environment-grounded learning for building truly intelligent embodied agents.
Abstract (translated)
嵌入式决策制定使代理能够通过与物理世界的持续互动将高层次的目标转化为可执行的动作，这是通用型嵌入智能的核心。大型语言模型（LLM）凭借其广泛的应用决策能力为实现这一潜力提供了可能路径；然而，仅接受语言训练的LLM缺乏对物理环境的实际接触，这限制了它们在真正意义上理解并适应物理世界的能力。为了弥合这种差距，我们提出了一种训练平台的概念：一个提供任务和场景模拟、嵌入式互动及反馈信号的全面基础设施，为LLM获取真正的嵌入式决策制定技能提供一站式解决方案。 在这项工作中，我们介绍了EmboMatrix——首个此类训练平台，它提供了大量且多样化的任务，并配备了高效仿真的功能和精确奖励机制。EmboMatrix整合了一系列新颖技术：一个多代理数据引擎用于大规模的任务和场景生成、一个分布式的异构硬件系统支持可扩展模拟以及一个多层级的奖励架构以实现精准监督。 通过利用EmboMatrix，我们培养出了EmboBrain——一种LLM，其嵌入式决策能力源于广泛的物理世界互动。实验表明，在两个具有挑战性的嵌入式决策制定基准测试中，EmboBrain-7B的表现比671B的DeepSeek-R1基线模型高出9.5%，这证明了基于环境交互的学习对于构建真正智能的嵌入式代理的重要性。
URL
https://arxiv.org/abs/2510.12072
PDF
