Scene_Generation
-
RoboLayout: Differentiable 3D Scene Generation for Embodied Agents
2026-02-18 08:17:05Ali ShamsaddinlouAbstract
Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.
Abstract (translated)
最近，视觉语言模型（VLM）在从开放式语言指令生成空间推理和3D场景布局方面展示了强大的潜力。然而，创建既语义连贯又能被具身代理交互的可行布局仍然具有挑战性，尤其是在物理受限的室内环境中。在这篇论文中，介绍了一种名为RoboLayout的新方法，它是对LayoutVLM框架的一种扩展，通过加入代理感知推理和优化稳定性改进来增强原模型的功能。RoboLayout将显式的可达性约束整合进可微布局优化过程中，从而生成能够被具身代理导航并执行的场景布局。 重要的是，这种代理抽象不仅限于特定的机器人平台，还可以表示具有不同物理能力的各种实体，例如服务型机器人、仓库机器人、各年龄段的人类或动物等，从而使环境设计可以适应不同的目标代理。此外，还提出了一种局部细化阶段，在该阶段中选择性地重新优化有问题的对象位置，同时保持场景其余部分不变，从而在不增加全局优化迭代次数的情况下提高了收敛效率。 总体而言，通过跨多种不同配置的实验结果证明，RoboLayout不仅保留了LayoutVLM的强大语义对齐和物理合理性，而且还增强了其应用于代理中心视角的室内场景生成的能力。
URL
https://arxiv.org/abs/2603.05522
PDF
-
DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation
2026-02-14 07:02:36Haoyu Zhao, Yuang Zhang, Junqi Cheng, Jiaxi Gu, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang JiangAbstract
Recent video generative models have demonstrated impressive visual fidelity, yet they often struggle with semantic, geometric, and identity consistency. In this paper, we propose a system-level framework, termed the Divide-and-Conquer Diffusion Model (DCDM), to address three key challenges: (1) intra-clip world knowledge consistency, (2) inter-clip camera consistency, and (3) inter-shot element consistency. DCDM decomposes video consistency modeling under these scenarios into three dedicated components while sharing a unified video generation backbone. For intra-clip consistency, DCDM leverages a large language model to parse input prompts into structured semantic representations, which are subsequently translated into coherent video content by a diffusion transformer. For inter-clip camera consistency, we propose a temporal camera representation in the noise space that enables precise and stable camera motion control, along with a text-to-image initialization mechanism to further enhance controllability. For inter-shot consistency, DCDM adopts a holistic scene generation paradigm with windowed cross-attention and sparse inter-shot self-attention, ensuring long-range narrative coherence while maintaining computational efficiency. We validate our framework on the test set of the CVM Competition at AAAI'26, and the results demonstrate that the proposed strategies effectively address these challenges.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13637
PDF
-
Code2Worlds: Empowering Coding LLMs for 4D World Generation
2026-02-12 09:34:28Yi Zhang, Yunshuang Wang, Zeyu Zhang, Hao TangAbstract
Achieving spatial intelligence requires moving beyond visual plausibility to build world simulators grounded in physical laws. While coding LLMs have advanced static 3D scene generation, extending this paradigm to 4D dynamics remains a critical frontier. This task presents two fundamental challenges: multi-scale context entanglement, where monolithic generation fails to balance local object structures with global environmental layouts; and a semantic-physical execution gap, where open-loop code generation leads to physical hallucinations lacking dynamic fidelity. We introduce Code2Worlds, a framework that formulates 4D generation as language-to-simulation code generation. First, we propose a dual-stream architecture that disentangles retrieval-augmented object generation from hierarchical environmental orchestration. Second, to ensure dynamic fidelity, we establish a physics-aware closed-loop mechanism in which a PostProcess Agent scripts dynamics, coupled with a VLM-Motion Critic that performs self-reflection to iteratively refine simulation code. Evaluations on the Code4D benchmark show Code2Worlds outperforms baselines with a 41% SGS gain and 49% higher Richness, while uniquely generating physics-aware dynamics absent in prior static methods. Code: this https URL. Website: this https URL.
Abstract (translated)
实现空间智能需要超越视觉逼真度，构建基于物理定律的现实世界模拟器。虽然编码大型语言模型（LLM）已经推进了静态3D场景生成的发展，但将这一范式扩展到4D动态领域仍然是一个关键的研究前沿。这项任务面临两个基本挑战：多尺度上下文纠缠问题，在这种情况下，单一的整体生成方法无法平衡局部物体结构和全局环境布局；以及语义-物理执行差距，开放循环代码生成会导致缺乏动态真实的物理幻觉。我们引入了Code2Worlds框架，该框架将4D生成作为语言到模拟代码生成来处理。 首先，我们提出了一种双流架构，它分离出增强检索的物体生成与分层环境协调。其次，为了确保动态真实度，我们建立了一个具有物理学感知能力的闭环机制，在这个机制中，后处理代理（PostProcess Agent）编写动力学脚本，并结合视觉语言模型-运动批评家（VLM-Motion Critic），该批评家执行自我反思以迭代地改进模拟代码。 在Code4D基准测试上的评估表明，Code2Worlds相较于基线方法，在SGS指标上提高了41%，而在丰富度方面提高了49%。同时，它还能独特生成以前静态方法中缺失的具有物理学感知能力的动力学效果。 请注意，原文末尾提及了代码链接和网站链接，但这些链接在您的问题中是以占位符的形式出现的（"this https URL."），因此我无法直接提供它们的具体内容或访问。如果您需要访问这些资源，请前往原发布页面获取详细信息。
URL
https://arxiv.org/abs/2602.11757
PDF
-
LLM-Driven 3D Scene Generation of Agricultural Simulation Environments
2026-02-12 08:33:01Arafa Yoncalik, Wouter Jansen, Nico Huebel, Mohammad Hasan Rahmani, Jan SteckelAbstract
Procedural generation techniques in 3D rendering engines have revolutionized the creation of complex environments, reducing reliance on manual design. Recent approaches using Large Language Models (LLMs) for 3D scene generation show promise but often lack domain-specific reasoning, verification mechanisms, and modular design. These limitations lead to reduced control and poor scalability. This paper investigates the use of LLMs to generate agricultural synthetic simulation environments from natural language prompts, specifically to address the limitations of lacking domain-specific reasoning, verification mechanisms, and modular design. A modular multi-LLM pipeline was developed, integrating 3D asset retrieval, domain knowledge injection, and code generation for the Unreal rendering engine using its API. This results in a 3D environment with realistic planting layouts and environmental context, all based on the input prompt and the domain knowledge. To enhance accuracy and scalability, the system employs a hybrid strategy combining LLM optimization techniques such as few-shot prompting, Retrieval-Augmented Generation (RAG), finetuning, and validation. Unlike monolithic models, the modular architecture enables structured data handling, intermediate verification, and flexible expansion. The system was evaluated using structured prompts and semantic accuracy metrics. A user study assessed realism and familiarity against real-world images, while an expert comparison demonstrated significant time savings over manual scene design. The results confirm the effectiveness of multi-LLM pipelines in automating domain-specific 3D scene generation with improved reliability and precision. Future work will explore expanding the asset hierarchy, incorporating real-time generation, and adapting the pipeline to other simulation domains beyond agriculture.
Abstract (translated)
三维渲染引擎中的程序生成技术已经彻底改变了复杂环境的创建，减少了对手动设计的依赖。最近使用大型语言模型（LLMs）进行3D场景生成的方法显示出潜力，但往往缺乏领域特定推理、验证机制和模块化设计能力。这些限制导致控制减弱和可扩展性差。本文探讨了利用LLM根据自然语言提示生成农业合成仿真环境的方法，以解决缺乏领域特定推理、验证机制和模块化设计的局限性。 为此，开发了一种模块化的多LLM管道，结合3D资产检索、领域知识注入以及使用Unreal渲染引擎API进行代码生成。这产生了一个基于输入提示和领域知识的真实种植布局和环境上下文的三维环境。 为了提高准确性和可扩展性，该系统采用了一种混合策略，结合了包括少量样本提示（few-shot prompting）、增强式检索生成（Retrieval-Augmented Generation, RAG）、微调和验证在内的LLM优化技术。与单一模型相比，模块化架构允许结构化数据处理、中间验证和灵活扩展。 通过使用结构化提示和语义准确性度量对系统进行了评估。一项用户研究评估了合成环境的真实感和熟悉程度，并将其与真实世界的图像进行比较；专家对比显示该方法比手动场景设计节省大量时间。结果证实了多LLM管道在自动化特定领域的3D场景生成方面的有效性，提高了可靠性和精度。 未来的工作将探索扩展资产层次结构、结合实时生成以及适应农业以外的其他模拟领域。
URL
https://arxiv.org/abs/2602.11706
PDF
-
SAGE: Scalable Agentic 3D Scene Generation for Embodied AI
2026-02-10 18:59:55Hongchi Xia, Xuan Li, Zhaoshuo Li, Qianli Ma, Jiashu Xu, Ming-Yu Liu, Yin Cui, Tsung-Yi Lin, Wei-Chiu Ma, Shenlong Wang, Shuran Song, Fangyin WeiAbstract
Real-world data collection for embodied agents remains costly and unsafe, calling for scalable, realistic, and simulator-ready 3D environments. However, existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes. We present SAGE, an agentic framework that, given a user-specified embodied task (e.g., "pick up a bowl and place it on the table"), understands the intent and automatically generates simulation-ready environments at scale. The agent couples multiple generators for layout and object composition with critics that evaluate semantic plausibility, visual realism, and physical stability. Through iterative reasoning and adaptive tool selection, it self-refines the scenes until meeting user intent and physical validity. The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training. Policies trained purely on this data exhibit clear scaling trends and generalize to unseen objects and layouts, demonstrating the promise of simulation-driven scaling for embodied AI. Code, demos, and the SAGE-10k dataset can be found on the project page here: this https URL.
Abstract (translated)
真实世界中为具身智能体（如机器人或虚拟角色）收集数据既昂贵又不安全，因此需要可扩展、逼真且适用于模拟器的3D环境。然而，现有的场景生成系统通常依赖于基于规则的方法或特定任务的管道，这会导致生成的效果不佳和物理上无效的场景。我们介绍了SAGE框架，这是一种智能体驱动的方法，能够根据用户指定的任务（例如，“拿起碗并把它放在桌子上”），理解意图，并自动生成大规模的、可用于模拟器的环境。 该方法集成了多个布局与物体组合的生成器以及评估语义合理性、视觉真实性和物理稳定性的批评者。通过迭代推理和适应性工具选择，智能体能够不断自我优化场景，直至符合用户意图并满足物理有效性要求。最终产生的环境既逼真又多样化，并且可以直接部署在现代模拟器中用于策略训练。 仅使用这种数据进行的策略训练显示出明确的规模效应，并能推广到未见过的对象和布局上，这表明了基于模拟的数据驱动扩展对具身AI的前景。该项目的相关代码、演示以及SAGE-10k数据集可以在项目页面（此链接）找到。
URL
https://arxiv.org/abs/2602.10116
PDF
-
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
2026-02-10 15:19:17Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Junhao He, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu YueAbstract
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
Abstract (translated)
基于世界模型的想象然后行动成为机器人操作的一个有前景的新范式，但现有方法通常仅支持纯图像预测或在部分3D几何结构上的推理，这限制了它们预测完整4D场景动态的能力。这项工作提出了一种新型的具身化4D世界模型，该模型能够生成任意视角下的一致RGBD数据：给定单一视角的RGBD观测输入后，模型可以想象其他视点，并将这些视图反投影融合以构建更完整的3D结构（跨时间）。为了高效地学习多视角、跨模态的数据生成，我们设计了跨视图和跨模态特征融合，从而共同促进RGB与深度数据之间的一致性和不同视图间的几何对齐。除了预测之外，通常通过逆动力学将生成的未来转换为动作执行，但由于多种动作可能解释相同的过渡过程，这种做法是不适定的问题。为此，我们提出了一种测试时的动作优化策略，该策略通过生成模型反向传播以推断出与预测未来最匹配的轨迹级潜在变量，并结合残差逆动力学模型将此轨迹先验转化为准确可执行的动作。 实验在三个数据集上进行，展示了在4D场景生成和下游操作任务上的强大性能。消融研究提供了对关键设计选择的实际见解。
URL
https://arxiv.org/abs/2602.09878
PDF
-
PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation
2026-02-04 18:58:55Jiahao Zhan, Zizhang Li, Hong-Xing Yu, Jiajun WuAbstract
We introduce PerpetualWonder, a hybrid generative simulator that enables long-horizon, action-conditioned 4D scene generation from a single image. Current works fail at this task because their physical state is decoupled from their visual representation, which prevents generative refinements to update the underlying physics for subsequent interactions. PerpetualWonder solves this by introducing the first true closed-loop system. It features a novel unified representation that creates a bidirectional link between the physical state and visual primitives, allowing generative refinements to correct both the dynamics and appearance. It also introduces a robust update mechanism that gathers supervision from multiple viewpoints to resolve optimization ambiguity. Experiments demonstrate that from a single image, PerpetualWonder can successfully simulate complex, multi-step interactions from long-horizon actions, maintaining physical plausibility and visual consistency.
Abstract (translated)
我们介绍了PerpetualWonder，这是一个混合生成模拟器，它能够从单张图像中生成长期时序、动作条件下的4D场景。当前的工作在此任务上表现不佳，因为它们的物理状态与其视觉表示是脱钩的，这阻碍了生成性细化以更新后续交互的基础物理原理。PerpetualWonder通过引入首个真正的闭环系统解决了这一问题。它具备一种新颖的统一表征方法，这种方法在物理状态和视觉元素之间创建了一个双向联系，使得生成性的细化可以同时修正动态和外观。此外，它还推出了一种稳健的更新机制，该机制可以从多个视角收集监督信息以解决优化模糊性。 实验表明，从单张图像出发，PerpetualWonder能够成功模拟复杂的多步交互，并且这些交互来自长期动作序列，同时还保持物理真实性和视觉一致性。
URL
https://arxiv.org/abs/2602.04876
PDF
-
SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequences
2026-02-03 01:22:07Seok-Young Kim, Dooyoung Kim, Woojin Cho, Hail Song, Suji Kang, Woontack WooAbstract
We introduce SceneLinker, a novel framework that generates compositional 3D scenes via semantic scene graph from RGB sequences. To adaptively experience Mixed Reality (MR) content based on each user's space, it is essential to generate a 3D scene that reflects the real-world layout by compactly capturing the semantic cues of the surroundings. Prior works struggled to fully capture the contextual relationship between objects or mainly focused on synthesizing diverse shapes, making it challenging to generate 3D scenes aligned with object arrangements. We address these challenges by designing a graph network with cross-check feature attention for scene graph prediction and constructing a graph-variational autoencoder (graph-VAE), which consists of a joint shape and layout block for 3D scene generation. Experiments on the 3RScan/3DSSG and SG-FRONT datasets demonstrate that our approach outperforms state-of-the-art methods in both quantitative and qualitative evaluations, even in complex indoor environments and under challenging scene graph constraints. Our work enables users to generate consistent 3D spaces from their physical environments via scene graphs, allowing them to create spatial MR content. Project page is this https URL.
Abstract (translated)
我们介绍了一种名为SceneLinker的新框架，该框架通过从RGB序列生成语义场景图来创建组合式3D场景。为了根据每个用户的实际空间自适应地体验混合现实（MR）内容，有必要生成一个能够捕捉周围环境语义线索并反映真实世界布局的3D场景。先前的研究难以全面捕捉对象之间的上下文关系，或者主要关注于合成各种形状，这使得在与物体排列一致的情况下生成3D场景变得具有挑战性。 我们通过设计一种带有交叉检查特征注意机制的图网络来解决这些问题，并构建了一个图变分自动编码器（graph-VAE），其中包括一个用于3D场景生成的联合形状和布局块。在3RScan/3DSSG和SG-FRONT数据集上的实验表明，我们的方法在定量和定性评估中均优于现有技术，在复杂的室内环境以及具有挑战性的场景图约束下也表现良好。 我们的工作使得用户能够通过场景图从物理环境中生成一致的3D空间，并允许他们创建基于空间的MR内容。项目页面可以在这里访问：[https URL]（请将方括号中的文本替换为实际链接）。
URL
https://arxiv.org/abs/2602.02974
PDF
-
PSGS: Text-driven Panorama Sliding Scene Generation via Gaussian Splatting
2026-01-31 02:34:46Xin Zhang, Shen Chen, Jiale Zhou, Lei LiAbstract
Generating realistic 3D scenes from text is crucial for immersive applications like VR, AR, and gaming. While text-driven approaches promise efficiency, existing methods suffer from limited 3D-text data and inconsistent multi-view stitching, resulting in overly simplistic scenes. To address this, we propose PSGS, a two-stage framework for high-fidelity panoramic scene generation. First, a novel two-layer optimization architecture generates semantically coherent panoramas: a layout reasoning layer parses text into structured spatial relationships, while a self-optimization layer refines visual details via iterative MLLM feedback. Second, our panorama sliding mechanism initializes globally consistent 3D Gaussian Splatting point clouds by strategically sampling overlapping perspectives. By incorporating depth and semantic coherence losses during training, we greatly improve the quality and detail fidelity of rendered scenes. Our experiments demonstrate that PSGS outperforms existing methods in panorama generation and produces more appealing 3D scenes, offering a robust solution for scalable immersive content creation.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00463
PDF
-
GeoDiff3D: Self-Supervised 3D Scene Generation with Geometry-Constrained 2D Diffusion Guidance
2026-01-27 16:47:35Haozhi Zhu, Miaomiao Zhao, Dingyao Liu, Runze Tian, Yan Zhang, Jie Guo, Fenggen YuAbstract
3D scene generation is a core technology for gaming, film/VFX, and VR/AR. Growing demand for rapid iteration, high-fidelity detail, and accessible content creation has further increased interest in this area. Existing methods broadly follow two paradigms - indirect 2D-to-3D reconstruction and direct 3D generation - but both are limited by weak structural modeling and heavy reliance on large-scale ground-truth supervision, often producing structural artifacts, geometric inconsistencies, and degraded high-frequency details in complex scenes. We propose GeoDiff3D, an efficient self-supervised framework that uses coarse geometry as a structural anchor and a geometry-constrained 2D diffusion model to provide texture-rich reference images. Importantly, GeoDiff3D does not require strict multi-view consistency of the diffusion-generated references and remains robust to the resulting noisy, inconsistent guidance. We further introduce voxel-aligned 3D feature aggregation and dual self-supervision to maintain scene coherence and fine details while substantially reducing dependence on labeled data. GeoDiff3D also trains with low computational cost and enables fast, high-quality 3D scene generation. Extensive experiments on challenging scenes show improved generalization and generation quality over existing baselines, offering a practical solution for accessible and efficient 3D scene construction.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19785
PDF
-
RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming
2026-01-27 10:10:55Jisheng Chu, Wenrui Li, Rui Zhao, Wangmeng Zuo, Shifeng Chen, Xiaopeng FanAbstract
Generating immersive 3D scenes from texts is a core task in computer vision, crucial for applications in virtual reality and game development. Despite the promise of leveraging 2D diffusion priors, existing methods suffer from spatial blindness and rely on predefined trajectories that fail to exploit the inner relationships among salient objects. Consequently, these approaches are unable to comprehend the semantic layout, preventing them from exploring the scene adaptively to infer occluded content. Moreover, current inpainting models operate in 2D image space, struggling to plausibly fill holes caused by camera motion. To address these limitations, we propose RoamScene3D, a novel framework that bridges the gap between semantic guidance and spatial generation. Our method reasons about the semantic relations among objects and produces consistent and photorealistic scenes. Specifically, we employ a vision-language model (VLM) to construct a scene graph that encodes object relations, guiding the camera to perceive salient object boundaries and plan an adaptive roaming trajectory. Furthermore, to mitigate the limitations of static 2D priors, we introduce a Motion-Injected Inpainting model that is fine-tuned on a synthetic panoramic dataset integrating authentic camera trajectories, making it adaptive to camera motion. Extensive experiments demonstrate that with semantic reasoning and geometric constraints, our method significantly outperforms state-of-the-art approaches in producing consistent and photorealistic scenes. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19433
PDF
-
NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation
2026-01-27 00:04:02Han-Hung Lee, Cheng-Yu Yang, Yu-Lun Liu, Angel X. ChangAbstract
World generation is a fundamental capability for applications like video games, simulation, and robotics. However, existing approaches face three main obstacles: controllability, scalability, and efficiency. End-to-end scene generation models have been limited by data scarcity. While object-centric generation approaches rely on fixed resolution representations, degrading fidelity for larger scenes. Training-free approaches, while flexible, are often slow and computationally expensive at inference time. We present NuiWorld, a framework that attempts to address these challenges. To overcome data scarcity, we propose a generative bootstrapping strategy that starts from a few input images. Leveraging recent 3D reconstruction and expandable scene generation techniques, we synthesize scenes of varying sizes and layouts, producing enough data to train an end-to-end model. Furthermore, our framework enables controllability through pseudo sketch labels, and demonstrates a degree of generalization to previously unseen sketches. Our approach represents scenes as a collection of variable scene chunks, which are compressed into a flattened vector-set representation. This significantly reduces the token length for large scenes, enabling consistent geometric fidelity across scenes sizes while improving training and inference efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19048
PDF
-
AnchoredDream: Zero-Shot 360{deg} Indoor Scene Generation from a Single View via Geometric Grounding
2026-01-23 08:08:12Runmao Yao, Junsheng Zhou, Zhen Dong, Yu-Shen LiuAbstract
Single-view indoor scene generation plays a crucial role in a range of real-world applications. However, generating a complete 360° scene from a single image remains a highly ill-posed and challenging problem. Recent approaches have made progress by leveraging diffusion models and depth estimation networks, yet they still struggle to maintain appearance consistency and geometric plausibility under large viewpoint changes, limiting their effectiveness in full-scene generation. To address this, we propose AnchoredDream, a novel zero-shot pipeline that anchors 360° scene generation on high-fidelity geometry via an appearance-geometry mutual boosting mechanism. Given a single-view image, our method first performs appearance-guided geometry generation to construct a reliable 3D scene layout. Then, we progressively generate the complete scene through a series of modules: warp-and-inpaint, warp-and-refine, post-optimization, and a novel Grouting Block, which ensures seamless transitions between the input view and generated regions. Extensive experiments demonstrate that AnchoredDream outperforms existing methods by a large margin in both appearance consistency and geometric plausibility--all in a zero-shot manner. Our results highlight the potential of geometric grounding for high-quality, zero-shot single-view scene generation.
Abstract (translated)
URL
https://arxiv.org/abs/2601.16532
PDF
-
Collision-Free Humanoid Traversal in Cluttered Indoor Scenes
2026-01-22 15:08:53Han Xue, Sikai Liang, Zhikai Zhang, Zicheng Zeng, Yun Liu, Yunrui Lian, Jilong Wang, Qingtao Liu, Xuesong Shi, Li YiAbstract
We study the problem of collision-free humanoid traversal in cluttered indoor scenes, such as hurdling over objects scattered on the floor, crouching under low-hanging obstacles, or squeezing through narrow passages. To achieve this goal, the humanoid needs to map its perception of surrounding obstacles with diverse spatial layouts and geometries to the corresponding traversal skills. However, the lack of an effective representation that captures humanoid-obstacle relationships during collision avoidance makes directly learning such mappings difficult. We therefore propose Humanoid Potential Field (HumanoidPF), which encodes these relationships as collision-free motion directions, significantly facilitating RL-based traversal skill learning. We also find that HumanoidPF exhibits a surprisingly negligible sim-to-real gap as a perceptual representation. To further enable generalizable traversal skills through diverse and challenging cluttered indoor scenes, we further propose a hybrid scene generation method, incorporating crops of realistic 3D indoor scenes and procedurally synthesized obstacles. We successfully transfer our policy to the real world and develop a teleoperation system where users could command the humanoid to traverse in cluttered indoor scenes with just a single click. Extensive experiments are conducted in both simulation and the real world to validate the effectiveness of our method. Demos and code can be found in our website: this https URL.
Abstract (translated)
我们研究了在充满障碍物的室内环境中实现人类机器人无碰撞行走的问题，包括跨越地面上散落的物体、低矮障碍物下蹲通过或穿过狭窄通道等情况。为了达到这一目标，人类机器人需要将其对周围障碍物的各种空间布局和几何形状的认知映射到相应的行走技能上。然而，在避免碰撞的过程中缺乏有效的表示方法来捕捉人形机器人与障碍物之间的关系，这使得直接学习这种映射变得困难。因此，我们提出了“人形势场”（HumanoidPF），它将这些关系编码为无碰撞运动方向，从而显著促进了基于强化学习的行走技能的学习过程。此外，我们发现作为感知表示方法，HumanoidPF在模拟环境与真实环境之间的差异非常小。 为了进一步使机器人能够在多样且具有挑战性的充满障碍物的室内环境中具备泛化的行走能力，我们还提出了一种混合场景生成方法，结合了现实世界3D室内环境的截屏以及程序合成的障碍物。我们成功地将策略转移到实际环境中，并开发了一个遥操作系统，在该系统中用户只需点击一下即可指挥人形机器人在充满障碍物的室内环境中行走。 我们在模拟和真实世界环境中进行了广泛的实验以验证我们的方法的有效性。演示视频和代码可以在我们的网站上找到：此链接。
URL
https://arxiv.org/abs/2601.16035
PDF
-
ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation
2026-01-21 17:53:21Hanlei Guo, Jiahao Shao, Xinya Chen, Xiyang Tan, Sheng Miao, Yujun Shen, Yiyi LiaoAbstract
Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach.
Abstract (translated)
最近使用扩散模型生成三维物体取得了显著的成功，但生成逼真的三维城市场景依然具有挑战性。现有的仅依赖于3D扩散模型的方法往往会损失外观细节的清晰度，而那些仅仅利用2D扩散模型的方法则会牺牲相机控制能力。为了克服这一局限性，我们提出了ScenDi方法，这是一种结合了3D和2D扩散模型的城市场景生成技术。 首先，我们训练一个三维潜在扩散模型来生成三维高斯分布，这使得以相对较低的分辨率渲染图像成为可能。为了实现可控合成，这个三维高斯生成过程可以被指定输入（如三维边界框、道路地图或文本提示）所条件化。接着，我们训练了一个2D视频扩散模型，在此基础上提升从3D高斯渲染出的图像中的外观细节。 通过利用粗略的3D场景作为引导来指导2D视频扩散，ScenDi能够根据输入条件生成所需的场景，并且成功地遵循精确的相机轨迹。我们在两个具有挑战性的真实数据集（Waymo和KITTI-360）上的实验结果证明了我们方法的有效性。
URL
https://arxiv.org/abs/2601.15221
PDF
-
HCVR Scene Generation: High Compatibility Virtual Reality Environment Generation for Extended Redirected Walking
2026-01-21 05:51:03Yiran Zhang, Xingpeng Sun, Aniket BeraAbstract
Natural walking enhances immersion in virtual environments (VEs), but physical space limitations and obstacles hinder exploration, especially in large virtual scenes. Redirected Walking (RDW) techniques mitigate this by subtly manipulating the virtual camera to guide users away from physical collisions within pre-defined VEs. However, RDW efficacy diminishes significantly when substantial geometric divergence exists between the physical and virtual environments, leading to unavoidable collisions. Existing scene generation methods primarily focus on object relationships or layout aesthetics, often neglecting the crucial aspect of physical compatibility required for effective RDW. To address this, we introduce HCVR (High Compatibility Virtual Reality Environment Generation), a novel framework that generates virtual scenes inherently optimized for alignment-based RDW controllers. HCVR first employs ENI++, a novel, boundary-sensitive metric to evaluate the incompatibility between physical and virtual spaces by comparing rotation-sensitive visibility polygons. Guided by the ENI++ compatibility map and user prompts, HCVR utilizes a Large Language Model (LLM) for context-aware 3D asset retrieval and initial layout generation. The framework then strategically adjusts object selection, scaling, and placement to maximize coverage of virtually incompatible regions, effectively guiding users towards RDW-feasible paths. User studies evaluating physical collisions and layout quality demonstrate HCVR's effectiveness with HCVR-generated scenes, resulting in 22.78 times fewer physical collisions and received 35.89\% less on ENI++ score compared to LLM-based generation with RDW, while also receiving 12.5\% higher scores on user feedback to layout design.
Abstract (translated)
自然行走可以增强虚拟环境（VE）中的沉浸感，但物理空间的限制和障碍物会阻碍探索，尤其是在大型虚拟场景中。重定向行走（RDW）技术通过微妙地操控虚拟摄像机来引导用户远离预定义虚拟环境中不可避免的身体碰撞，从而缓解这一问题。然而，当物理和虚拟环境之间存在显著的几何差异时，RDW的有效性会大幅下降，导致无法避免的碰撞发生。现有的场景生成方法主要关注对象之间的关系或布局美学，常常忽略了有效实现RDW所需的至关重要的物理兼容性方面。 为解决这个问题，我们提出了HCVR（高兼容性的虚拟现实环境生成），这是一种新颖的框架，能够根据基于对齐的重定向行走控制器的需求来生成优化后的虚拟场景。HCVR首先使用ENI++这一新提出的、具备边界敏感特性的度量标准来评估物理空间和虚拟空间之间的不兼容性，通过对比旋转敏感可见多边形来实现这一点。在ENI++兼容地图和用户提示的指导下，HCVR利用大型语言模型（LLM）进行上下文感知的3D资产检索以及初始布局生成。 随后，该框架策略地调整对象选择、缩放和平移以最大化虚拟不兼容区域的覆盖面积，从而有效地引导用户走向可行的重定向行走路径。通过评估物理碰撞和布局质量的用户研究显示，与仅使用LLM进行基于RDW的场景生成相比，HCVR生成的场景中的物理碰撞减少了22.78倍，并且在ENI++评分上降低了35.89%，同时从用户对布局设计反馈的评分中也获得了12.5%的增长。
URL
https://arxiv.org/abs/2601.14679
PDF
-
Class-Partitioned VQ-VAE and Latent Flow Matching for Point Cloud Scene Generation
2026-01-18 13:05:35Dasith de Silva Edirimuni, Ajmal Saeed MianAbstract
Most 3D scene generation methods are limited to only generating object bounding box parameters while newer diffusion methods also generate class labels and latent features. Using object size or latent feature, they then retrieve objects from a predefined database. For complex scenes of varied, multi-categorical objects, diffusion-based latents cannot be effectively decoded by current autoencoders into the correct point cloud objects which agree with target classes. We introduce a Class-Partitioned Vector Quantized Variational Autoencoder (CPVQ-VAE) that is trained to effectively decode object latent features, by employing a pioneering $\textit{class-partitioned codebook}$ where codevectors are labeled by class. To address the problem of $\textit{codebook collapse}$, we propose a $\textit{class-aware}$ running average update which reinitializes dead codevectors within each partition. During inference, object features and class labels, both generated by a Latent-space Flow Matching Model (LFMM) designed specifically for scene generation, are consumed by the CPVQ-VAE. The CPVQ-VAE's class-aware inverse look-up then maps generated latents to codebook entries that are decoded to class-specific point cloud shapes. Thereby, we achieve pure point cloud generation without relying on an external objects database for retrieval. Extensive experiments reveal that our method reliably recovers plausible point cloud scenes, with up to 70.4% and 72.3% reduction in Chamfer and Point2Mesh errors on complex living room scenes.
Abstract (translated)
大多数的三维场景生成方法仅限于生成物体边界框参数，而较新的扩散方法还能生成类别标签和潜在特征。利用这些对象大小或潜在特征信息，它们从预定义数据库中检索对象。对于包含多样且多类别的复杂场景而言，基于扩散的方法无法通过当前自动编码器有效解码为符合目标类别的点云物体。 我们引入了一种名为分类分区向量量化变分自编码器（Class-Partitioned Vector Quantized Variational Autoencoder, CPVQ-VAE）的模型。它通过使用一种创新性的按类别分割代码簿，其中每个代码向量都由其所属类别标记来进行训练，从而能够有效地解码对象潜在特征。 为了解决“代码簿塌陷”问题，我们提出了一种“类别感知”的运行平均更新方法，这种方法可以重新初始化每个分区内的无效代码向量。在推理阶段，CPVQ-VAE会接收由专门用于场景生成的潜在空间流匹配模型（Latent-space Flow Matching Model, LFMM）产生的对象特征和类别标签。 通过使用具有类别感知逆查找功能的CPVQ-VAE，所生成的潜在特性会被映射到代码簿中的相应条目，并且这些条目随后被解码为特定类别的点云形状。这样一来，我们就能实现纯粹基于点云的场景生成，而无需依赖外部对象数据库。 大量的实验表明，我们的方法能够可靠地恢复出合理的点云场景，在复杂的客厅环境中分别减少了高达70.4%和72.3%的Chamfer误差和Point2Mesh误差。
URL
https://arxiv.org/abs/2601.12391
PDF
-
Generating metamers of human scene understanding
2026-01-16 06:24:59Ritik Raina, Abe Leite, Alexandros Graikos, Seoyoung Ahn, Dimitris Samaras, Gregory J. ZelinskyAbstract
Human vision combines low-resolution "gist" information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a "same" or "different" response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers' own fixated regions.
Abstract (translated)
人类的视觉系统结合了来自视网膜周边区域的低分辨率“概览”信息与固定注视点处稀疏但高分辨率的信息，从而构建出对视觉场景的一致理解。本文介绍了MetamerGen这一工具，它用于生成符合潜在的人类场景表示的场景图像。MetamerGen是一种潜扩散模型，结合了从视网膜周边区域获取的概览信息和从固定注视点处获得的信息，以生成与人类在观看某一场景后所形成的理解一致的图像代用品（metamers）。同时从高分辨率（即“中央凹”）和低分辨率输入中生成图像构成了一个新的图到图合成问题。为了解决这一问题，我们引入了双重流表示形式来描述固定注视区域的场景：DINOv2令牌融合来自固定注视点详细特征与周边区域捕捉场景背景信息的降级特征。 为了评估MetamerGen生成的图像与其潜在的人类场景表征之间的感知一致性，我们进行了一个相同/不同行为实验，在该实验中参与者被要求在生成图像和原始图像之间做出“相同”或“不同”的判断。通过这种方式，我们可以识别出与观众形成的场景表示一致的场景代用品。MetamerGen是一个理解场景理解的强大工具。我们的概念验证分析揭示了视觉处理多个层级中的特定特征对人类判断的影响。尽管它可以基于随机注视点生成图像代用品，我们发现当产生的场景条件设定为观众自己的固定区域时，高层次语义一致性最能预测其是否为图像代用品。 该研究展示了通过结合不同层次的视觉信息来合成逼真的场景图像的新方法，并揭示了这些合成图像与人类感知之间的关联。MetamerGen不仅在技术上具有创新性，在理解人类如何处理复杂视觉场景方面也提供了重要的见解。
URL
https://arxiv.org/abs/2601.11675
PDF
-
Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation
2026-01-12 16:20:20Nicolas Sereyjol-Garros, Ellington Kirby, Victor Besnier, Nermin SametAbstract
LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at this https URL.
Abstract (translated)
LiDAR场景合成是解决机器人任务（如自动驾驶）中三维数据稀缺问题的一种新兴解决方案。最近的方法使用扩散模型或流匹配模型来生成逼真的场景，但与包含数百万样本的RGB数据集相比，3D数据仍然非常有限。我们引入了R3DPA，这是第一个利用图像预训练先验解锁LiDAR点云潜力，并采用自我监督三维表示以实现最佳性能的LiDAR场景生成方法。具体来说： 1. 我们将生成模型的中间特征与自监督三维特征对齐，这显著提高了生成质量。 2. 将大规模图像预训练生成模型的知识转移到LiDAR生成中，从而缓解了有限的LiDAR数据集问题。 3. 仅使用无条件模型就能够在推理过程中实现点云控制，进行对象修复和场景混合。 在KITTI-360基准测试上，R3DPA实现了最佳性能。代码和预训练模型可在提供的链接中获得。
URL
https://arxiv.org/abs/2601.07692
PDF
-
Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction
2026-01-07 16:57:30Jiaxin Huang, Yuanbo Yang, Bangbang Yang, Lin Ma, Yuewen Ma, Yiyi LiaoAbstract
We present Gen3R, a method that bridges the strong priors of foundational reconstruction models and video diffusion models for scene-level 3D generation. We repurpose the VGGT reconstruction model to produce geometric latents by training an adapter on its tokens, which are regularized to align with the appearance latents of pre-trained video diffusion models. By jointly generating these disentangled yet aligned latents, Gen3R produces both RGB videos and corresponding 3D geometry, including camera poses, depth maps, and global point clouds. Experiments demonstrate that our approach achieves state-of-the-art results in single- and multi-image conditioned 3D scene generation. Additionally, our method can enhance the robustness of reconstruction by leveraging generative priors, demonstrating the mutual benefit of tightly coupling reconstruction and generative models.
Abstract (translated)
我们介绍了Gen3R方法，该方法将基础重建模型的强先验与视频扩散模型相结合，以实现场景级别的三维生成。通过在VGGT重建模型的令牌上训练一个适配器，我们将VGGT模型重新用于产生几何潜变量，并且这些令牌被正则化以与预训练的视频扩散模型的外观潜变量对齐。通过联合生成这些分离但又对齐的潜变量，Gen3R可以同时生成RGB视频和对应的三维几何结构，包括相机姿态、深度图和全局点云。 实验表明，在单图像和多图像条件下的三维场景生成中，我们的方法达到了最先进的性能水平。此外，我们的方法可以通过利用生成先验来增强重建的鲁棒性，从而展示出紧密耦合重建模型与生成模型的相互收益。
URL
https://arxiv.org/abs/2601.04090
PDF
