Abstract
3D content generation has recently attracted significant research interest due to its applications in VR/AR and embodied AI. In this work, we address the challenging task of synthesizing multiple 3D assets within a single scene image. Concretely, our contributions are fourfold: (i) we present SceneGen, a novel framework that takes a scene image and corresponding object masks as input, simultaneously producing multiple 3D assets with geometry and texture. Notably, SceneGen operates with no need for optimization or asset retrieval; (ii) we introduce a novel feature aggregation module that integrates local and global scene information from visual and geometric encoders within the feature extraction module. Coupled with a position head, this enables the generation of 3D assets and their relative spatial positions in a single feedforward pass; (iii) we demonstrate SceneGen's direct extensibility to multi-image input scenarios. Despite being trained solely on single-image inputs, our architectural design enables improved generation performance with multi-image inputs; and (iv) extensive quantitative and qualitative evaluations confirm the efficiency and robust generation abilities of our approach. We believe this paradigm offers a novel solution for high-quality 3D content generation, potentially advancing its practical applications in downstream tasks. The code and model will be publicly available at: this https URL.
Abstract (translated)
最近，由于在虚拟现实（VR）、增强现实（AR）和具身人工智能中的应用，三维内容生成引起了广泛的研究兴趣。在这项工作中，我们解决了一个具有挑战性的任务：从单个场景图像中合成多个三维资产。具体来说，我们的贡献有四个方面： (i) 我们提出了SceneGen，一个新颖的框架，它接受场景图像及其对应对象掩码作为输入，并同时产生带有几何形状和纹理的多个三维资产。值得注意的是，SceneGen无需优化或资产检索即可运行； (ii) 我们引入了一个新颖的功能聚合模块，在特征提取模块中整合了来自视觉和几何编码器的局部和全局场景信息。与位置头相结合，这使得在一个前馈传递过程中生成三维资产及其相对空间位置成为可能； (iii) 我们展示了SceneGen在多图像输入场景中的直接扩展性。尽管仅使用单个图像输入进行训练，但我们的架构设计允许使用多图像输入来提高生成性能； (iv) 全面的定量和定性评估确认了我们方法的有效性和鲁棒生成能力。 我们认为这一范式为高质量三维内容生成提供了新颖解决方案，并有可能推进其在下游任务中的实际应用。代码和模型将在以下网址公开发布：[此URL]。
URL
https://arxiv.org/abs/2508.15769
