Abstract
3D scene generation conditioned on text prompts has significantly progressed due to the development of 2D diffusion generation models. However, the textual description of 3D scenes is inherently inaccurate and lacks fine-grained control during training, leading to implausible scene generation. As an intuitive and feasible solution, the 3D layout allows for precise specification of object locations within the scene. To this end, we present a text-to-scene generation method (namely, Layout2Scene) using additional semantic layout as the prompt to inject precise control of 3D object positions. Specifically, we first introduce a scene hybrid representation to decouple objects and backgrounds, which is initialized via a pre-trained text-to-3D model. Then, we propose a two-stage scheme to optimize the geometry and appearance of the initialized scene separately. To fully leverage 2D diffusion priors in geometry and appearance generation, we introduce a semantic-guided geometry diffusion model and a semantic-geometry guided diffusion model which are finetuned on a scene dataset. Extensive experiments demonstrate that our method can generate more plausible and realistic scenes as compared to state-of-the-art approaches. Furthermore, the generated scene allows for flexible yet precise editing, thereby facilitating multiple downstream applications.
Abstract (translated)
基于文本提示的3D场景生成由于2D扩散生成模型的发展而取得了显著进步。然而，对3D场景的文字描述本质上不够准确，并且在训练过程中缺乏细粒度控制，导致生成的场景不合理。作为一种直观可行的解决方案，3D布局允许精确指定场景中对象的位置。为此，我们提出了一种使用附加语义布局作为提示来注入对3D物体位置精准控制的文本到场景生成方法（即Layout2Scene）。具体而言，我们首先引入一种场景混合表示法，以分离物体和背景，并通过预训练的文本到3D模型进行初始化。然后，我们提出了一个两阶段方案，分别优化初始化场景的几何形状和外观。为了充分利用在几何形状和外观生成中的2D扩散先验知识，我们引入了语义引导的几何扩散模型以及基于语义-几何的引导扩散模型，并在这两类模型上使用场景数据集进行微调。 广泛的实验表明，我们的方法能够比最先进的方法生成更合理、更逼真的场景。此外，所生成的场景允许灵活而精确地编辑，从而促进了多种下游应用的发展。
URL
https://arxiv.org/abs/2501.02519
