Abstract
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.
Abstract (translated)
使用3D资产生成场景是一项复杂的挑战，需要同时具备高层次的语义理解和低层次的几何推理能力。虽然多模态大型语言模型（MLLMs）在语义任务上表现出色，但它们在应用于3D场景生成时受到对3D几何理解有限的影响。在这篇论文中，我们探讨了如何最有效地将MLLMs应用到物体放置的任务中。为此，我们引入了一个新颖的框架FirePlace，该框架利用现有的MLLMs进行： 1. 3D几何推理，并从3D场景中提取相关的几何细节； 2. 构建并解决基于提取出的低层次几何的约束条件； 3. 剪枝以获得符合常识的最终放置位置。 通过结合几何推理和MLLMs对现实世界的理解，我们的方法可以提出既满足几何约束又能考虑高层次语义常识的对象放置建议。实验表明，这些能力使我们的方法能够更有效地在具有复杂几何结构的场景中放置物体，并且其质量超过了先前的工作。
URL
https://arxiv.org/abs/2503.04919
