Abstract
Despite the impressive progress on understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified understanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Additionally, we propose a geometric-semantic learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both spatial understanding and generation. Extensive experimental results demonstrate the superiority of our method in visual representation, spatial understanding, and 3D generation. The source code will be released upon paper acceptance.
Abstract (translated)
尽管近期统一架构在理解和生成图像方面取得了令人瞩目的进展，但将三维任务整合进来仍然颇具挑战且尚未被广泛探索。本文介绍了UniUGG，这是首个用于三维模态的统一理解与生成框架。我们的统一框架采用大型语言模型（LLM）来解析和解码句子及3D表示。在核心部分，我们提出了一种基于潜在扩散模型的空间解码器，用以生成高质量的3D表示。这使得根据参考图像以及任意视角转换来生成和想象三维场景成为可能，并且支持空间视觉问答（VQA）任务。此外，我们还提出了几何-语义学习策略来进行视觉编码器的预训练。这一设计同时捕捉输入的语义与几何线索，从而增强空间理解和生成能力。广泛的实验结果证明了我们的方法在视觉表示、空间理解及3D生成方面的优越性。论文一经接受，源代码将公开发布。
URL
https://arxiv.org/abs/2508.11952
