Abstract
Painting textures for existing geometries is a critical yet labor-intensive process in 3D asset generation. Recent advancements in text-to-image (T2I) models have led to significant progress in texture generation. Most existing research approaches this task by first generating images in 2D spaces using image diffusion models, followed by a texture baking process to achieve UV texture. However, these methods often struggle to produce high-quality textures due to inconsistencies among the generated multi-view images, resulting in seams and ghosting artifacts. In contrast, 3D-based texture synthesis methods aim to address these inconsistencies, but they often neglect 2D diffusion model priors, making them challenging to apply to real-world objects To overcome these limitations, we propose RomanTex, a multiview-based texture generation framework that integrates a multi-attention network with an underlying 3D representation, facilitated by our novel 3D-aware Rotary Positional Embedding. Additionally, we incorporate a decoupling characteristic in the multi-attention block to enhance the model's robustness in image-to-texture task, enabling semantically-correct back-view synthesis. Furthermore, we introduce a geometry-related Classifier-Free Guidance (CFG) mechanism to further improve the alignment with both geometries and images. Quantitative and qualitative evaluations, along with comprehensive user studies, demonstrate that our method achieves state-of-the-art results in texture quality and consistency.
Abstract (translated)
为现有几何体绘制纹理是3D资产生成中的一个关键但耗时的过程。最近，文本到图像（T2I）模型的进展在纹理生成方面取得了显著进步。大多数现有的研究方法首先使用图像扩散模型在二维空间中生成图像，然后通过烘焙过程实现UV纹理。然而，这些方法往往难以产生高质量的纹理，因为它们生成的多视角图像之间存在不一致性，从而导致接缝和鬼影（ghosting）伪影。相比之下，基于3D的方法旨在解决这些问题，但通常忽视了二维扩散模型的先验知识，使其在应用于现实世界对象时具有挑战性。 为克服这些限制，我们提出了RomanTex，这是一种多视角纹理生成框架，它将一个多注意力网络与底层3D表示相结合，并通过我们的新型3D感知旋转位置嵌入（Rotary Positional Embedding）来实现。此外，我们在多注意力块中引入了一种解耦特性，以增强模型在图像到纹理任务中的鲁棒性，从而能够生成语义正确的背面合成结果。我们还引入了一种与几何相关的无分类器引导（Classifier-Free Guidance, CFG）机制，进一步提高纹理与几何和图像的一致性。 定量评估、定性评估以及全面的用户研究证明了我们的方法在纹理质量和一致性方面取得了最先进的成果。
URL
https://arxiv.org/abs/2503.19011
