Abstract
This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: this https URL
Abstract (translated)
本文旨在从文本描述中生成3D网格材料。与现有的方法不同，我们提出了一种生成段级程序性材料图作为外观表示的方法，该方法支持高品质渲染并提供了编辑方面的很大灵活性。我们不依赖于大量的配对数据，即具有材料图和相应文本描述的3D网格，来训练一个材料图生成模型。相反，我们利用预训练的2D扩散模型作为连接文本和材料图的桥梁。具体来说，我们的方法将形状分解为一系列段，并设计了一个段控扩散模型来合成与网格部分对齐的2D图像。基于生成的图像，我们通过不同的渲染模块初始化材料图的参数，并通过微调来产生与文本描述相符的材料。大量的实验证明，与其他方法相比，我们的框架在真实感、分辨率和平滑度方面具有卓越的性能。项目页面：此链接
URL
https://arxiv.org/abs/2404.17569
