Abstract
We propose a zero-shot method for generating images in arbitrary spaces (e.g., a sphere for 360° panoramas and a mesh surface for texture) using a pretrained image diffusion model. The zero-shot generation of various visual content using a pretrained image diffusion model has been explored mainly in two directions. First, Diffusion Synchronization-performing reverse diffusion processes jointly across different projected spaces while synchronizing them in the target space-generates high-quality outputs when enough conditioning is provided, but it struggles in its absence. Second, Score Distillation Sampling-gradually updating the target space data through gradient descent-results in better coherence but often lacks detail. In this paper, we reveal for the first time the interconnection between these two methods while highlighting their differences. To this end, we propose StochSync, a novel approach that combines the strengths of both, enabling effective performance with weak conditioning. Our experiments demonstrate that StochSync provides the best performance in 360° panorama generation (where image conditioning is not given), outperforming previous finetuning-based methods, and also delivers comparable results in 3D mesh texturing (where depth conditioning is provided) with previous methods.
Abstract (translated)
我们提出了一种零样本方法，用于在任意空间（例如，球体用于360°全景图和网格表面用于纹理）生成图像，采用预训练的图像扩散模型。使用预训练的图像扩散模型进行各种视觉内容的零样本生成主要从两个方向进行了探索。首先，通过同时跨不同投影空间执行反向扩散过程并在目标空间中同步实现高质量输出，当有足够的条件信息时效果良好，但缺乏足够条件时则表现不佳。其次，通过梯度下降逐步更新目标空间数据的分数蒸馏采样方法产生了更好的连贯性，但往往缺少细节。在这篇论文中，我们首次揭示了这两种方法之间的相互联系，并突出了它们的不同之处。为此，我们提出了StochSync这一新方法，结合了两种方法的优点，在条件信息较弱的情况下也能实现有效性能。我们的实验表明，StochSync在360°全景图生成（此时未提供图像条件）中表现出最佳性能，优于以往基于微调的方法，并且在3D网格纹理化（其中提供了深度条件）方面与之前的方法相比结果相当。
URL
https://arxiv.org/abs/2501.15445
