Abstract
Diffusion models have revolutionized image generation, yet several challenges restrict their application to large-image domains, such as digital pathology and satellite imagery. Given that it is infeasible to directly train a model on 'whole' images from domains with potential gigapixel sizes, diffusion-based generative methods have focused on synthesizing small, fixed-size patches extracted from these images. However, generating small patches has limited applicability since patch-based models fail to capture the global structures and wider context of large images, which can be crucial for synthesizing (semantically) accurate samples. In this paper, to overcome this limitation, we present ZoomLDM, a diffusion model tailored for generating images across multiple scales. Central to our approach is a novel magnification-aware conditioning mechanism that utilizes self-supervised learning (SSL) embeddings and allows the diffusion model to synthesize images at different 'zoom' levels, i.e., fixed-size patches extracted from large images at varying scales. ZoomLDM achieves state-of-the-art image generation quality across all scales, excelling particularly in the data-scarce setting of generating thumbnails of entire large images. The multi-scale nature of ZoomLDM unlocks additional capabilities in large image generation, enabling computationally tractable and globally coherent image synthesis up to $4096 \times 4096$ pixels and $4\times$ super-resolution. Additionally, multi-scale features extracted from ZoomLDM are highly effective in multiple instance learning experiments. We provide high-resolution examples of the generated images on our website this https URL.
Abstract (translated)
扩散模型彻底改变了图像生成领域，然而一些挑战限制了它们在大型图像领域的应用，比如数字病理学和卫星影像。由于直接对潜在的数十亿像素大小的“完整”图像进行训练是不切实际的，在这些领域中，基于扩散的生成方法主要集中于合成从这些图像中提取的小型固定尺寸块。然而，生成小型补丁的应用范围有限，因为基于补丁的方法无法捕捉到大型图像的整体结构和更广泛的上下文信息，这对于合成（语义上）准确的样本可能是至关重要的。 在这篇论文中，为了解决这一限制，我们提出了ZoomLDM，一种专为多尺度图像生成定制的扩散模型。我们的方法核心是一种新颖的比例感知条件机制，该机制利用自监督学习（SSL）嵌入，并允许扩散模型在不同的“缩放”级别上合成图像，即从大型图像中提取的不同比例下的固定尺寸块。 ZoomLDM在所有尺度上的图像生成质量达到了最先进的水平，在稀缺数据环境下尤其擅长于生成整个大型图像的缩略图。ZoomLDM的多尺度特性解锁了大型图像生成中的额外能力，使得能够进行计算上可行且全局一致的图像合成，最高可达$4096 \times 4096$像素和$4\times$超分辨率。 此外，从ZoomLDM中提取的多尺度特征在多次实例学习实验中表现出高度有效性。我们可以在我们的网站（此处提供此 https URL）上查看生成图像的高分辨率示例。
URL
https://arxiv.org/abs/2411.16969
