Abstract
Text-to-image (T2I) diffusion models have demonstrated impressive capabilities in generating high-quality images given a text prompt. However, ensuring the prompt-image alignment remains a considerable challenge, i.e., generating images that faithfully align with the prompt's semantics. Recent works attempt to improve the faithfulness by optimizing the latent code, which potentially could cause the latent code to go out-of-distribution and thus produce unrealistic images. In this paper, we propose FRAP, a simple, yet effective approach based on adaptively adjusting the per-token prompt weights to improve prompt-image alignment and authenticity of the generated images. We design an online algorithm to adaptively update each token's weight coefficient, which is achieved by minimizing a unified objective function that encourages object presence and the binding of object-modifier pairs. Through extensive evaluations, we show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets, while having a lower average latency compared to recent latent code optimization methods, e.g., 4 seconds faster than D&B on the COCO-Subject dataset. Furthermore, through visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that FRAP not only improves prompt-image alignment but also generates more authentic images with realistic appearances. We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment, where we observe improvements in both prompt-image alignment and image quality.
Abstract (translated)
文本到图像（T2I）扩散模型在根据文本提示生成高质量图像方面具有令人印象深刻的潜力。然而，确保提示与图像对齐仍然是一个相当大的挑战，即生成与提示语义一致的图像。最近的工作试图通过优化潜在码来提高提示的忠实度，这可能导致潜在码超出分布，从而产生不现实的图像。在本文中，我们提出了一种简单而有效的方法——基于自适应调整每个词的提示权重来改善提示与图像对齐和生成图像的真诚度。我们设计了一个在线算法，通过最小化一个鼓励物体存在和物体词缀对之间的结合的统一目标函数来自适应地更新每个词的权重系数。通过广泛的评估，我们发现FRAP生成的图像与复杂数据集的提示图像对齐显著更高，而与最近的年份码优化方法相比，平均延迟较低，例如在COCO-Subject数据集上快4秒。此外，通过视觉比较和CLIP-IQA-Real指标的评估，我们发现FRAP不仅提高了提示图像对齐的真诚度，而且生成了更逼真的图像。我们还探索了将FRAP与提示重写LLM相结合以恢复其衰减的提示图像对齐，我们观察到提示图像对齐和图像质量都有所提高。
URL
https://arxiv.org/abs/2408.11706
