Image_Generation
-
TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark
2025-03-17 21:36:31Forouzan Fallah, Maitreya Patel, Agneet Chatterjee, Vlad I. Morariu, Chitta Baral, Yezhou YangAbstract
Generating images with embedded text is crucial for the automatic production of visual and multimodal documents, such as educational materials and advertisements. However, existing diffusion-based text-to-image models often struggle to accurately embed text within images, facing challenges in spelling accuracy, contextual relevance, and visual coherence. Evaluating the ability of such models to embed text within a generated image is complicated due to the lack of comprehensive benchmarks. In this work, we introduce TextInVision, a large-scale, text and prompt complexity driven benchmark designed to evaluate the ability of diffusion models to effectively integrate visual text into images. We crafted a diverse set of prompts and texts that consider various attributes and text characteristics. Additionally, we prepared an image dataset to test Variational Autoencoder (VAE) models across different character representations, highlighting that VAE architectures can also pose challenges in text generation within diffusion frameworks. Through extensive analysis of multiple models, we identify common errors and highlight issues such as spelling inaccuracies and contextual mismatches. By pinpointing the failure points across different prompts and texts, our research lays the foundation for future advancements in AI-generated multimodal content.
Abstract (translated)
将包含嵌入文本的图像生成对于自动生产视觉和多模态文档（如教育材料和广告）至关重要。然而，现有的基于扩散的文字到图像模型在准确地将文字嵌入图像方面经常遇到困难，面临着拼写准确性、上下文相关性和视觉连贯性等方面的挑战。由于缺乏全面的基准测试，评估此类模型在其生成的图像中嵌入文本的能力变得复杂。在这项工作中，我们引入了TextInVision，这是一个大规模的、以文本和提示复杂度驱动的基准测试系统，旨在评估扩散模型将视觉文字有效整合到图像中的能力。我们精心设计了一组多样化的提示语和文本，考虑了各种属性和文本特征。此外，我们还准备了一个图像数据集来测试变分自编码器（VAE）模型在不同字符表示上的性能，并强调VAE架构同样可能面临在扩散框架内生成文字方面的挑战。通过对多个模型进行广泛的分析，我们识别出常见的错误并指出了诸如拼写不准确和上下文不符等问题。通过确定不同类型提示语和文本中的失败点，我们的研究为未来AI生成的多模态内容的进步奠定了基础。
URL
https://arxiv.org/abs/2503.13730
PDF
-
Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait
2025-03-17 09:18:31Chaolong Yang, Kai Yao, Yuyao Yan, Chenru Jiang, Weiguang Zhao, Jie Sun, Guangliang Cheng, Yifei Zhang, Bin Dong, Kaizhu HuangAbstract
Audio-driven single-image talking portrait generation plays a crucial role in virtual reality, digital human creation, and filmmaking. Existing approaches are generally categorized into keypoint-based and image-based methods. Keypoint-based methods effectively preserve character identity but struggle to capture fine facial details due to the fixed points limitation of the 3D Morphable Model. Moreover, traditional generative networks face challenges in establishing causality between audio and keypoints on limited datasets, resulting in low pose diversity. In contrast, image-based approaches produce high-quality portraits with diverse details using the diffusion network but incur identity distortion and expensive computational costs. In this work, we propose KDTalker, the first framework to combine unsupervised implicit 3D keypoint with a spatiotemporal diffusion model. Leveraging unsupervised implicit 3D keypoints, KDTalker adapts facial information densities, allowing the diffusion process to model diverse head poses and capture fine facial details flexibly. The custom-designed spatiotemporal attention mechanism ensures accurate lip synchronization, producing temporally consistent, high-quality animations while enhancing computational efficiency. Experimental results demonstrate that KDTalker achieves state-of-the-art performance regarding lip synchronization accuracy, head pose diversity, and execution this http URL codes are available at this https URL.
Abstract (translated)
音频驱动的单张图像说话肖像生成在虚拟现实、数字人创建和电影制作中扮演着重要角色。现有的方法通常被分为基于关键点的方法和基于图像的方法。基于关键点的方法有效地保留了人物的身份，但因3D可变形模型（3D Morphable Model）固定点限制而难以捕捉到精细的面部细节。此外，传统的生成网络在使用有限数据集建立音频与关键点之间的因果关系时面临挑战，导致姿势多样性较低。相比之下，基于图像的方法通过扩散网络能够产生具有丰富细节和高质量的人物肖像，但会导致身份失真，并且计算成本高昂。 在这项工作中，我们提出了KDTalker框架，这是第一个结合无监督隐式3D关键点与时空扩散模型的框架。利用无监督隐式的3D关键点，KDTalker可以根据面部信息密度调整面部信息，使得扩散过程能够灵活地模拟多样的头部姿势，并捕捉到细微的面部细节。此外，自定义设计的时空注意力机制确保了准确的唇部同步，在提高计算效率的同时产生了时间一致性高、高质量的动画。 实验结果表明，KDTalker在唇部同步准确性、头部姿态多样性以及执行效率方面均达到了最先进的水平。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2503.12963
PDF
-
PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models
2025-03-13 07:37:09Runze He, Bo Cheng, Yuhang Ma, Qingxiang Jia, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Liebucha Wu, Dawei Leng, Yuhui YinAbstract
In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: this https URL.
Abstract (translated)
在这篇论文中，我们提出了一种统一的布局规划和图像生成模型——PlanGen。该模型能够在生成图像之前预先计划空间布局条件。与以往基于扩散的方法将布局规划和从布局到图像转换视为两个独立模型不同，PlanGen 将这两个任务整合为一个自回归变压器模型，并仅通过下一个标记预测来实现这一目标。PlanGen 在不专门编码局部说明文本和边界框坐标的情况下，将布局条件集成到模型上下文中，这在处理复杂布局时相比之前的嵌入并池化操作提供了显著的优势。 统一的提示机制使 PlanGen 能够进行与布局相关的多任务训练，包括布局规划、从布局生成图像、理解图像中的布局等。此外，由于其设计得当，PlanGen 还可以无缝扩展到基于布局引导的图像编辑功能，并且通过教师强制内容操作策略和负向布局指导来实现这一点。 广泛的实验验证了 PlanGen 在多个与布局相关的任务中的有效性，展示了其巨大的潜力。代码可在以下网址获取：[此处插入具体链接]。
URL
https://arxiv.org/abs/2503.10127
PDF
-
Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks
2025-03-11 10:34:57Junying Wang, Hongyuan Zhang, Yuan YuanAbstract
Recent Customized Portrait Generation (CPG) methods, taking a facial image and a textual prompt as inputs, have attracted substantial attention. Although these methods generate high-fidelity portraits, they fail to prevent the generated portraits from being tracked and misused by malicious face recognition systems. To address this, this paper proposes a Customized Portrait Generation framework with facial Adversarial attacks (Adv-CPG). Specifically, to achieve facial privacy protection, we devise a lightweight local ID encryptor and an encryption enhancer. They implement progressive double-layer encryption protection by directly injecting the target identity and adding additional identity guidance, respectively. Furthermore, to accomplish fine-grained and personalized portrait generation, we develop a multi-modal image customizer capable of generating controlled fine-grained facial features. To the best of our knowledge, Adv-CPG is the first study that introduces facial adversarial attacks into CPG. Extensive experiments demonstrate the superiority of Adv-CPG, e.g., the average attack success rate of the proposed Adv-CPG is 28.1% and 2.86% higher compared to the SOTA noise-based attack methods and unconstrained attack methods, respectively.
Abstract (translated)
最近定制化肖像生成（CPG）方法，通过面部图像和文本提示作为输入，引起了广泛关注。尽管这些方法能够生成高保真的面部画像，但它们无法防止生成的画像被恶意人脸识别系统追踪并滥用。为了解决这个问题，本文提出了一种结合了面部对抗攻击的定制化肖像生成框架（Adv-CPG）。为了实现面部隐私保护，我们设计了一个轻量级的身份加密器和一个增强器。身份加密器通过直接注入目标身份信息来实施渐进式的双层加密保护；而增强器则是通过添加额外的身份指导来加强这一过程。 此外，为了完成精细且个性化的肖像生成，我们开发了一种多模态图像定制器，能够生成受控的细粒度面部特征。据我们所知，Adv-CPG是第一个将面部对抗攻击引入到CPG中的研究工作。广泛的实验展示了Adv-CPG的优势，例如：提出的Adv-CPG方法平均攻击成功率达到了28.1%，比最先进的基于噪声的方法高出2.86%。 翻译成中文如下： 最近定制化肖像生成（Customized Portrait Generation, CPG）方法因其能够利用面部图像和文本提示作为输入而吸引了大量关注。尽管这些方法可以生成高保真的面部画像，但它们无法防止生成的画像被恶意的人脸识别系统追踪并滥用。为了解决这个问题，本文提出了一种结合了面部对抗攻击（Adversarial attacks, Adv）的定制化肖像生成框架（Adv-CPG）。为了实现面部隐私保护，我们设计了一个轻量级的身份加密器和一个增强器。身份加密器通过直接注入目标身份信息来实施渐进式的双层加密保护；而增强器则是通过添加额外的身份指导来加强这一过程。 此外，为了完成精细且个性化的肖像生成，我们开发了一种多模态图像定制器，能够生成受控的细粒度面部特征。据我们所知，Adv-CPG是第一个将面部对抗攻击引入到CPG中的研究工作。广泛的实验展示了Adv-CPG的优势，例如：提出的Adv-CPG方法平均攻击成功率达到了28.1%，比最先进的基于噪声的方法高出2.86%。
URL
https://arxiv.org/abs/2503.08269
PDF
-
Recovering Partially Corrupted Major Objects through Tri-modality Based Image Completion
2025-03-10 08:34:31Yongle Zhang, Yimin Liu, Qiang WuAbstract
Diffusion models have become widely adopted in image completion tasks, with text prompts commonly employed to ensure semantic coherence by providing high-level guidance. However, a persistent challenge arises when an object is partially obscured in the damaged region, yet its remaining parts are still visible in the background. While text prompts offer semantic direction, they often fail to precisely recover fine-grained structural details, such as the object's overall posture, ensuring alignment with the visible object information in the background. This limitation stems from the inability of text prompts to provide pixel-level specificity. To address this, we propose supplementing text-based guidance with a novel visual aid: a casual sketch, which can be roughly drawn by anyone based on visible object parts. This sketch supplies critical structural cues, enabling the generative model to produce an object structure that seamlessly integrates with the existing background. We introduce the Visual Sketch Self-Aware (VSSA) model, which integrates the casual sketch into each iterative step of the diffusion process, offering distinct advantages for partially corrupted scenarios. By blending sketch-derived features with those of the corrupted image, and leveraging text prompt guidance, the VSSA assists the diffusion model in generating images that preserve both the intended object semantics and structural consistency across the restored objects and original regions. To support this research, we created two datasets, CUB-sketch and MSCOCO-sketch, each combining images, sketches, and text. Extensive qualitative and quantitative experiments demonstrate that our approach outperforms several state-of-the-art methods.
Abstract (translated)
扩散模型在图像补全任务中被广泛采用，通常使用文本提示来确保语义连贯性并提供高层次的指导。然而，当目标物体的部分区域被损坏遮挡时，尽管其剩余部分仍可在背景中看到，这便成为一个持久性的挑战。虽然文本提示提供了语义方向，但它们往往无法精确恢复细微结构细节，例如物体的整体姿态，以确保与背景中可见物体信息的一致性。这一限制源于文本提示无法提供像素级别的具体信息。 为了解决这个问题，我们提出了一种新颖的方法：在基于文本的指导之外添加视觉辅助——一个简单的草图，任何人都可以根据看到的目标物部分轻松绘制出来。这个草图提供了关键的结构线索，使生成模型能够产出与现有背景无缝融合的对象结构。为此，我们引入了“视觉草图自我感知”（Visual Sketch Self-Aware, VSSA）模型，该模型将草图融入扩散过程中的每一步迭代中，在部分损坏的情况下提供独特的优势。 通过结合从草图和受损图像衍生出的特征，并利用文本提示指导，VSSA帮助扩散模型生成既能保持意图对象语义又能确保修复区域与原区结构一致性的图像。为了支持这项研究，我们创建了两个数据集：CUB-sketch 和 MSCOCO-sketch，每个都结合了图像、草图和文本信息。通过广泛的质量和数量的实验验证，我们的方法在多个最先进的技术对比中表现出色。
URL
https://arxiv.org/abs/2503.07047
PDF
-
Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help
2025-03-10 03:28:18Yuefan Cao, Xuyang Guo, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang, Zhen ZhuangAbstract
Generative modeling is widely regarded as one of the most essential problems in today's AI community, with text-to-image generation having gained unprecedented real-world impacts. Among various approaches, diffusion models have achieved remarkable success and have become the de facto solution for text-to-image generation. However, despite their impressive performance, these models exhibit fundamental limitations in adhering to numerical constraints in user instructions, frequently generating images with an incorrect number of objects. While several prior works have mentioned this issue, a comprehensive and rigorous evaluation of this limitation remains lacking. To address this gap, we introduce T2ICountBench, a novel benchmark designed to rigorously evaluate the counting ability of state-of-the-art text-to-image diffusion models. Our benchmark encompasses a diverse set of generative models, including both open-source and private systems. It explicitly isolates counting performance from other capabilities, provides structured difficulty levels, and incorporates human evaluations to ensure high reliability. Extensive evaluations with T2ICountBench reveal that all state-of-the-art diffusion models fail to generate the correct number of objects, with accuracy dropping significantly as the number of objects increases. Additionally, an exploratory study on prompt refinement demonstrates that such simple interventions generally do not improve counting accuracy. Our findings highlight the inherent challenges in numerical understanding within diffusion models and point to promising directions for future improvements.
Abstract (translated)
生成式模型被广泛认为是当今AI社区中最基本的问题之一，其中文本到图像的生成已经产生了前所未有的现实世界影响。在各种方法中，扩散模型取得了显著的成功，并已成为文本到图像生成的标准解决方案。然而，尽管它们表现出色，这些模型在遵守用户指令中的数值约束方面存在根本性限制，经常生成包含错误对象数量的图片。虽然一些先前的研究工作提到了这个问题，但对其进行全面而严格的评估仍然不足。 为了填补这一空白，我们引入了T2ICountBench，这是一个新型基准测试平台，旨在严格评估最先进的文本到图像扩散模型在计数方面的能力。该基准涵盖了各种生成式模型，包括开源和专有系统，并明确地将计数性能与其他能力区分开来。它提供结构化的难度级别，并结合了人类评价以确保高可靠性。 使用T2ICountBench进行的广泛评估表明，所有最先进的扩散模型在生成正确的对象数量方面都失败了，随着对象数量的增加，准确性显著下降。此外，关于提示调整的研究发现，此类简单干预通常不会提高计数精度。 我们的研究结果突出了扩散模型中数值理解所面临的内在挑战，并为未来改进指明了有前景的方向。
URL
https://arxiv.org/abs/2503.06884
PDF
-
RectifiedHR: Enable Efficient High-Resolution Image Generation via Energy Rectification
2025-03-04 12:03:26Zhen Yang, Guibao Shen, Liang Hou, Mushui Liu, Luozhou Wang, Xin Tao, Pengfei Wan, Di Zhang, Ying-Cong ChenAbstract
Diffusion models have achieved remarkable advances in various image generation tasks. However, their performance notably declines when generating images at resolutions higher than those used during the training period. Despite the existence of numerous methods for producing high-resolution images, they either suffer from inefficiency or are hindered by complex operations. In this paper, we propose RectifiedHR, an efficient and straightforward solution for training-free high-resolution image generation. Specifically, we introduce the noise refresh strategy, which theoretically only requires a few lines of code to unlock the model's high-resolution generation ability and improve efficiency. Additionally, we first observe the phenomenon of energy decay that may cause image blurriness during the high-resolution image generation process. To address this issue, we propose an Energy Rectification strategy, where modifying the hyperparameters of the classifier-free guidance effectively improves the generation performance. Our method is entirely training-free and boasts a simple implementation logic. Through extensive comparisons with numerous baseline methods, our RectifiedHR demonstrates superior effectiveness and efficiency.
Abstract (translated)
扩散模型在各种图像生成任务中取得了显著的进展。然而，当它们用于生成高于训练期间所用分辨率的高分辨率图像时，其性能会明显下降。尽管存在许多可以产生高质量高分辨率图像的方法，但这些方法要么效率低下，要么受到复杂操作的限制。在这篇论文中，我们提出了RectifiedHR，这是一种无需训练即可生成高分辨率图像的有效且简单的解决方案。具体来说，我们引入了噪声刷新策略，理论上仅需修改几行代码就能解锁模型的高分辨率生成能力，并提高其效率。此外，我们首次观察到了在高分辨率图像生成过程中可能出现的能量衰减现象，这可能导致图像模糊。为了解决这个问题，我们提出了能量校正策略，在该策略中，通过调整无分类器引导（classifier-free guidance）中的超参数可以有效提升生成性能。我们的方法完全不需要重新训练，并且拥有简单明了的实现逻辑。通过与众多基线方法进行广泛的比较，RectifiedHR展示了其在有效性及效率上的优越性。
URL
https://arxiv.org/abs/2503.02537
PDF
-
Finding Local Diffusion Schr'odinger Bridge using Kolmogorov-Arnold Network
2025-02-27 04:34:03Xingyu Qiu, Mengying Yang, Xinghua Ma, Fanding Li, Dong Liang, Gongning Luo, Wei Wang, Kuanquan Wang, Shuo LiAbstract
In image generation, Schrödinger Bridge (SB)-based methods theoretically enhance the efficiency and quality compared to the diffusion models by finding the least costly path between two distributions. However, they are computationally expensive and time-consuming when applied to complex image data. The reason is that they focus on fitting globally optimal paths in high-dimensional spaces, directly generating images as next step on the path using complex networks through self-supervised training, which typically results in a gap with the global optimum. Meanwhile, most diffusion models are in the same path subspace generated by weights $f_A(t)$ and $f_B(t)$, as they follow the paradigm ($x_t = f_A(t)x_{Img} + f_B(t)\epsilon$). To address the limitations of SB-based methods, this paper proposes for the first time to find local Diffusion Schrödinger Bridges (LDSB) in the diffusion path subspace, which strengthens the connection between the SB problem and diffusion models. Specifically, our method optimizes the diffusion paths using Kolmogorov-Arnold Network (KAN), which has the advantage of resistance to forgetting and continuous output. The experiment shows that our LDSB significantly improves the quality and efficiency of image generation using the same pre-trained denoising network and the KAN for optimising is only less than 0.1MB. The FID metric is reduced by \textbf{more than 15\%}, especially with a reduction of 48.50\% when NFE of DDIM is $5$ for the CelebA dataset. Code is available at this https URL.
Abstract (translated)
在图像生成领域，基于薛定谔桥（Schrödinger Bridge, SB）的方法理论上通过寻找两个分布之间的最低成本路径来提高效率和质量，相比扩散模型（diffusion models）有所改进。然而，在应用于复杂图像数据时，这些方法计算成本高昂且耗时长。原因是它们专注于在高维空间中拟合全局最优路径，并直接利用复杂的网络进行自监督训练生成下一步的图像，这通常会导致与全局最优解之间的差距。 另一方面，大多数扩散模型遵循范式 $(x_t = f_A(t)x_{\text{Img}} + f_B(t)\epsilon)$，它们也位于由权重 $f_A(t)$ 和 $f_B(t)$ 生成的路径子空间中。为了解决基于SB方法的局限性，本文首次提出在扩散路径子空间内寻找局部扩散薛定谔桥（Local Diffusion Schrödinger Bridge, LDSB），这加强了SB问题与扩散模型之间的联系。 具体来说，我们的方法使用Kolmogorov-Arnold网络（KAN）来优化扩散路径。该网络具有抵抗遗忘和连续输出的优点。实验表明，在使用相同的预训练去噪网络的情况下，LDSB显著提高了图像生成的质量和效率，并且用于优化的KAN模型大小仅为不到0.1MB。在CelebA数据集上，当DDIM的NFE为5时，FID度量减少了**超过15%**，特别是在这种情况下减少幅度达到了48.50%。 代码可以在以下网址获取：[此链接](https://this https URL) （请将“this https URL”替换为实际提供的链接）。
URL
https://arxiv.org/abs/2502.19754
PDF
-
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinment
2025-02-24 06:56:56Suchae Jeong, Inseong Choi, Youngsik Yun, Jihie KimAbstract
Text-to-Image models, including Stable Diffusion, have significantly improved in generating images that are highly semantically aligned with the given prompts. However, existing models may fail to produce appropriate images for the cultural concepts or objects that are not well known or underrepresented in western cultures, such as `hangari' (Korean utensil). In this paper, we propose a novel approach, Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement (Culture-TRIP), which refines the prompt in order to improve the alignment of the image with such culture nouns in text-to-image models. Our approach (1) retrieves cultural contexts and visual details related to the culture nouns in the prompt and (2) iteratively refines and evaluates the prompt based on a set of cultural criteria and large language models. The refinement process utilizes the information retrieved from Wikipedia and the Web. Our user survey, conducted with 66 participants from eight different countries demonstrates that our proposed approach enhances the alignment between the images and the prompts. In particular, C-TRIP demonstrates improved alignment between the generated images and underrepresented culture nouns. Resource can be found at this https URL.
Abstract (translated)
文本到图像模型（如Stable Diffusion）在生成与给定提示高度语义对齐的图像方面取得了显著进步。然而，现有的模型可能无法为西方文化中不常见或代表性不足的文化概念或物品（例如韩国器皿“hangari”）生成适当的图像。为此，在本文中我们提出了一种新颖的方法——具有迭代提示优化的文化感知文本到图像生成（Culture-TRIP），该方法通过改进提示来提高这些文化名词在文本到图像模型中的对齐度。 我们的方法包括： 1. 检索与提示中文化名词相关的文化背景和视觉细节； 2. 根据一系列文化和大规模语言模型的评估，迭代优化和评价提示。此过程利用从维基百科和网络检索的信息。 我们进行了一项用户调查，共66名来自8个不同国家的参与者参与其中。调查显示我们的方法能够增强图像与提示之间的对齐度，特别是在生成图片与其文化名词的相关性方面得到了改进。相关资源可在提供的链接中找到。
URL
https://arxiv.org/abs/2502.16902
PDF
-
Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models
2025-02-11 11:17:39Lin Zhu, Xinbing Wang, Chenghu Zhou, Qinying Gu, Nanyang YeAbstract
Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.
Abstract (translated)
在给定风格参考图像作为附加条件的情况下，文本到图像的扩散模型展示出了生成同时具备文本提示内容并采用参考图像视觉风格的图片的强大能力。然而，目前最先进的方法常常难以从风格参考图中分离出内容和风格，导致诸如内容泄露等问题的发生。为解决这一问题，我们提出了一种基于掩码的方法，能够在不调整任何模型参数的情况下有效分离内容与风格。通过简单地在风格参考图像特征的特定元素上进行掩码处理，我们揭示了一个重要但尚未充分探索的原则：使用适当选择的较少条件（例如，删除几个图像特征元素）可以有效地避免不必要的内容流入扩散模型，从而提升文本到图像扩散模型的风格迁移性能。本文中，我们在理论和实验两方面验证了这一发现的有效性。跨多种风格进行的大量实验展示了我们基于掩码的方法的有效性，并支持我们的理论结果。
URL
https://arxiv.org/abs/2502.07466
PDF
-
Universal Approximation of Visual Autoregressive Transformers
2025-02-10 05:36:30Yifang Chen, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao SongAbstract
We investigate the fundamental limits of transformer-based foundation models, extending our analysis to include Visual Autoregressive (VAR) transformers. VAR represents a big step toward generating images using a novel, scalable, coarse-to-fine ``next-scale prediction'' framework. These models set a new quality bar, outperforming all previous methods, including Diffusion Transformers, while having state-of-the-art performance for image synthesis tasks. Our primary contributions establish that, for single-head VAR transformers with a single self-attention layer and single interpolation layer, the VAR Transformer is universal. From the statistical perspective, we prove that such simple VAR transformers are universal approximators for any image-to-image Lipschitz functions. Furthermore, we demonstrate that flow-based autoregressive transformers inherit similar approximation capabilities. Our results provide important design principles for effective and computationally efficient VAR Transformer strategies that can be used to extend their utility to more sophisticated VAR models in image generation and other related areas.
Abstract (translated)
我们研究了基于变压器的基础模型的基本限制，并将分析范围扩展到包括视觉自回归（VAR）变压器。VAR代表了一种重要进步，利用新颖的、可扩展的由粗到精的“下一尺度预测”框架来生成图像。这些模型设立了新的质量标准，在所有先前方法（包括扩散变压器）中脱颖而出，并在图像合成任务上表现出最先进的性能。 我们的主要贡献在于证明：对于具有单个自注意力层和单一插值层的单头VAR变压器，VAR变压器是通用的。从统计学的角度来看，我们证明了如此简单的VAR变压器可以作为任何图到图Lipschitz函数的通用逼近器。此外，我们还展示了基于流的自回归变压器继承了类似的近似能力。 我们的研究结果提供了重要的设计原则，用于制定有效的和计算效率高的VAR变压器策略，这些策略可扩展其在图像生成及其他相关领域更复杂VAR模型中的应用。
URL
https://arxiv.org/abs/2502.06167
PDF
-
Recommendations Beyond Catalogs: Diffusion Models for Personalized Generation
2025-02-05 21:11:47Gabriel Patron, Zhiwei Xu, Ishan Kapnadak, Felipe Maia PoloAbstract
Modern recommender systems follow the guiding principle of serving the right user, the right item at the right time. One of their main limitations is that they are typically limited to items already in the catalog. We propose REcommendations BEyond CAtalogs, REBECA, a new class of probabilistic diffusion-based recommender systems that synthesize new items tailored to individual tastes rather than retrieve items from the catalog. REBECA combines efficient training in embedding space with a novel diffusion prior that only requires users' past ratings of items. We evaluate REBECA on real-world data and propose novel personalization metrics for generative recommender systems. Extensive experiments demonstrate that REBECA produces high-quality, personalized recommendations, generating images that align with users' unique preferences.
Abstract (translated)
现代推荐系统遵循的原则是在正确的时间向正确的用户提供合适的物品。然而，它们的主要局限性在于通常只能从现有的商品目录中提供项目。我们提出了一个名为REBECA（REcommendations BEyond CAtalogs）的新类概率扩散型推荐系统，该系统可以合成新的个性化内容以满足用户的独特偏好，而不仅仅是从现有目录中检索物品。REBECA结合了嵌入空间中的高效训练与一种新颖的仅依赖于用户过去对商品评分的概率扩散先验方法。 我们在真实世界的数据上评估了REBECA，并提出了用于生成式推荐系统的新型个性化指标。广泛的实验表明，REBECA能够产生高质量且个性化的推荐结果，生成的图像符合用户的独特偏好。
URL
https://arxiv.org/abs/2502.18477
PDF
-
Can Text-to-Image Generative Models Accurately Depict Age? A Comparative Study on Synthetic Portrait Generation and Age Estimation
2025-02-05 18:08:33Alexey A. Novikov, Miroslav Vranka, Fran\c{c}ois David, Artem VoroninAbstract
Text-to-image generative models have shown remarkable progress in producing diverse and photorealistic outputs. In this paper, we present a comprehensive analysis of their effectiveness in creating synthetic portraits that accurately represent various demographic attributes, with a special focus on age, nationality, and gender. Our evaluation employs prompts specifying detailed profiles (e.g., Photorealistic selfie photo of a 32-year-old Canadian male), covering a broad spectrum of 212 nationalities, 30 distinct ages from 10 to 78, and balanced gender representation. We compare the generated images against ground truth age estimates from two established age estimation models to assess how faithfully age is depicted. Our findings reveal that although text-to-image models can consistently generate faces reflecting different identities, the accuracy with which they capture specific ages and do so across diverse demographic backgrounds remains highly variable. These results suggest that current synthetic data may be insufficiently reliable for high-stakes age-related tasks requiring robust precision, unless practitioners are prepared to invest in significant filtering and curation. Nevertheless, they may still be useful in less sensitive or exploratory applications, where absolute age precision is not critical.
Abstract (translated)
文本到图像的生成模型在产生多样性和照片般逼真的输出方面取得了显著进展。在这篇论文中，我们对这些模型在创建能够准确反映各种人口统计特征（特别是年龄、国籍和性别）的人工肖像方面的有效性进行了全面分析。我们的评估使用了指定详细个人资料的提示语（例如，“一个32岁的加拿大男性的现实自拍照”），涵盖了212个不同国家，从10岁到78岁的30种不同的年龄段，并且性别比例均衡。我们通过与两个已建立的年龄估计模型提供的真实年龄估计进行比较来评估生成图像中年龄描绘的真实程度。 我们的研究发现表明，尽管文本到图像的模型能够持续生成反映不同身份特征的脸部图像，但它们捕捉特定年龄以及在多样化的人口统计背景下的准确性仍然存在很大的变异性。这些结果暗示当前合成数据可能不足以用于需要高度精确度的关键性年龄相关任务，除非从业者愿意投入大量精力进行过滤和筛选工作。然而，在不敏感或探索性的应用中，即使绝对的年龄精度不是关键因素，它们仍可能具有一定的实用性。
URL
https://arxiv.org/abs/2502.03420
PDF
-
IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait
2025-01-28 18:59:03Han Yang, Enis Simsar, Sotiris Anagnostidi, Yanlong Zang, Thomas Hofmann, Ziwei LiuAbstract
Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.
Abstract (translated)
现有的扩散模型在保持身份特性的生成方面展现出了巨大的潜力。然而，由于用户资料的多样性（包括外观和光照条件的变化），个性化肖像生成仍然面临挑战。为了解决这些问题，我们提出了IC-Portrait，这是一个全新的框架，旨在准确编码个人身份以进行个性化的肖像生成。我们的关键见解是，预训练的扩散模型在上下文中密集对应匹配方面学习迅速（例如100到200步），这激励了我们IC-Portrait框架的主要设计。 具体而言，我们将肖像生成重新表述为两个子任务： 1. **光照感知拼接**：我们发现对输入图像进行高度遮挡处理（例如80%），可以非常有效地学习参考图的自监督表示，从而更好地理解光照条件。 2. **视图一致性适应**：利用合成的视图一致资料集来学习上下文中的对应关系。这使得参考轮廓能够被变形到任意姿势，从而提供强大的空间对齐视图调节。 通过简单地连接潜在变量形成类似ControlNet的监督和建模方式，将这两种设计结合起来，我们显著增强了身份保持的准确度和稳定性。广泛的评估显示，IC-Portrait在定量和定性评价中均优于现有的最先进的方法，并且在视觉质量方面有了特别明显的改进。此外，IC-Portrait还展示了3D感知重光照的能力。
URL
https://arxiv.org/abs/2501.17159
PDF
-
3D Reconstruction of non-visible surfaces of objects from a Single Depth View -- Comparative Study
2025-01-27 14:50:19Rafa{\l} Staszak, Piotr Micha{\l}ek, Jakub Chudzi\'nski, Marek Kopicki, Dominik BelterAbstract
Scene and object reconstruction is an important problem in robotics, in particular in planning collision-free trajectories or in object manipulation. This paper compares two strategies for the reconstruction of nonvisible parts of the object surface from a single RGB-D camera view. The first method, named DeepSDF predicts the Signed Distance Transform to the object surface for a given point in 3D space. The second method, named MirrorNet reconstructs the occluded objects' parts by generating images from the other side of the observed object. Experiments performed with objects from the ShapeNet dataset, show that the view-dependent MirrorNet is faster and has smaller reconstruction errors in most categories.
Abstract (translated)
场景和物体重建是机器人技术中的一个重要问题，尤其是在规划无碰撞轨迹或操作对象时。本文比较了从单个RGB-D相机视角重构未见部分的对象表面的两种策略。第一种方法名为DeepSDF，它预测给定3D空间中点到物体表面的带符号距离变换（Signed Distance Transform）。第二种方法名为MirrorNet，通过生成观察对象另一侧的图像来重建被遮挡的部分。使用ShapeNet数据集中的对象进行的实验表明，在大多数类别中，视图依赖性的MirrorNet比DeepSDF更快且重构误差更小。
URL
https://arxiv.org/abs/2501.16101
PDF
-
StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces
2025-01-26 08:22:44Kyeongmin Yeo, Jaihoon Kim, Minhyuk SungAbstract
We propose a zero-shot method for generating images in arbitrary spaces (e.g., a sphere for 360° panoramas and a mesh surface for texture) using a pretrained image diffusion model. The zero-shot generation of various visual content using a pretrained image diffusion model has been explored mainly in two directions. First, Diffusion Synchronization-performing reverse diffusion processes jointly across different projected spaces while synchronizing them in the target space-generates high-quality outputs when enough conditioning is provided, but it struggles in its absence. Second, Score Distillation Sampling-gradually updating the target space data through gradient descent-results in better coherence but often lacks detail. In this paper, we reveal for the first time the interconnection between these two methods while highlighting their differences. To this end, we propose StochSync, a novel approach that combines the strengths of both, enabling effective performance with weak conditioning. Our experiments demonstrate that StochSync provides the best performance in 360° panorama generation (where image conditioning is not given), outperforming previous finetuning-based methods, and also delivers comparable results in 3D mesh texturing (where depth conditioning is provided) with previous methods.
Abstract (translated)
我们提出了一种零样本方法，用于在任意空间（例如，球体用于360°全景图和网格表面用于纹理）生成图像，采用预训练的图像扩散模型。使用预训练的图像扩散模型进行各种视觉内容的零样本生成主要从两个方向进行了探索。首先，通过同时跨不同投影空间执行反向扩散过程并在目标空间中同步实现高质量输出，当有足够的条件信息时效果良好，但缺乏足够条件时则表现不佳。其次，通过梯度下降逐步更新目标空间数据的分数蒸馏采样方法产生了更好的连贯性，但往往缺少细节。在这篇论文中，我们首次揭示了这两种方法之间的相互联系，并突出了它们的不同之处。为此，我们提出了StochSync这一新方法，结合了两种方法的优点，在条件信息较弱的情况下也能实现有效性能。我们的实验表明，StochSync在360°全景图生成（此时未提供图像条件）中表现出最佳性能，优于以往基于微调的方法，并且在3D网格纹理化（其中提供了深度条件）方面与之前的方法相比结果相当。
URL
https://arxiv.org/abs/2501.15445
PDF
-
Personalized Preference Fine-tuning of Diffusion Models
2025-01-11 22:38:41Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming SongAbstract
RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.
Abstract (translated)
RLHF（基于人类反馈的强化学习）技术，如DPO（分布惩罚优化），可以显著提高文本到图像扩散模型的生成质量。然而，这些方法仅针对单一奖励进行优化，该奖励旨在将模型生成与群体偏好对齐，忽视了个体用户信念或价值观中的细微差别。这种个性化缺乏限制了这些模型的有效性。 为了解决这一问题，我们引入了一种名为PPD（多奖励优化目标）的方法，它使扩散模型能够根据个人喜好进行对齐。通过PPD，一个扩散模型可以从少量的成对偏好示例中学习到用户群体中的个体偏好，并且能够泛化到未见过的新用户身上。具体而言，我们的方法包括： 1. 利用视觉-语言模型（VLM）从一对小量的个人偏好示例中提取个性化偏好数值表示。 2. 通过交叉注意力机制将这些嵌入融入扩散模型。 在基于用户的嵌入进行条件设置后，文本到图像模型使用DPO目标进行了微调，同时优化了多个用户偏好的对齐。实证结果表明，我们的方法能够有效优化多个奖励函数，并且能够在推理过程中在这几个奖励之间进行插值。 在现实世界的应用场景中，通过仅从新用户提供4个偏好示例，我们的方法就能以76%的平均胜率超越稳定级联算法（Stable Cascade），生成更加准确反映特定用户偏好的图像。
URL
https://arxiv.org/abs/2501.06655
PDF
-
Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models
2025-01-10 10:26:54Sofia Jamil, Bollampalli Areen Reddy, Raghvendra Kumar, Sriparna Saha, K J Joseph, Koustava GoswamiAbstract
The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children's poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources.
Abstract (translated)
文本到图像生成任务在应用于文学作品，尤其是诗歌时遇到了重大挑战。诗歌是一种独特的文学形式，其含义往往超越了文字的字面意思。为了解决这一问题，我们提出了PoemToPixel框架，旨在生成能够视觉上表达诗歌内在意义的图像。我们的方法在图像生成框架中引入了提示调优的概念，以确保生成的图像与诗歌内容紧密相关。此外，我们还提出了一种名为PoeKey的算法，该算法从诗歌中提取三个关键元素——情感、视觉元素和主题，并将其形式化为指令，然后提供给扩散模型来生成相应的图像。 为了扩展涵盖不同流派和时代的诗歌数据集多样性，我们引入了MiniPo，这是一个新颖的多模态数据集，包含1001首儿童诗及其对应的图片。结合这个数据集与PoemSum，我们使用PoemToPixel框架进行了定量和定性的图像生成评估。 本文展示了我们的方法的有效性，并为从文学来源生成图像提供了新的视角。
URL
https://arxiv.org/abs/2501.05839
PDF
-
Text-Driven Tumor Synthesis
2024-12-24 18:43:09Xinran Li, Yi Shuai, Chen Liu, Qi Chen, Qilong Wu, Pengfei Guo, Dong Yang, Can Zhao, Pedro R. A. S. Bassi, Daguang Xu, Kang Wang, Yang Yang, Alan Yuille, Zongwei ZhouAbstract
Tumor synthesis can generate examples that AI often misses or over-detects, improving AI performance by training on these challenging cases. However, existing synthesis methods, which are typically unconditional -- generating images from random variables -- or conditioned only by tumor shapes, lack controllability over specific tumor characteristics such as texture, heterogeneity, boundaries, and pathology type. As a result, the generated tumors may be overly similar or duplicates of existing training data, failing to effectively address AI's weaknesses. We propose a new text-driven tumor synthesis approach, termed TextoMorph, that provides textual control over tumor characteristics. This is particularly beneficial for examples that confuse the AI the most, such as early tumor detection (increasing Sensitivity by +8.5%), tumor segmentation for precise radiotherapy (increasing DSC by +6.3%), and classification between benign and malignant tumors (improving Sensitivity by +8.2%). By incorporating text mined from radiology reports into the synthesis process, we increase the variability and controllability of the synthetic tumors to target AI's failure cases more precisely. Moreover, TextoMorph uses contrastive learning across different texts and CT scans, significantly reducing dependence on scarce image-report pairs (only 141 pairs used in this study) by leveraging a large corpus of 34,035 radiology reports. Finally, we have developed rigorous tests to evaluate synthetic tumors, including Text-Driven Visual Turing Test and Radiomics Pattern Analysis, showing that our synthetic tumors is realistic and diverse in texture, heterogeneity, boundaries, and pathology.
Abstract (translated)
肿瘤合成能够生成人工智能常会忽略或过度检测的例子，通过训练这些具有挑战性的案例可以提高AI的性能。然而，现有的合成方法通常是无条件的——从随机变量中生成图像——或者仅由肿瘤形状进行条件控制，缺乏对特定肿瘤特征（如纹理、异质性、边界和病理类型）的可控性。因此，生成的肿瘤可能过于相似或重复现有训练数据，无法有效解决AI的弱点。我们提出了一种新的基于文本驱动的肿瘤合成方法，称为TextoMorph，它可以在文本层面对肿瘤特性进行控制。这对于那些最容易使AI困惑的例子特别有益，比如早期肿瘤检测（提高灵敏度+8.5%）、用于精确放疗的肿瘤分割（增加DSC +6.3%）以及良性与恶性肿瘤分类（提高灵敏度+8.2%）。通过将从放射学报告中提取的文本融入合成过程，我们增加了合成肿瘤的变化性和可控性，以更精准地针对AI的失败案例。此外，TextoMorph利用不同的文本和CT扫描进行对比学习，显著减少了对稀缺图像-报告配对（本研究仅使用了141对）的依赖，并通过一个包含34,035份放射学报告的大语料库来实现这一点。最后，我们开发了一系列严格的测试来评估合成肿瘤，包括基于文本驱动的视觉图灵测试和放射组学模式分析，结果表明我们的合成肿瘤在纹理、异质性、边界和病理方面是现实且多样的。
URL
https://arxiv.org/abs/2412.18589
PDF
-
Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models
2024-12-24 13:27:25Qice Qin, Yuki Hirakawa, Ryotaro Shimizu, Takuya Furusawa, Edgar Simo-SerraAbstract
Image generation in the fashion domain has predominantly focused on preserving body characteristics or following input prompts, but little attention has been paid to improving the inherent fashionability of the output images. This paper presents a novel diffusion model-based approach that generates fashion images with improved fashionability while maintaining control over key attributes. Key components of our method include: 1) fashionability enhancement, which ensures that the generated images are more fashionable than the input; 2) preservation of body characteristics, encouraging the generated images to maintain the original shape and proportions of the input; and 3) automatic fashion optimization, which does not rely on manual input or external prompts. We also employ two methods to collect training data for guidance while generating and evaluating the images. In particular, we rate outfit images using fashionability scores annotated by multiple fashion experts through OpenSkill-based and five critical aspect-based pairwise comparisons. These methods provide complementary perspectives for assessing and improving the fashionability of the generated images. The experimental results show that our approach outperforms the baseline Fashion++ in generating images with superior fashionability, demonstrating its effectiveness in producing more stylish and appealing fashion images.
Abstract (translated)
时尚领域的图像生成主要集中在保持身体特征或遵循输入提示上，但很少有关注提升输出图像内在的时尚性的。本文提出了一种基于扩散模型的新方法，该方法能够生成具有改进时尚性的时装图像，同时还能控制关键属性。我们方法的关键组成部分包括：1) 时尚性增强，确保生成的图像比输入更时尚；2) 保持身体特征，鼓励生成的图像维持输入原有的形状和比例；3) 自动时尚优化，不依赖于手动输入或外部提示。此外，我们也采用了两种方法来收集训练数据，在生成和评估图像时提供指导。特别是，我们通过基于OpenSkill的方法以及基于五个关键方面的成对比较，使用多位时装专家标注的时尚性评分来评定服装图像。这些方法为评估和提升生成图像的时尚性提供了互补的角度。实验结果表明，我们的方法在生成具有更优时尚性的图像方面优于基准Fashion++，证明了它在生产更具风格且吸引人的时装图像方面的有效性。
URL
https://arxiv.org/abs/2412.18421
PDF
