Abstract
RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.
Abstract (translated)
RLHF（基于人类反馈的强化学习）技术，如DPO（分布惩罚优化），可以显著提高文本到图像扩散模型的生成质量。然而，这些方法仅针对单一奖励进行优化，该奖励旨在将模型生成与群体偏好对齐，忽视了个体用户信念或价值观中的细微差别。这种个性化缺乏限制了这些模型的有效性。 为了解决这一问题，我们引入了一种名为PPD（多奖励优化目标）的方法，它使扩散模型能够根据个人喜好进行对齐。通过PPD，一个扩散模型可以从少量的成对偏好示例中学习到用户群体中的个体偏好，并且能够泛化到未见过的新用户身上。具体而言，我们的方法包括： 1. 利用视觉-语言模型（VLM）从一对小量的个人偏好示例中提取个性化偏好数值表示。 2. 通过交叉注意力机制将这些嵌入融入扩散模型。 在基于用户的嵌入进行条件设置后，文本到图像模型使用DPO目标进行了微调，同时优化了多个用户偏好的对齐。实证结果表明，我们的方法能够有效优化多个奖励函数，并且能够在推理过程中在这几个奖励之间进行插值。 在现实世界的应用场景中，通过仅从新用户提供4个偏好示例，我们的方法就能以76%的平均胜率超越稳定级联算法（Stable Cascade），生成更加准确反映特定用户偏好的图像。
URL
https://arxiv.org/abs/2501.06655
