Abstract
Subject-driven image generation models face a fundamental trade-off between identity preservation (fidelity) and prompt adherence (editability). While online reinforcement learning (RL), specifically GPRO, offers a promising solution, we find that a naive application of GRPO leads to competitive degradation, as the simple linear aggregation of rewards with static weights causes conflicting gradient signals and a misalignment with the temporal dynamics of the diffusion process. To overcome these limitations, we propose Customized-GRPO, a novel framework featuring two key innovations: (i) Synergy-Aware Reward Shaping (SARS), a non-linear mechanism that explicitly penalizes conflicted reward signals and amplifies synergistic ones, providing a sharper and more decisive gradient. (ii) Time-Aware Dynamic Weighting (TDW), which aligns the optimization pressure with the model's temporal dynamics by prioritizing prompt-following in the early, identity preservation in the later. Extensive experiments demonstrate that our method significantly outperforms naive GRPO baselines, successfully mitigating competitive degradation. Our model achieves a superior balance, generating images that both preserve key identity features and accurately adhere to complex textual prompts.
Abstract (translated)
基于主题的图像生成模型在身份保真（fidelity）和指令遵循（editability）之间存在根本性的权衡。虽然在线强化学习（RL），特别是GPRO，提供了一个有前景的解决方案，但我们发现直接应用GRPO会导致竞争性退化，因为简单的线性奖励聚合使用静态权重导致了冲突的梯度信号，并且与扩散过程的时间动态不一致。为了解决这些问题，我们提出了Customized-GRPO，这是一个包含两个关键创新的新框架： (i) 协同感知奖励塑形（Synergy-Aware Reward Shaping, SARS）：这是一种非线性机制，它明确地惩罚冲突的奖励信号并放大协同作用的信号，提供更尖锐和更具决定性的梯度。 (ii) 时间感知动态加权（Time-Aware Dynamic Weighting, TDW）：通过优先考虑早期阶段遵循指令、晚期阶段保持身份特征，该方法使优化压力与模型的时间动态相一致。 广泛的实验表明，我们的方法显著优于简单的GRPO基线，成功地缓解了竞争性退化。我们的模型实现了更佳的平衡，生成出既能保留关键的身份特征又能准确遵循复杂文本提示的图像。
URL
https://arxiv.org/abs/2510.18263
