Abstract
Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation personalized knowledge-driven generation. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and knowledge-driven generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized knowledge-driven generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{this https URL}{this https URL}.
Abstract (translated)
个性化模型在理解并生成用户提供的概念方面取得了显著的成功。然而，现有的方法使用单独的概念令牌来处理理解和生成任务，并将这两个任务视为孤立的。这可能会限制生成具有复杂提示图像的能力。例如，在给定概念$\langle bo\rangle$的情况下，无法仅通过不提供帽子的文字描述就生成“$\langle bo\rangle$戴着它的帽子”。我们将这种类型的生成称为个性化知识驱动生成。 为了解决这一局限性，我们提出了UniCTokens，这是一种新型框架，可以将个性化的信息有效地整合到一个统一的视觉语言模型（VLM）中，以支持理解和生成任务。UniCTokens训练一组统一的概念令牌来利用互补语义，从而增强两个个性化任务的表现。此外，我们还提出了一种分阶段训练策略，包含三个步骤：理解预热、从理解启动生成、以及通过生成加深理解，以此来增强两项任务之间的相互利益。 为了定量评估统一的VLM个性化，我们推出了UnifyBench，这是首个用于评估概念理解、概念生成和知识驱动生成效果的标准基准。在UnifyBench上的实验结果表明，UniCTokens在概念理解和概念生成方面与现有领先方法的表现相当，在个性化知识驱动生成方面的表现更是达到了最先进的水平。 我们的研究证明了增强的理解可以提升生成能力，并且生成过程能够提供对理解的宝贵见解。我们的代码和数据集将在[此链接](this https URL)发布。
URL
https://arxiv.org/abs/2505.14671
