Abstract
Generating images from rhetorical languages remains a critical challenge for text-to-image models. Even state-of-the-art (SOTA) multimodal large language models (MLLM) fail to generate images based on the hidden meaning inherent in rhetorical language--despite such content being readily mappable to visual representations by humans. A key limitation is that current models emphasize object-level word embedding alignment, causing metaphorical expressions to steer image generation towards their literal visuals and overlook the intended semantic meaning. To address this, we propose Rhet2Pix, a framework that formulates rhetorical text-to-image generation as a multi-step policy optimization problem, incorporating a two-layer MDP diffusion module. In the outer layer, Rhet2Pix converts the input prompt into incrementally elaborated sub-sentences and executes corresponding image-generation actions, constructing semantically richer visuals. In the inner layer, Rhet2Pix mitigates reward sparsity during image generation by discounting the final reward and optimizing every adjacent action pair along the diffusion denoising trajectory. Extensive experiments demonstrate the effectiveness of Rhet2Pix in rhetorical text-to-image generation. Our model outperforms SOTA MLLMs such as GPT-4o, Grok-3 and leading academic baselines across both qualitative and quantitative evaluations. The code and dataset used in this work are publicly available.
Abstract (translated)
从修辞语言生成图像仍然是文本到图像模型面临的关键挑战。即使是最先进的多模态大型语言模型（MLLM）也无法根据修辞语言中固有的含义生成图像——尽管这些内容可以很容易地由人类映射为视觉表示形式。目前的模型主要侧重于对象级别的词嵌入对齐，这会导致比喻表达被导向其字面意义的视觉效果，并忽略所意图的意义。为了应对这一挑战，我们提出了Rhet2Pix框架，该框架将修辞文本到图像生成问题定义为一个多步骤策略优化问题，并引入了一个两层MDP扩散模块。 在外层，Rhet2Pix将输入提示转换为逐步详细化的子句并执行相应的图像生成操作，以此构建语义更丰富的视觉效果。在内层，Rhet2Pix通过减少最终奖励并沿扩散去噪轨迹优化每一个相邻的动作对来缓解图像生成过程中的奖励稀疏性问题。 大量的实验展示了Rhet2Pix在修辞文本到图像生成方面的有效性。我们的模型在定性和定量评估中均超过了最先进的MLLM（如GPT-4o，GroK-3）和学术基准模型。本研究使用的代码和数据集已公开提供。
URL
https://arxiv.org/abs/2505.22792
