Abstract
The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children's poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources.
Abstract (translated)
文本到图像生成任务在应用于文学作品，尤其是诗歌时遇到了重大挑战。诗歌是一种独特的文学形式，其含义往往超越了文字的字面意思。为了解决这一问题，我们提出了PoemToPixel框架，旨在生成能够视觉上表达诗歌内在意义的图像。我们的方法在图像生成框架中引入了提示调优的概念，以确保生成的图像与诗歌内容紧密相关。此外，我们还提出了一种名为PoeKey的算法，该算法从诗歌中提取三个关键元素——情感、视觉元素和主题，并将其形式化为指令，然后提供给扩散模型来生成相应的图像。 为了扩展涵盖不同流派和时代的诗歌数据集多样性，我们引入了MiniPo，这是一个新颖的多模态数据集，包含1001首儿童诗及其对应的图片。结合这个数据集与PoemSum，我们使用PoemToPixel框架进行了定量和定性的图像生成评估。 本文展示了我们的方法的有效性，并为从文学来源生成图像提供了新的视角。
URL
https://arxiv.org/abs/2501.05839
