Abstract
In Text-to-Image (T2I) generation, the complexity of entities and their intricate interactions pose a significant challenge for T2I method based on diffusion model: how to effectively control entity and their interactions to produce high-quality images. To address this, we propose CEIDM, a image generation method based on diffusion model with dual controls for entity and interaction. First, we propose an entity interactive relationships mining approach based on Large Language Models (LLMs), extracting reasonable and rich implicit interactive relationships through chain of thought to guide diffusion models to generate high-quality images that are closer to realistic logic and have more reasonable interactive relationships. Furthermore, We propose an interactive action clustering and offset method to cluster and offset the interactive action features contained in each text prompts. By constructing global and local bidirectional offsets, we enhance semantic understanding and detail supplementation of original actions, making the model's understanding of the concept of interactive "actions" more accurate and generating images with more accurate interactive actions. Finally, we design an entity control network which generates masks with entity semantic guidance, then leveraging multi-scale convolutional network to enhance entity feature and dynamic network to fuse feature. It effectively controls entities and significantly improves image quality. Experiments show that the proposed CEIDM method is better than the most representative existing methods in both entity control and their interaction control.
Abstract (translated)
在文本到图像（T2I）生成领域，实体的复杂性和它们之间错综复杂的相互作用对基于扩散模型的方法构成了重大挑战：如何有效控制这些实体及其相互作用以产生高质量的图像。为了解决这一问题，我们提出了CEIDM方法，这是一种基于扩散模型并具备双控机制（用于实体和交互）的图像生成方法。 首先，我们提出了一种基于大型语言模型（LLMs）的实体互动关系挖掘方法，通过链式思考提取合理的、丰富的隐性互动关系，以指导扩散模型生成更接近现实逻辑且具有更多合理互动关系的高质量图像。 其次，为了进一步优化这一过程，我们还提出了一个互动动作聚类和偏移的方法。该方法可以对每个文本提示中包含的互动动作特征进行聚类和偏移处理。通过构建全局和局部双向偏置，这种方法增强了原始行动的语义理解和细节补充能力，使得模型对于“互动行为”的概念理解更加准确，并能生成具有更精确互动行为的图像。 最后，我们设计了一个实体控制网络，该网络在实体语义引导下产生掩码，然后通过多尺度卷积网络增强实体特征并利用动态网络融合这些特征。这一机制有效控制了实体，并显著提升了图像质量。 实验表明，所提出的CEIDM方法不仅在实体控制方面优于现有的最先进方法，在互动控制方面的表现同样出色。
URL
https://arxiv.org/abs/2508.17760
