Abstract
Cooking process visualization is a promising task in the intersection of image generation and food analysis, which aims to generate an image for each cooking step of a recipe. However, most existing works focus on generating images of finished foods based on the given recipes, and face two challenges to visualize the cooking process. First, the appearance of ingredients changes variously across cooking steps, it is difficult to generate the correct appearances of foods that match the textual description, leading to semantic inconsistency. Second, the current step might depend on the operations of previous step, it is crucial to maintain the contextual coherence of images in sequential order. In this work, we present a cooking process visualization model, called Chain-of-Cooking. Specifically, to generate correct appearances of ingredients, we present a Dynamic Patch Selection Module to retrieve previously generated image patches as references, which are most related to current textual contents. Furthermore, to enhance the coherence and keep the rational order of generated images, we propose a Semantic Evolution Module and a Bidirectional Chain-of-Thought (CoT) Guidance. To better utilize the semantics of previous texts, the Semantic Evolution Module establishes the semantical association between latent prompts and current cooking step, and merges it with the latent features. Then the CoT Guidance updates the merged features to guide the current cooking step remain coherent with the previous step. Moreover, we construct a dataset named CookViz, consisting of intermediate image-text pairs for the cooking process. Quantitative and qualitative experiments show that our method outperforms existing methods in generating coherent and semantic consistent cooking process.
Abstract (translated)
烹饪过程可视化是在图像生成和食品分析交叉领域的一项有前景的任务，旨在为食谱中的每一步骤生成一张图像。然而，大多数现有的工作主要集中在根据给定的食谱生成完成食物的图片上，并且在可视化烹饪过程时面临着两个挑战：首先，食材外观会随着步骤的变化而发生变化，因此很难根据文本描述生成匹配的食物外观，从而导致语义不一致；其次，当前步骤可能依赖于前一操作的结果，所以保持图像序列中的上下文连贯性至关重要。为此，在这项工作中我们提出了一种名为“Cooking Chain”的烹饪过程可视化模型。 具体而言，为了生成正确的食材外观，我们设计了一个动态补丁选择模块来检索先前生成的与当前文本内容最相关的图像片段作为参考。此外，为增强图像的一致性和保持合理顺序，我们提出了语义演变模块和双向思考链（CoT）引导策略。为了更好地利用之前文本的语义信息，语义演变模块建立了隐式提示和当前烹饪步骤之间的语义关联，并将其与潜在特征进行融合。随后，CoT 引导更新这些合并的特征以指导当前步骤保持连贯性，使其与前一步骤一致。 此外，我们构建了一个名为CookViz的数据集，该数据集中包含了用于描述整个烹饪过程中的中间图像-文本对。定量和定性的实验表明，在生成连贯且语义一致的烹饪过程中，我们的方法优于现有的其他方法。
URL
https://arxiv.org/abs/2507.21529
