Abstract
Diffusion models have achieved state-of-the-art performance in generating images, audio, and video, but their adaptation to text remains challenging due to its discrete nature. Prior approaches either apply Gaussian diffusion in continuous latent spaces, which inherits semantic structure but struggles with token decoding, or operate in categorical simplex space, which respect discreteness but disregard semantic relation between tokens. In this paper, we propose Smoothing Diffusion on Token Embeddings (Smoothie), a novel diffusion method that combines the strengths of both approaches by progressively smoothing token embeddings based on semantic similarity. This technique enables gradual information removal while maintaining a natural decoding process. Experimental results on several sequence-to-sequence generation tasks demonstrate that Smoothie outperforms existing diffusion-based models in generation quality. Furthermore, ablation studies show that our proposed diffusion space yields better performance than both the standard embedding space and the categorical simplex. Our code is available at this https URL.
Abstract (translated)
扩散模型在生成图像、音频和视频方面取得了最先进的性能，但在文本领域的应用仍面临挑战，主要是因为文本的离散特性。先前的方法要么在连续潜在空间中应用高斯扩散，这种方法虽然继承了语义结构但难以解决词元解码的问题；要么在分类单纯形空间中操作，这虽尊重了离散性但却忽略了词元之间的语义关系。在这篇论文中，我们提出了 Smoothie（基于 Token Embeddings 的平滑扩散），这是一种结合了两种方法优势的新型扩散方法，通过逐步根据语义相似度平滑词元嵌入来实现信息渐进式移除的同时保持自然解码过程。在几个序列到序列生成任务上的实验结果表明，Smoothie 在生成质量上超越了现有的基于扩散的方法。此外，消融研究表明我们提出的扩散空间比标准嵌入空间和分类单纯形的表现都要好。我们的代码可在该链接获取：[此URL]（请将 [此URL] 替换为实际的代码链接）。
URL
https://arxiv.org/abs/2505.18853
