Abstract
This paper documents our characterization study and practices for serving text-to-image requests with stable diffusion models in production. We first comprehensively analyze inference request traces for commercial text-to-image applications. It commences with our observation that add-on modules, i.e., ControlNets and LoRAs, that augment the base stable diffusion models, are ubiquitous in generating images for commercial applications. Despite their efficacy, these add-on modules incur high loading overhead, prolong the serving latency, and swallow up expensive GPU resources. Driven by our characterization study, we present SwiftDiffusion, a system that efficiently generates high-quality images using stable diffusion models and add-on modules. To achieve this, SwiftDiffusion reconstructs the existing text-to-image serving workflow by identifying the opportunities for parallel computation and distributing ControlNet computations across multiple GPUs. Further, SwiftDiffusion thoroughly analyzes the dynamics of image generation and develops techniques to eliminate the overhead associated with LoRA loading and patching while preserving the image quality. Last, SwiftDiffusion proposes specialized optimizations in the backbone architecture of the stable diffusion models, which are also compatible with the efficient serving of add-on modules. Compared to state-of-the-art text-to-image serving systems, SwiftDiffusion reduces serving latency by up to 5x and improves serving throughput by up to 2x without compromising image quality.
Abstract (translated)
本文记录了我们在生产环境中使用稳定扩散模型处理文本到图像请求的分类和实现。我们首先对商业文本到图像应用的推理请求跟踪进行全面分析。我们的观察是，增强基稳定扩散模型的附加模块（即控制网络和LoRAs）无处不在，都在生成图像。尽管它们非常有效，但这些附加模块却带来了高昂的加载开销，延长了服务延迟，并吞噬了昂贵的GPU资源。为了实现我们的分类研究，我们提出了SwiftDiffusion系统，这是一种高效使用稳定扩散模型和附加模块生成高质量图像的系统。为了实现这一目标，SwiftDiffusion通过识别并分配计算机会并行执行控制网络计算，对现有的文本到图像服务流程进行重构。此外，SwiftDiffusion还深入分析了图像生成的动态，并开发了消除LoRA加载和补丁开销，同时保留图像质量的技术。最后，SwiftDiffusion在稳定扩散模型的基础架构上提出了专用的优化措施，这些措施与附加模块的高效服务是兼容的。与最先进的文本到图像服务系统相比，SwiftDiffusion将服务延迟减少到原来的5倍，同时将服务吞吐量提高到原来的2倍，而不会牺牲图像质量。
URL
https://arxiv.org/abs/2407.02031
