Abstract
Edge computing's growing prominence, due to its ability to reduce communication latency and enable real-time processing, is promoting the rise of high-performance, heterogeneous System-on-Chip solutions. While current approaches often involve scaling down modern hardware, the performance characteristics of neural network workloads on these platforms can vary significantly, especially when it comes to parallel processing, which is a critical consideration for edge deployments. To address this, we conduct a comprehensive study comparing the latency and throughput of various linear algebra and neural network inference tasks across CPU-only, CPU/GPU, and CPU/NPU integrated solutions. {We find that the Neural Processing Unit (NPU) excels in matrix-vector multiplication (58.6% faster) and some neural network tasks (3.2$\times$ faster for video classification and large language models). GPU outperforms in matrix multiplication (22.6% faster) and LSTM networks (2.7$\times$ faster) while CPU excels at less parallel operations like dot product. NPU-based inference offers a balance of latency and throughput at lower power consumption. GPU-based inference, though more energy-intensive, performs best with large dimensions and batch sizes. We highlight the potential of heterogeneous computing solutions for edge AI, where diverse compute units can be strategically leveraged to boost accurate and real-time inference.
Abstract (translated)
由于其能够降低通信延迟并实现实时处理，使边缘计算的重要性不断增加，推动了高性能、异构 System-on-Chip 解决方案的发展。然而，通常的方法往往会缩小现代硬件的规模，这些平台上的神经网络负载的性能特征差异很大，特别是在并行处理方面，这是边缘部署的关键考虑因素。为了应对这个问题，我们进行了全面的研究，比较了各种线性代数和神经网络推理任务在CPU-only、CPU/GPU和CPU/NPU集成解决方案中的延迟和吞吐量。我们发现，神经处理单元（NPU）在矩阵向量乘法（58.6% faster）和某些神经网络任务（视频分类和大语言模型的速度快3.2倍）方面表现优异。GPU 在矩阵乘法（22.6% faster）和LSTM网络（2.7倍 faster）方面优于CPU，而在并行操作方面，比如点积，CPU的性能较低。基于NPU的推理在较低的功耗下实现了平衡的延迟和吞吐量。尽管GPU 的性能更耗费能量，但在大维度和高批量的场景下表现最佳。我们强调了异构计算解决方案在边缘 AI 中的潜力，通过 strategic leveraging diverse compute units，可以提高准确和实时的推理。
URL
https://arxiv.org/abs/2409.14803
