Abstract
Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.
Abstract (translated)
尖峰神经网络（SNNs）已成为传统人工神经网络（ANNs）在静态图像任务（如图像分类和分割）上的一个引人注目的、能源高效的替代方案。然而，在更复杂的视频分类领域，基于SNN的方法与基于ANN的方法相比，基准测试的结果差距很大，因为处理密集帧序列的挑战。为了弥合这一差距，我们提出了ReSpike，一种结合ANNs和SNNs优势的混合框架，以解决高准确性和低能量成本的动作识别任务。通过将电影剪辑分解为空间和时间组件，即RGB图像关键帧和事件状的残差帧，ReSpike利用ANNs学习空间信息，SNNs学习时间信息。此外，我们还提出了多尺度 cross-attention 机制来实现有效的特征融合。与最先进的SNN基线相比，我们的ReSpike混合架构显示出显著的性能改进（例如，HMDB-51、UCF-101和Kinetics-400的绝对准确度改进率均大于30%）。此外，ReSpike与先前的ANN方法实现了相当的表演，同时带来了更好的准确-能量权衡。
URL
https://arxiv.org/abs/2409.01564
