Abstract
This paper introduces ViNet-S, a 36MB model based on the ViNet architecture with a U-Net design, featuring a lightweight decoder that significantly reduces model size and parameters without compromising performance. Additionally, ViNet-A (148MB) incorporates spatio-temporal action localization (STAL) features, differing from traditional video saliency models that use action classification backbones. Our studies show that an ensemble of ViNet-S and ViNet-A, by averaging predicted saliency maps, achieves state-of-the-art performance on three visual-only and six audio-visual saliency datasets, outperforming transformer-based models in both parameter efficiency and real-time performance, with ViNet-S reaching over 1000fps.
Abstract (translated)
本文介绍了ViNet-S，这是一种基于ViNet架构并采用U-Net设计的36MB模型，其轻量级解码器在不牺牲性能的前提下大幅减少了模型大小和参数。此外，ViNet-A（148MB）融合了时空动作定位（STAL）特征，与传统的视频显著性模型不同，后者依赖于动作分类骨干网络。我们的研究表明，在三个仅视觉数据集以及六个视听显著性数据集中，通过平均预测的显著性图，由ViNet-S和ViNet-A组成的集成模型达到了最先进的性能水平，并且在参数效率和实时性能方面均优于基于变换器的模型，其中ViNet-S可以达到每秒超过1000帧的速度。
URL
https://arxiv.org/abs/2502.00397
