Abstract
The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wild (VSPW) test set, respectively. This paper details our research work that achieved the 1st place winner in the PVUW'24 VPS challenge, establishing state of art results in all metrics, including the Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). With minor fine-tuning our approach also achieved the 3rd place in the PVUW'24 VSS challenge ranked by the mIoU (mean intersection over union) metric and the first place ranked by the VC16 (16-frame video consistency) metric. Our winning solution stands on the shoulders of giant foundational vision transformer model (DINOv2 ViT-g) and proven multi-stage Decoupled Video Instance Segmentation (DVIS) frameworks for video understanding.
Abstract (translated)
第三个在野外像素级别的视频理解挑战（PVUW CVPR 2024）旨在通过在大型野外视频全景分割（VIPSeg）测试集和大型野外视频场景解析（VSPW）测试集中对具有挑战性的视频和场景进行基准测试，分别推动视频理解技术的进步。本文详细介绍了我们在PVUW'24 VPS挑战中取得第一名的 research 工作，建立了包括 Video Panoptic Quality（VPQ）和 Segmentation and Tracking Quality（STQ）在内的所有指标的最佳结果。通过微调我们的方法，我们的解决方案还获得了 PVUW'24 VSS 挑战中的第三名，根据mIoU（mean intersection over union）指标排名第三，根据 VC16（16-帧视频一致性）指标排名第一。我们的获胜解决方案站在了大型基础视觉变换模型（DINOv2 ViT-g）和经过验证的多阶段解耦视频实例分割（DVIS）框架的肩膀上，为视频理解技术的发展做出了巨大贡献。
URL
https://arxiv.org/abs/2406.05352
