Abstract
Scene understanding is essential for enhancing driver safety, generating human-centric explanations for Automated Vehicle (AV) decisions, and leveraging Artificial Intelligence (AI) for retrospective driving video analysis. This study developed a dynamic scene retrieval system using Contrastive Language-Image Pretraining (CLIP) models, which can be optimized for real-time deployment on edge devices. The proposed system outperforms state-of-the-art in-context learning methods, including the zero-shot capabilities of GPT-4o, particularly in complex scenarios. By conducting frame-level analysis on the Honda Scenes Dataset, which contains a collection of about 80 hours of annotated driving videos capturing diverse real-world road and weather conditions, our study highlights the robustness of CLIP models in learning visual concepts from natural language supervision. Results also showed that fine-tuning the CLIP models, such as ViT-L/14 and ViT-B/32, significantly improved scene classification, achieving a top F1 score of 91.1%. These results demonstrate the ability of the system to deliver rapid and precise scene recognition, which can be used to meet the critical requirements of Advanced Driver Assistance Systems (ADAS). This study shows the potential of CLIP models to provide scalable and efficient frameworks for dynamic scene understanding and classification. Furthermore, this work lays the groundwork for advanced autonomous vehicle technologies by fostering a deeper understanding of driver behavior, road conditions, and safety-critical scenarios, marking a significant step toward smarter, safer, and more context-aware autonomous driving systems.
Abstract (translated)
场景理解对于提升驾驶员安全、生成针对自动驾驶汽车（AV）决策的人类可解释性，以及利用人工智能进行驾驶视频回顾分析至关重要。本研究开发了一种基于对比语言-图像预训练（CLIP）模型的动态场景检索系统，该系统可以优化为在边缘设备上的实时部署。所提出的系统在复杂场景中比最先进的情境学习方法表现更优，包括GPT-4o的零样本能力。通过对本田场景数据集进行帧级分析——这是一个包含大约80小时注释驾驶视频的数据集，涵盖了各种真实世界的道路和天气条件——我们的研究强调了CLIP模型从自然语言监督中学习视觉概念的能力之强健性。 实验结果还表明，对ViT-L/14和ViT-B/32等CLIP模型进行微调能够显著提高场景分类的准确性，达到了最高的F1分数为91.1%。这些结果展示了该系统能够提供快速且精确的场景识别能力，可用于满足高级驾驶辅助系统（ADAS）的关键需求。 这项研究展示了CLIP模型在动态场景理解和分类中具有提供可扩展和高效框架的巨大潜力。此外，本工作通过促进对驾驶员行为、道路条件及安全关键情景的理解，为先进自动驾驶技术的发展奠定了基础，标志着向更智能、更安全、更具上下文感知的自主驾驶系统迈进的重要一步。
URL
https://arxiv.org/abs/2501.05566
