Abstract
Vision Large Language Models (VLLMs) exhibit promising potential for multi-modal understanding, yet their application to video-based emotion recognition remains limited by insufficient spatial and contextual awareness. Traditional approaches, which prioritize isolated facial features, often neglect critical non-verbal cues such as body language, environmental context, and social interactions, leading to reduced robustness in real-world scenarios. To address this gap, we propose Set-of-Vision-Text Prompting (SoVTP), a novel framework that enhances zero-shot emotion recognition by integrating spatial annotations (e.g., bounding boxes, facial landmarks), physiological signals (facial action units), and contextual cues (body posture, scene dynamics, others' emotions) into a unified prompting strategy. SoVTP preserves holistic scene information while enabling fine-grained analysis of facial muscle movements and interpersonal dynamics. Extensive experiments show that SoVTP achieves substantial improvements over existing visual prompting methods, demonstrating its effectiveness in enhancing VLLMs' video emotion recognition capabilities.
Abstract (translated)
视觉大型语言模型（VLLM）在多模态理解方面展现出巨大潜力，但它们在基于视频的情绪识别中的应用仍然受限于空间和上下文感知不足。传统方法侧重于孤立的面部特征，往往忽视了诸如肢体语言、环境背景和社会互动等关键非言语线索，导致其实用性在现实场景中大打折扣。为解决这一问题，我们提出了集合视觉-文本提示框架（SoVTP），这是一种新型框架，通过整合空间标注（如边界框、面部地标）、生理信号（面部动作单位）和上下文线索（身体姿态、场景动态以及他人的情绪），采用统一的提示策略来增强零样本情绪识别能力。SoVTP能够保留整体场景信息，并支持对面部肌肉运动和人际互动进行精细分析。广泛的实验表明，相较于现有的视觉提示方法，SoVTP实现了显著提升，在增强VLLM视频情感识别功能方面展现了有效性。
URL
https://arxiv.org/abs/2504.17224
