Abstract
Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.
Abstract (translated)
视觉大型语言模型（VLLMs）正在改变计算机视觉和自然语言处理之间的交叉。然而，在這些模型中使用視覺提示進行情感識別的潛力仍然被大大開發和利用。傳統的VLLM方法在空間定位方面存在困難，通常會舍棄有價值的全身上下文。為了解決這個問題，我們提出了一个Set-of-Vision prompting（SoV）方法，通過使用空間信息（如邊界框和面部 landmarks）來精確標記目標，來增強 zero-shot情感識別的準確度。SoV在face count和 emotion categorization方面改善了準確度，同時保留了豐富的情感上下文。通過對最新商業或開源VLLM進行了大量實驗和分析，我們評估了SoV模型在自然環境中理解面部表情的能力。我們的研究結果表明，將空間視覺提示集成到VLLM中可以提高情感識別性能。
URL
https://arxiv.org/abs/2410.02244
