Abstract
Understanding how visual content communicates sentiment is critical in an era where online interaction is increasingly dominated by this kind of media on social platforms. However, this remains a challenging problem, as sentiment perception is closely tied to complex, scene-level semantics. In this paper, we propose an original framework, MLLMsent, to investigate the sentiment reasoning capabilities of Multimodal Large Language Models (MLLMs) through three perspectives: (1) using those MLLMs for direct sentiment classification from images; (2) associating them with pre-trained LLMs for sentiment analysis on automatically generated image descriptions; and (3) fine-tuning the LLMs on sentiment-labeled image descriptions. Experiments on a recent and established benchmark demonstrate that our proposal, particularly the fine-tuned approach, achieves state-of-the-art results outperforming Lexicon-, CNN-, and Transformer-based baselines by up to 30.9%, 64.8%, and 42.4%, respectively, across different levels of evaluators' agreement and sentiment polarity categories. Remarkably, in a cross-dataset test, without any training on these new data, our model still outperforms, by up to 8.26%, the best runner-up, which has been trained directly on them. These results highlight the potential of the proposed visual reasoning scheme for advancing affective computing, while also establishing new benchmarks for future research.
Abstract (translated)
理解视觉内容如何传达情感在社交媒体平台上这种媒体形式主导在线互动的时代至关重要。然而，这仍然是一个挑战性问题，因为情感感知与复杂的场景级语义紧密相关。为此，本文提出了一种原创框架MLLMsent，用于通过三种视角来研究多模态大型语言模型（MLLM）的情感推理能力：(1)直接使用这些MLLM从图像进行情感分类；(2)将它们与预训练的语言模型关联起来，在自动产生的图像描述上进行情感分析；以及 (3)在带有情感标签的图像描述上对语言模型进行微调。实验表明，基于最近且已被确立的基准测试，我们的提案——尤其是微调方法——实现了最先进的结果，分别比词典、CNN和Transformer基线方法提高了高达30.9%、64.8% 和 42.4%，在不同级别的评估者共识和情感极性类别中均表现出色。特别值得注意的是，在跨数据集测试中，即使没有在这类新数据上进行任何训练，我们的模型仍然优于最佳竞争对手多达8.26%，后者直接在这些数据上进行了训练。这些结果突显了所提出的视觉推理方案在推进情绪计算方面的潜力，并为未来研究建立了新的基准。
URL
https://arxiv.org/abs/2508.16873
