Abstract
Sentiment classification in short text datasets faces significant challenges such as class imbalance, limited training samples, and the inherent subjectivity of sentiment labels -- issues that are further intensified by the limited context in short texts. These factors make it difficult to resolve ambiguity and exacerbate data sparsity, hindering effective learning. In this paper, we evaluate the effectiveness of small Transformer-based models (i.e., BERT and RoBERTa, with fewer than 1 billion parameters) for multi-label sentiment classification, with a particular focus on short-text settings. Specifically, we evaluated three key factors influencing model performance: (1) continued domain-specific pre-training, (2) data augmentation using automatically generated examples, specifically generative data augmentation, and (3) architectural variations of the classification head. Our experiment results show that data augmentation improves classification performance, while continued pre-training on augmented datasets can introduce noise rather than boost accuracy. Furthermore, we confirm that modifications to the classification head yield only marginal benefits. These findings provide practical guidance for optimizing BERT-based models in resource-constrained settings and refining strategies for sentiment classification in short-text datasets.
Abstract (translated)
在短文本数据集上进行情感分类面临诸如类别不平衡、训练样本有限以及情感标签主观性强等重大挑战，这些问题因短文中上下文信息的局限性而变得更加复杂。这些因素使得解决模棱两可的问题变得困难，并加剧了数据稀疏性，从而阻碍了有效的学习过程。在这篇论文中，我们评估了基于小规模Transformer模型（即参数数量少于10亿个的BERT和RoBERTa）在多标签情感分类任务中的有效性，特别是在处理短文本场景时的效果。具体来说，我们重点研究了影响模型性能的三个关键因素：(1) 在特定领域内进行持续预训练；(2) 使用自动生成样例的数据增强技术（特别是利用生成式数据增强方法）；以及 (3) 分类头的设计变体。 我们的实验结果显示，在短文本情感分类中，数据增强可以显著提高模型的分类性能。然而，继续在扩充后的数据集上进行预训练可能会引入噪声而不是提升准确率。此外，我们确认对分类头部所做的修改仅带来了边际效益。 这些发现为优化资源受限环境下的BERT及其衍生模型提供了实用指导，并有助于完善针对短文本情感分类策略的方法论。
URL
https://arxiv.org/abs/2509.04982
