Abstract
Ensuring fairness in natural language processing for moral sentiment classification is challenging, particularly under cross-domain shifts where transformer models are increasingly deployed. Using the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC), this work evaluates BERT and DistilBERT in a multi-label setting with in-domain and cross-domain protocols. Aggregate performance can mask disparities: we observe pronounced asymmetry in transfer, with Twitter->Reddit degrading micro-F1 by 14.9% versus only 1.5% for Reddit->Twitter. Per-label analysis reveals fairness violations hidden by overall scores; notably, the authority label exhibits Demographic Parity Differences of 0.22-0.23 and Equalized Odds Differences of 0.40-0.41. To address this gap, we introduce the Moral Fairness Consistency (MFC) metric, which quantifies the cross-domain stability of moral foundation detection. MFC shows strong empirical validity, achieving a perfect negative correlation with Demographic Parity Difference (rho = -1.000, p < 0.001) while remaining independent of standard performance metrics. Across labels, loyalty demonstrates the highest consistency (MFC = 0.96) and authority the lowest (MFC = 0.78). These findings establish MFC as a complementary, diagnosis-oriented metric for fairness-aware evaluation of moral reasoning models, enabling more reliable deployment across heterogeneous linguistic contexts. .
Abstract (translated)
确保在道德情感分类中的自然语言处理的公平性是一项挑战，尤其是在变换器模型日益被部署的情况下。当出现跨域转移时，这一问题尤为突出。本研究使用《道德基础推特语料库》(MFTC)和《道德基础Reddit语料库》(MFRC)，在多标签设置下评估了BERT和DistilBERT模型，并采用了同领域及跨领域的协议。总体性能掩盖不平等：我们观察到显著的转移不对称，从Twitter到Reddit的微F1得分下降了14.9%，而从Reddit到Twitter则只下降了1.5%。按标签分析揭示了一些整体分数所未能显现出来的公平性问题；特别地，“权威”这一标签在不同人口统计学中的表现差异显著，其人口均衡差异为0.22-0.23，平等机会差异为0.40-0.41。 为了填补这项空白，我们引入了道德公平一致性（MFC）指标，以量化道德基础检测的跨域稳定性。MFC在经验验证中表现出很强的有效性，与人口均衡差异呈完美的负相关关系（rho = -1.000, p < 0.001），同时保持独立于标准性能度量。 各项标签下，“忠诚”表现出了最高的一致性（MFC = 0.96）而“权威”则最低（MFC = 0.78）。这些发现确立了MFC作为一个补充的、以诊断为导向的公平性评估指标的地位，有助于在异质语言环境中实现道德推理模型更可靠的部署。
URL
https://arxiv.org/abs/2510.11222
