Abstract
Multimodal relation extraction (MRE) is a crucial task in the fields of Knowledge Graph and Multimedia, playing a pivotal role in multimodal knowledge graph construction. However, existing methods are typically limited to extracting a single type of relational triplet, which restricts their ability to extract triplets beyond the specified types. Directly combining these methods fails to capture dynamic cross-modal interactions and introduces significant computational redundancy. Therefore, we propose a novel \textit{unified multimodal Relation Extraction framework with Multilevel Optimal Transport and mixture-of-Experts}, termed REMOTE, which can simultaneously extract intra-modal and inter-modal relations between textual entities and visual objects. To dynamically select optimal interaction features for different types of relational triplets, we introduce mixture-of-experts mechanism, ensuring the most relevant modality information is utilized. Additionally, considering that the inherent property of multilayer sequential encoding in existing encoders often leads to the loss of low-level information, we adopt a multilevel optimal transport fusion module to preserve low-level features while maintaining multilayer encoding, yielding more expressive representations. Correspondingly, we also create a Unified Multimodal Relation Extraction (UMRE) dataset to evaluate the effectiveness of our framework, encompassing diverse cases where the head and tail entities can originate from either text or image. Extensive experiments show that REMOTE effectively extracts various types of relational triplets and achieves state-of-the-art performanc on almost all metrics across two other public MRE datasets. We release our resources at this https URL.
Abstract (translated)
多模态关系抽取（MRE）是知识图谱和多媒体领域中的一个关键任务，在多模态知识图构建中扮演着重要角色。然而，现有的方法通常只能提取单一类型的三元组关系，这限制了它们在指定类型之外的三元组提取能力。直接组合这些方法无法捕捉动态跨模式交互，并且会引入显著的计算冗余。因此，我们提出了一种新型框架——“多级最优传输与专家混合型统一多模态关系抽取”（REMOTE），它可以同时提取文本实体和视觉对象之间的同模态和异模态关系。 为了为不同类型的三元组动态选择最佳交互特征，我们在混合专家机制中引入了这一方法，确保最相关的模式信息被利用。此外，鉴于现有编码器的多层序列编码特性往往会失去低级信息，我们采用了多层次最优传输融合模块来保留低级特征的同时保持多层编码，从而产生更具有表现力的表示形式。 相应地，为了评估我们的框架的有效性，我们还创建了一个统一的多模态关系抽取（UMRE）数据集，涵盖了各种情况，在这些情况下，头部和尾部实体可以分别来自文本或图像。广泛的实验表明，REMOTE能够有效地提取各种类型的三元组，并在其他两个公开MRE数据集中几乎所有度量标准上均实现了最先进的性能。 我们已将资源发布在此链接：[this https URL]（请根据实际需求替换具体的URL地址）。
URL
https://arxiv.org/abs/2509.04844
