Abstract
Context-aware methods have achieved remarkable advancements in supervised scene text recognition by leveraging semantic priors from words. Considering the heterogeneity of text and background in STR, we propose that such contextual priors can be reinterpreted as the relations between textual elements, serving as effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of the dataset due to lexical dependencies, which causes over-fitting problem, thus compromising the representation quality. To address this, our work introduces a unified framework of Relational Contrastive Learning and Masked Image Modeling for STR (RCMSTR), which explicitly models the enriched textual relations. For the RCL branch, we first introduce the relational rearrangement module to cultivate new relations on the fly. Based on this, we further conduct relational contrastive learning to model the intra- and inter-hierarchical relations for frames, sub-words and this http URL the other hand, MIM can naturally boost the context information via masking, where we find that the block masking strategy is more effective for STR. For the effective integration of RCL and MIM, we also introduce a novel decoupling design aimed at mitigating the impact of masked images on contrastive learning. Additionally, to enhance the compatibility of MIM with CNNs, we propose the adoption of sparse convolutions and directly sharing the weights with dense convolutions in training. The proposed RCMSTR demonstrates superior performance in various evaluation protocols for different STR-related downstream tasks, outperforming the existing state-of-the-art self-supervised STR techniques. Ablation studies and qualitative experimental results further validate the effectiveness of our this http URL code and pre-trained models will be available at this https URL .
Abstract (translated)
上下文感知的方法通过利用单词中的语义先验，在监督场景文本识别中取得了显著的进步。考虑到STR中文本和背景的异质性，我们提出可以将这些上下文先验重新解释为文本元素之间的关系，作为表示学习的有效自监督标签。然而，由于词汇依赖性，文本关系受到数据集有限大小的限制，这会导致过拟合问题，从而影响表示的质量。为了应对这一挑战，我们的工作提出了一个用于STR（RCMSTR）的关系对比学习和掩码图像建模统一框架，该框架明确地建模了丰富的文本关系。 对于RCL分支，我们首先引入了一个关系重新排列模块来实时培养新的关系。基于此，我们进一步进行关系对比学习以模型帧、子词之间以及与之相关的层级关系。另一方面，MIM可以通过掩码自然地增强上下文信息，在STR中发现块掩码策略更为有效。为了实现RCL和MIM的有效集成，我们也引入了一种新颖的解耦设计，旨在减轻被掩码图像对对比学习的影响。此外，为了提高MIM与CNNs之间的兼容性，我们建议采用稀疏卷积，并直接在训练过程中与密集卷积共享权重。 所提出的RCMSTR在各种评估协议下展示了卓越的表现，超越了现有的最先进的自监督STR技术，在不同的STR相关下游任务中表现出色。消融研究和定性实验结果进一步验证了我们的方法的有效性。代码和预训练模型将于这个网址提供：[此处为链接]。
URL
https://arxiv.org/abs/2411.11219
