Abstract
Text images are unique in their dual nature, encompassing both visual and linguistic information. The visual component encompasses structural and appearance-based features, while the linguistic dimension incorporates contextual and semantic elements. In scenarios with degraded visual quality, linguistic patterns serve as crucial supplements for comprehension, highlighting the necessity of integrating both aspects for robust scene text recognition (STR). Contemporary STR approaches often use language models or semantic reasoning modules to capture linguistic features, typically requiring large-scale annotated datasets. Self-supervised learning, which lacks annotations, presents challenges in disentangling linguistic features related to the global context. Typically, sequence contrastive learning emphasizes the alignment of local features, while masked image modeling (MIM) tends to exploit local structures to reconstruct visual patterns, resulting in limited linguistic knowledge. In this paper, we propose a Linguistics-aware Masked Image Modeling (LMIM) approach, which channels the linguistic information into the decoding process of MIM through a separate branch. Specifically, we design a linguistics alignment module to extract vision-independent features as linguistic guidance using inputs with different visual appearances. As features extend beyond mere visual structures, LMIM must consider the global context to achieve reconstruction. Extensive experiments on various benchmarks quantitatively demonstrate our state-of-the-art performance, and attention visualizations qualitatively show the simultaneous capture of both visual and linguistic information.
Abstract (translated)
文本图像的独特之处在于其双重性质，既包含视觉信息也包括语言信息。视觉部分涵盖了结构和外观特征，而语言维度则包含了上下文与语义元素。在视觉质量下降的情况下，语言模式作为理解的重要补充显得尤为关键，这强调了为了实现稳健的场景文本识别（STR），融合这两种特性的重要性。 现代STR方法通常采用语言模型或语义推理模块来捕捉语言特征，这些方法往往需要大规模标注数据集的支持。无监督学习由于缺乏标注信息，在分离与全局上下文相关的语言特征方面存在挑战。通常情况下，序列对比学习侧重于局部特征的对齐，而掩码图像建模（MIM）倾向于利用局部结构来重建视觉模式，从而导致语言知识的获取有限。 在本文中，我们提出了一种基于意识的语言掩码图像建模（LMIM）方法，该方法通过单独分支将语言信息引入MIM的解码过程中。具体而言，我们设计了一个语言对齐模块，利用具有不同视觉外观的输入提取与视觉无关的特征作为语言指导。随着特征扩展到超越单纯的视觉结构之外，LMIM必须考虑全局上下文以实现重建。 在多个基准测试中的大量实验定量地展示了我们的方法达到了最先进的性能水平，并且注意力可视化定性地显示了同时捕获视觉和语言信息的能力。
URL
https://arxiv.org/abs/2503.18746
