Abstract
Chinese scene text retrieval is a practical task that aims to search for images containing visual instances of a Chinese query text. This task is extremely challenging because Chinese text often features complex and diverse layouts in real-world scenes. Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance. In this paper, we establish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DL-CSVTR), which is specifically designed to evaluate retrieval performance across various text layouts, including vertical, cross-line, and partial alignments. To address the limitations in existing methods, we propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a novel model that integrates global visual information with multi-granularity alignment training. CSTR-CLIP applies a two-stage training process to overcome previous limitations, such as the exclusion of visual features outside the text region and reliance on single-granularity alignment, thereby enabling the model to effectively handle diverse text layouts. Experiments on existing benchmark show that CSTR-CLIP outperforms the previous state-of-the-art model by 18.82% accuracy and also provides faster inference speed. Further analysis on DL-CSVTR confirms the superior performance of CSTR-CLIP in handling various text layouts. The dataset and code will be publicly available to facilitate research in Chinese scene text retrieval.
Abstract (translated)
中文场景文本检索是一项实用的任务，旨在搜索包含查询文本视觉实例的图像。由于实际场景中中文文本经常具有复杂多样的布局形式，这一任务极具挑战性。目前的研究通常沿用了英文场景文本检索的方法，导致无法取得令人满意的性能。 在本文中，我们建立了一个多样化布局基准（Diversified Layout benchmark）用于评估中文街景文本检索（DL-CSVTR），该基准特别设计来评估不同文本布局下的检索性能，包括竖直、跨行和部分对齐的场景。为了解决现有方法中的局限性，我们提出了一个新的模型——中文场景文本检索CLIP (CSTR-CLIP)，这个模型整合了全局视觉信息并采用多粒度对齐训练策略。 CSTR-CLIP 通过两阶段训练过程克服了先前模型的一些限制，比如忽略文字区域外的视觉特征和依赖单一粒度的对齐方式。这使得模型能够有效地处理多样化的文本布局。 在现有基准上的实验表明，与最先进的方法相比，CSTR-CLIP 在准确率上提高了18.82%，并且提供了更快的推理速度。进一步在 DL-CSVTR 上的分析证实了 CSTR-CLIP 处理各种文本布局时的优越性。数据集和代码将公开发布，以促进中文场景文本检索的研究进展。 通过这种方式，CSTR-CLIP 不仅提升了模型在处理复杂、多变中文街景文本任务上的表现，同时也为研究人员提供了更加全面且实用的数据资源与工具支持。
URL
https://arxiv.org/abs/2506.04999
