Abstract
Recently supervised learning rapidly develops in scene text segmentation. However, the lack of high-quality datasets and the high cost of pixel annotation greatly limit the development of them. Considering the well-performed few-shot learning methods for downstream tasks, we investigate the application of the few-shot learning method to scene text segmentation. We propose TSAL, which leverages CLIP's prior knowledge to learn text attributes for segmentation. To fully utilize the semantic and texture information in the image, a visual-guided branch is proposed to separately extract text and background features. To reduce data dependency and improve text detection accuracy, the adaptive prompt-guided branch employs effective adaptive prompt templates to capture various text attributes. To enable adaptive prompts capture distinctive text features and complex background distribution, we propose Adaptive Feature Alignment module(AFA). By aligning learnable tokens of different attributes with visual features and prompt prototypes, AFA enables adaptive prompts to capture both general and distinctive attribute information. TSAL can capture the unique attributes of text and achieve precise segmentation using only few images. Experiments demonstrate that our method achieves SOTA performance on multiple text segmentation datasets under few-shot settings and show great potential in text-related domains.
Abstract (translated)
最近，监督学习在场景文本分割领域取得了迅速发展。然而，高质量数据集的缺乏和像素标注成本高昂极大地限制了该领域的进步。鉴于针对下游任务表现良好的少量样本学习方法，我们研究了将少量样本学习方法应用于场景文本分割的应用。为此，我们提出了TSAL（Text Segmentation with Adaptive Prompt Learning），它利用CLIP的先验知识来学习用于分割的文字属性。为了充分利用图像中的语义和纹理信息，提出了一种视觉引导分支，以分别提取文字和背景特征。 为减少对数据的依赖并提高文本检测精度，适应性提示引导分支采用了有效的自适应提示模板来捕捉各种文字属性。为了让自适应提示能够捕获独特且复杂的文字特征及背景分布，我们提出了自适应特征对齐模块（Adaptive Feature Alignment, AFA）。通过将不同属性的学习令牌与视觉特征和提示原型进行对齐，AFA使自适应提示能够同时捕捉通用和独特的属性信息。 TSAL能够在仅使用少量图像的情况下捕获文本的独特属性，并实现精确的分割。实验表明，在多个文本分割数据集下的少量样本设置中，我们的方法达到了最先进的性能，并在与文本相关的领域展示了巨大的潜力。
URL
https://arxiv.org/abs/2504.11164
