Abstract
Previous works based on Segment Anything Model (SAM) have achieved promising performance in unified scene text detection and layout analysis. However, the typical reliance on pixel-level text segmentation for sampling thousands of foreground points as prompts leads to unsatisfied inference latency and limited data utilization. To address above issues, we propose ET-SAM, an Efficient framework with two decoders for unified scene Text detection and layout analysis based on SAM. Technically, we customize a lightweight point decoder that produces word heatmaps for achieving a few foreground points, thereby eliminating excessive point prompts and accelerating inference. Without the dependence on pixel-level segmentation, we further design a joint training strategy to leverage existing data with heterogeneous text-level annotations. Specifically, the datasets with multi-level, word-level only, and line-level only annotations are combined in parallel as a unified training set. For these datasets, we introduce three corresponding sets of learnable task prompts in both the point decoder and hierarchical mask decoder to mitigate discrepancies across this http URL experiments demonstrate that, compared to the previous SAM-based architecture, ET-SAM achieves about 3$\times$ inference acceleration while obtaining competitive performance on HierText, and improves an average of 11.0% F-score on Total-Text, CTW1500, and ICDAR15.
Abstract (translated)
基于分割任意模型（SAM）的先前工作已在统一场景文本检测与布局分析中取得了令人瞩目的性能。然而，典型方法依赖像素级文本分割来采样数千个前景点作为提示，这导致了令人不满意的推理延迟和有限的数据利用率。为解决上述问题，我们提出了ET-SAM，一个基于SAM的高效双解码器框架，用于统一场景文本检测与布局分析。技术上，我们定制了一个轻量级点解码器，用于生成词热力图以实现少量前景点，从而消除过多的点提示并加速推理。在不依赖像素级分割的前提下，我们进一步设计了一种联合训练策略，以利用具有异构文本级标注的现有数据。具体而言，将具有多级、仅词级和仅行级标注的数据集并行合并为一个统一的训练集。针对这些数据集，我们在点解码器和层级掩码解码器中引入了三组相应的可学习任务提示，以缓解跨数据集差异。实验表明，与先前基于SAM的架构相比，ET-SAM实现了约3倍的推理加速，同时在HierText上获得了具有竞争力的性能，并在Total-Text、CTW1500和ICDAR15上平均提升了11.0%的F-score。
URL
https://arxiv.org/abs/2603.25168
