Abstract
Long-document topic segmentation plays an important role in information retrieval and document understanding, yet existing methods still show clear shortcomings in ultra-long text settings. Traditional discriminative models are constrained by fixed windows and cannot model document-level semantics; generative large language models can output paragraph boundaries, but inference is expensive and long inputs are difficult to support. To address these issues, we propose a discriminative segmentation model based on Qwen3-0.6B. On top of the backbone network, we add a cross-window context fusion layer and a boundary classification head, and combine them with an overlapping sliding-window strategy. Our model supports single-pass inputs of up to 13k tokens and can be extended to ultra-long documents for paragraph boundary detection. To further enhance downstream retrieval efficiency, we derive a vector fusion method with scalar correction, which compresses the representation of ultra-long segments into a single vector without semantic loss. Experiments on the Wikipedia long-document topic segmentation dataset WIKI-727K show that, compared with three generative models based on Qwen2-0.5B released by Jina, our method achieves a better macro-averaged F1 and delivers two orders of magnitude faster inference, substantially improving the practicality and scalability of long-document processing.
Abstract (translated)
长文档主题分段在信息检索和文档理解中扮演着重要角色，但现有方法在处理超长文本时仍存在明显不足。传统判别模型受到固定窗口的限制，无法捕捉到文档级别的语义；而生成式的大语言模型能够输出段落边界，但在推理过程中成本高昂，并且难以支持长输入。为解决这些问题，我们提出了一种基于Qwen3-0.6B的判别分段模型。在基础网络之上，我们添加了一个跨窗口上下文融合层和一个边界分类头，并结合了重叠滑动窗口策略。我们的模型能够单次处理长达13K令牌的输入，并可扩展至超长文档进行段落边界的检测。 为了进一步提高下游检索效率，我们推导出了一种向量融合方法，该方法通过标量修正将超长片段的表示压缩为单一向量，同时保持语义不变。在维基百科长文档主题分段数据集WIKI-727K上的实验表明，与Jina基于Qwen2-0.5B发布的三种生成模型相比，我们的方法实现了更好的宏平均F1分数，并且推理速度快两个数量级，显著提高了处理长文档的实用性和可扩展性。
URL
https://arxiv.org/abs/2602.23370
