Abstract
Dialogue topic segmentation supports summarization, retrieval, memory management, and conversational continuity. Despite decades of prior work, evaluation practice in dialogue topic segmentation remains dominated by strict boundary matching and F1-based metrics, even as modern LLM-based conversational systems increasingly rely on segmentation to manage conversation history beyond the model's fixed context window, where unstructured context accumulation degrades efficiency and coherence. This paper introduces an evaluation objective for dialogue topic segmentation that treats boundary density and segment coherence as primary criteria, alongside window-tolerant F1 (W-F1). Through extensive cross-dataset empirical evaluation, we show that reported performance differences across dialogue segmentation benchmarks are driven not by model quality, but by annotation granularity mismatches and sparse boundary labels. This indicates that many reported improvements arise from evaluation artifacts rather than improved boundary detection. We evaluated multiple, structurally distinct dialogue segmentation strategies across eight dialogue datasets spanning task-oriented, open-domain, meeting-style, and synthetic interactions. Across these settings, we observe high segment coherence combined with extreme oversegmentation relative to sparse labels, producing misleadingly low exact-match F1 scores. We show that topic segmentation is best understood as selecting an appropriate granularity rather than predicting a single correct boundary set. We operationalize this view by explicitly separating boundary scoring from boundary selection.
Abstract (translated)
对话话题分割支持总结、检索、记忆管理和会话连续性。尽管已有数十年的研究工作，对话话题分割的评估实践仍主要依赖严格的边界匹配和基于F1的度量标准，而现代大型语言模型（LLM）驱动的对话系统越来越依靠分割来管理超出固定上下文窗口的对话历史记录，在这种情况下，无结构的历史累积会降低效率和连贯性。本文引入了一种新的评估目标，该目标将边界密度和片段连贯性视为主要标准，并在基于窗口的F1（W-F1）度量之外进行考虑。 通过广泛的跨数据集实证评价，我们证明了对话分割基准测试中报告的不同性能主要是由于标注精细度不匹配和稀疏边界标签造成的，而不是模型质量。这表明许多所谓的改进实际上是评估偏差的结果，而非实际提高了边界检测的准确性。我们在八种不同类型的对话数据集中（包括任务导向型、开放领域型、会议式以及合成交互）评估了多种结构不同的对话分割策略，并发现在这些设置中，高片段连贯性与相对稀疏标签过度分割的情况相结合，导致看似较低的确切匹配F1分数。 我们还发现话题划分更应该被视为选择适当的粒度级别而不是预测一个单一正确的边界集合。为此，我们将边界评分和边界选择明确分开以操作化这一观点。
URL
https://arxiv.org/abs/2512.17083
