Abstract
Visual storytelling models that correctly ground entities in images may still hallucinate semantic relationships, generating incorrect dialogue attribution, character interactions, or emotional states. We introduce StoryMovie, a dataset of 1,757 stories aligned with movie scripts and subtitles through LCS matching. Our alignment pipeline synchronizes screenplay dialogue with subtitle timestamps, enabling dialogue attribution by linking character names from scripts to temporal positions from subtitles. Using this aligned content, we generate stories that maintain visual grounding tags while incorporating authentic character names, dialogue, and relationship dynamics. We fine-tune Qwen Storyteller3 on this dataset, building on prior work in visual grounding and entity re-identification. Evaluation using DeepSeek V3 as judge shows that Storyteller3 achieves an 89.9% win rate against base Qwen2.5-VL 7B on subtitle alignment. Compared to Storyteller, trained without script grounding, Storyteller3 achieves 48.5% versus 38.0%, confirming that semantic alignment progressively improves dialogue attribution beyond visual grounding alone.
Abstract (translated)
视觉叙事模型可能正确地将图像中的实体进行定位，但仍可能出现语义关系上的幻觉，从而生成不正确的对话归属、角色互动或情绪状态。为此，我们引入了StoryMovie数据集，该数据集包含1,757个故事，并通过最长公共子序列（LCS）匹配与电影剧本和字幕进行了对齐。我们的对齐管道同步了脚本中的对话与时长标记的时间戳，从而能够将角色名称从脚本中链接到字幕的时序位置上进行对话归属。使用这种对齐的内容生成故事时，我们在保持视觉定位标签的同时，融入真实的角色名、对话和关系动态。 我们在此数据集上对Qwen Storyteller3进行了微调，建立在先前的工作基础上，这些工作涵盖了视觉定位与实体重新识别领域。通过使用DeepSeek V3作为评判标准进行评估后发现，Storyteller3相对于基础版的Qwen2.5-VL 7B，在字幕对齐上获得了89.9%的胜率。相比于未经脚本对齐训练的故事生成器（即仅依赖视觉信息），Storyteller3在对话归属方面的性能得到了显著提升：48.5%对比38.0%，这表明语义上的精确对齐可以逐步改进对话归属，超越单纯依靠视觉定位的效果。
URL
https://arxiv.org/abs/2602.21829
