Abstract
Movie dubbing seeks to synthesize speech from a given script using a specific voice, while ensuring accurate lip synchronization and emotion-prosody alignment with the character's visual performance. However, existing alignment approaches based on visual features face two key limitations: (1)they rely on complex, handcrafted visual preprocessing pipelines, including facial landmark detection and feature extraction; and (2) they generalize poorly to unseen visual domains, often resulting in degraded alignment and dubbing quality. To address these issues, we propose InstructDubber, a novel instruction-based alignment dubbing method for both robust in-domain and zero-shot movie dubbing. Specifically, we first feed the video, script, and corresponding prompts into a multimodal large language model to generate natural language dubbing instructions regarding the speaking rate and emotion state depicted in the video, which is robust to visual domain variations. Second, we design an instructed duration distilling module to mine discriminative duration cues from speaking rate instructions to predict lip-aligned phoneme-level pronunciation duration. Third, for emotion-prosody alignment, we devise an instructed emotion calibrating module, which finetunes an LLM-based instruction analyzer using ground truth dubbing emotion as supervision and predicts prosody based on the calibrated emotion analysis. Finally, the predicted duration and prosody, together with the script, are fed into the audio decoder to generate video-aligned dubbing. Extensive experiments on three major benchmarks demonstrate that InstructDubber outperforms state-of-the-art approaches across both in-domain and zero-shot scenarios.
Abstract (translated)
电影配音旨在使用特定的声音从给定的剧本中合成语音，同时确保准确的唇形同步和情感声调与角色视觉表现的一致性。然而，现有的基于视觉特征的对齐方法面临着两个关键限制：（1）它们依赖于复杂的、手动设计的视觉预处理管道，包括面部标志检测和特征提取；（2）它们在未见过的视觉领域中泛化能力差，往往导致对齐质量下降和配音效果不佳。为了解决这些问题，我们提出了InstructDubber，这是一种新的基于指令的对齐配音方法，旨在实现稳健的同类场景内配音和零样本电影配音。 具体而言，首先我们将视频、剧本以及相应的提示输入多模态大型语言模型，生成有关说话速度和视频中描绘的情感状态的自然语言配音指令。这些指令对于视觉领域的变化具有鲁棒性。其次，我们设计了一个基于指令的持续时间提取模块，从说话速度指令中挖掘有辨别力的持续时间线索，以预测唇形同步的音素级发音时长。第三，为了实现情感声调对齐，我们开发了一个基于指令的情感校准模块，该模块通过真实配音情感作为监督来微调大型语言模型（LLM）基础的指令分析器，并根据校准后的情感分析预测声调。 最后，将预测出的持续时间和声调与剧本一起输入音频解码器以生成视频对齐的配音。在三个主要基准上的广泛实验表明，在同类场景内和零样本场景中，InstructDubber均超越了最先进的方法。
URL
https://arxiv.org/abs/2512.17154
