Abstract
Effectively handling temporal redundancy remains a key challenge in learning video models. Prevailing approaches often treat each set of frames independently, failing to effectively capture the temporal dependencies and redundancies inherent in videos. To address this limitation, we introduce RefTok, a novel reference-based tokenization method capable of capturing complex temporal dynamics and contextual information. Our method encodes and decodes sets of frames conditioned on an unquantized reference frame. When decoded, RefTok preserves the continuity of motion and the appearance of objects across frames. For example, RefTok retains facial details despite head motion, reconstructs text correctly, preserves small patterns, and maintains the legibility of handwriting from the context. Across 4 video datasets (K600, UCF-101, BAIR Robot Pushing, and DAVIS), RefTok significantly outperforms current state-of-the-art tokenizers (Cosmos and MAGVIT) and improves all evaluated metrics (PSNR, SSIM, LPIPS) by an average of 36.7% at the same or higher compression ratios. When a video generation model is trained using RefTok's latents on the BAIR Robot Pushing task, the generations not only outperform MAGVIT-B but the larger MAGVIT-L, which has 4x more parameters, across all generation metrics by an average of 27.9%.
Abstract (translated)
有效地处理时间冗余仍然是学习视频模型的关键挑战。当前的主流方法通常将每一组帧独立看待，未能有效捕捉视频中固有的时间和冗余关系。为了解决这一局限性，我们引入了RefTok，这是一种新型基于参考的标记化方法，能够捕获复杂的时态动态和上下文信息。我们的方法在未量化的参考帧条件下编码和解码一组帧。当进行解码时，RefTok保持了跨帧之间的运动连续性和对象外观的一致性。例如，即使头部有动作，RefTok也能保留面部细节；正确重构文本内容；保持小图案的完整性，并从上下文中维持手写的可读性。 在四个视频数据集（K600、UCF-101、BAIR机器人推拉任务和DAVIS）上，RefTok显著优于目前最先进的标记化方法（Cosmos 和 MAGVIT），并且在同一或更高的压缩比下，改进了所有评估指标（PSNR、SSIM、LPIPS），平均提高了36.7%。 当使用RefTok的潜在表示在BAIR机器人推拉任务上训练视频生成模型时，所产生的结果不仅超过了MAGVIT-B，在所有生成指标中平均高出27.9%，还超越了参数量是其4倍的更大规模模型MAGVIT-L。
URL
https://arxiv.org/abs/2507.02862
