Abstract
Video prediction is plagued by a fundamental trilemma: achieving high-resolution and perceptual quality typically comes at the cost of real-time speed, hindering its use in latency-critical applications. This challenge is most acute for autonomous UAVs in dense urban environments, where foreseeing events from high-resolution imagery is non-negotiable for safety. Existing methods, reliant on iterative generation (diffusion, autoregressive models) or quadratic-complexity attention, fail to meet these stringent demands on edge hardware. To break this long-standing trade-off, we introduce RAPTOR, a video prediction architecture that achieves real-time, high-resolution performance. RAPTOR's single-pass design avoids the error accumulation and latency of iterative approaches. Its core innovation is Efficient Video Attention (EVA), a novel translator module that factorizes spatiotemporal modeling. Instead of processing flattened spacetime tokens with $O((ST)^2)$ or $O(ST)$ complexity, EVA alternates operations along the spatial (S) and temporal (T) axes. This factorization reduces the time complexity to $O(S + T)$ and memory complexity to $O(max(S, T))$, enabling global context modeling at $512^2$ resolution and beyond, operating directly on dense feature maps with a patch-free design. Complementing this architecture is a 3-stage training curriculum that progressively refines predictions from coarse structure to sharp, temporally coherent details. Experiments show RAPTOR is the first predictor to exceed 30 FPS on a Jetson AGX Orin for $512^2$ video, setting a new state-of-the-art on UAVid, KTH, and a custom high-resolution dataset in PSNR, SSIM, and LPIPS. Critically, RAPTOR boosts the mission success rate in a real-world UAV navigation task by 18/%, paving the way for safer and more anticipatory embodied agents.
Abstract (translated)
视频预测面临着一个基本的三难困境：实现高分辨率和感知质量通常会牺牲实时速度，这阻碍了其在延迟关键应用中的使用。对于自主无人机（UAV）在密集城市环境中而言，这一挑战尤为严峻，因为从高清图像中预见事件对安全至关重要。现有方法依赖于迭代生成（如扩散模型、自回归模型）或二次复杂度注意力机制，在边缘硬件上无法满足这些严格要求。为打破这种长期存在的权衡关系，我们引入了RAPTOR架构，它在实现实时和高分辨率性能方面取得了突破。 **RAPTOR的关键特性：** 1. **一次性设计**：避免了迭代方法中的误差累积和延迟问题。 2. **高效视频注意力（EVA）模块**：作为核心创新点，该模块通过对空间(S)和时间(T)轴交替操作来分解时空建模。这种分解将时间复杂度降低至$O(S + T)$，内存复杂度降至$O(\max(S, T))$，从而支持在512^2分辨率及以上的全局上下文模型，并直接处理无块设计的密集特征图。 **训练体系结构：** 3. **三阶段训练课程**：此方法逐步细化预测从粗略结构到精细、时间连贯细节的过程。 实验结果显示，在Jetson AGX Orin上，RAPTOR是第一个在512^2视频中超过每秒30帧的预测器，并且在UAVid、KTH及自定义高分辨率数据集上分别在PSNR（峰值信噪比）、SSIM（结构相似度指标）和LPIPS（感知线性图像嵌入距离）方面设置了新的行业标准。最重要的是，RAPTOR在现实世界中的无人机导航任务中将成功率提高了18%，为更安全、更具预见性的自主体铺平了道路。 通过这些创新，RAPTOR架构不仅解决了视频预测领域的一个核心问题，还显著提升了实时高分辨率应用的性能，特别是对于需要即时决策的应用场景，如无人驾驶航空器。
URL
https://arxiv.org/abs/2512.21710
