Optical_Flow
-
VADMamba++: Efficient Video Anomaly Detection via Hybrid Modeling in Grayscale Space
2026-04-01 01:14:31Jihao Lyu, Minghua Zhao, Jing Hu, Yifei Chen, Shuangli Du, Cheng ShiAbstract
VADMamba pioneered the introduction of Mamba to Video Anomaly Detection (VAD), achieving high accuracy and fast inference through hybrid proxy tasks. Nevertheless, its heavy reliance on optical flow as auxiliary input and inter-task fusion scoring constrains its applicability to a single proxy task. In this paper, we introduce VADMamba++, an efficient VAD method based on the Gray-to-RGB paradigm that enforces a Single-Channel to Three-Channel reconstruction mapping, designed for a single proxy task and operating without auxiliary inputs. This paradigm compels inferring color appearances from grayscale structures, allowing anomalies to be more effectively revealed through dual inconsistencies between structure and chromatic cues. Specifically, VADMamba++ reconstructs grayscale frames into the RGB space to simultaneously discriminate structural geometry and chromatic fidelity, thereby enhancing sensitivity to explicit visual anomalies. We further design a hybrid modeling backbone that integrates Mamba, CNN, and Transformer modules to capture diverse normal patterns while suppressing the appearance of anomalies. Furthermore, an intra-task fusion scoring strategy integrates explicit future-frame prediction errors with implicit quantized feature errors, further improving accuracy under a single task setting. Extensive experiments on three benchmark datasets demonstrate that VADMamba++ outperforms state-of-the-art methods while meeting performance and efficiency, especially under a strict single-task setting with only frame-level inputs.
Abstract (translated)
URL
https://arxiv.org/abs/2604.00360
PDF
-
FlowIt: Global Matching for Optical Flow with Confidence-Guided Refinement
2026-03-30 17:58:12Sadra Safadoust, Fabio Tosi, Matteo Poggi, Fatma G\"uneyAbstract
We present FlowIt, a novel architecture for optical flow estimation designed to robustly handle large pixel displacements. At its core, FlowIt leverages a hierarchical transformer architecture that captures extensive global context, enabling the model to effectively model long-range correspondences. To overcome the limitations of localized matching, we formulate the flow initialization as an optimal transport problem. This formulation yields a highly robust initial flow field, alongside explicitly derived occlusion and confidence maps. These cues are then seamlessly integrated into a guided refinement stage, where the network actively propagates reliable motion estimates from high-confidence regions into ambiguous, low-confidence areas. Extensive experiments across the Sintel, KITTI, Spring, and LayeredFlow datasets validate the efficacy of our approach. FlowIt achieves state-of-the-art results on the competitive Sintel and KITTI benchmarks, while simultaneously establishing new state-of-the-art cross-dataset zero-shot generalization performance on Sintel, Spring, and LayeredFlow.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28759
PDF
-
RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing
2026-03-30 01:49:36Changyeon Won, Hyunjun Jung, Jungu Cho, Seonmi Park, Chi-Hoon Lee, Hae-Gon JeonAbstract
Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27948
PDF
-
Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach
2026-03-27 10:07:59Luu Tu Nguyen, Thi Bich Phuong Man, Vu Tram Anh Khuong, Thanh Ha Le, Thi Duyen NgoAbstract
Micro-expression recognition is vital for affective computing but remains challenging due to the extremely brief, low-intensity facial motions involved and the high-dimensional nature of 4D mesh data. To address these challenges, we introduce a dual-view optical flow approach that simplifies mesh processing by capturing each micro-expression sequence from two synchronized viewpoints and computing optical flow to represent motion. Our pipeline begins with view separation and sequence-wise face cropping to ensure spatial consistency, followed by automatic apex-frame detection based on peak motion intensity in both views. We decompose each sequence into onset-apex and apex-offset phases, extracting horizontal, vertical, and magnitude flow channels for each phase. These are fed into our Triple-Stream MicroAttNet, which employs a fusion attention module to adaptively weight modality-specific features and a squeeze-and-excitation block to enhance magnitude representations. Training uses focal loss to mitigate class imbalance and the Adam optimizer with early stopping. Evaluated on the multi-label 4DME dataset, comprising 24 subjects and five emotion categories, in the 4DMR IJCAI Workshop Challenge 2025, our method achieves a macro-UF1 score of 0.536, outperforming the official baseline by over 50\% and securing first place. Ablation studies confirm that both the fusion attention and SE components each contribute up to 3.6 points of UF1 gain. These results demonstrate that dual-view, phase-aware optical flow combined with multi-stream fusion yields a robust and interpretable solution for 4D micro-expression recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26849
PDF
-
ANVIL: Accelerator-Native Video Interpolation via Codec Motion Vector Priors
Abstract
Mobile displays refresh at 90-120 Hz, yet most video is encoded at 24-30 frames per second; real-time frame-rate doubling requires each synthesized frame within 33.3 ms on mobile neural processing units. We show that mainstream flow-based video frame interpolation faces three structural deployment barriers on mobile accelerators: spatial sampling operators exceed the frame budget or lack hardware support, iterative flow refinement collapses under 8-bit post-training quantization, and memory-bound operators dominate the inference graph. ANVIL addresses these barriers by reusing motion vectors already computed by the H.264 decoder to prealign input frames, removing learned optical flow, spatial sampling, and iterative accumulation from the accelerator graph. The remaining residual is refined by a convolution-dominated network whose inference graph is composed almost entirely of compute-bound operators. On a Snapdragon 8 Gen 3 device, ANVIL achieves 12.8 ms 1080p network inference in 8-bit integer precision; an open-source Android player sustains 28.4 ms median end-to-end latency per interpolated frame pair over 54,623 consecutively logged samples during 30-minute continuous playback. Per-operator causal analysis identifies quantized accumulation on recurrent flow states as a key mechanism behind integer quantization failure in iterative methods. The current design targets H.264 playback scenarios with decoder-exposed motion vectors.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26835
PDF
-
MegaFlow: Zero-Shot Large Displacement Optical Flow
2026-03-26 17:59:51Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys, Haofei XuAbstract
Accurate estimation of large displacement optical flow remains a critical challenge. Existing methods typically rely on iterative local search or/and domain-specific fine-tuning, which severely limits their performance in large displacement and zero-shot generalization scenarios. To overcome this, we introduce MegaFlow, a simple yet powerful model for zero-shot large displacement optical flow. Rather than relying on highly complex, task-specific architectural designs, MegaFlow adapts powerful pre-trained vision priors to produce temporally consistent motion fields. In particular, we formulate flow estimation as a global matching problem by leveraging pre-trained global Vision Transformer features, which naturally capture large displacements. This is followed by a few lightweight iterative refinements to further improve the sub-pixel accuracy. Extensive experiments demonstrate that MegaFlow achieves state-of-the-art zero-shot performance across multiple optical flow benchmarks. Moreover, our model also delivers highly competitive zero-shot performance on long-range point tracking benchmarks, demonstrating its robust transferability and suggesting a unified paradigm for generalizable motion estimation. Our project page is at: this https URL.
Abstract (translated)
大位移光流的精确估计仍是一项关键挑战。现有方法通常依赖迭代局部搜索和/或领域特定的微调，这严重限制了其在大位移和零样本泛化场景下的性能。为克服此问题，我们提出了MegaFlow——一个简单而强大的零样本大位移光流模型。MegaFlow不依赖高度复杂、任务特定的架构设计，而是通过适配强大的预训练视觉先验来生成时序一致的运动场。具体而言，我们利用预训练的全局视觉Transformer特征将光流估计建模为全局匹配问题，这些特征天然能够捕捉大位移；随后通过少量轻量级迭代优化进一步提升亚像素精度。大量实验表明，MegaFlow在多个光流基准测试中达到了最先进的零样本性能。此外，我们的模型在长程点跟踪基准测试中也展现出极具竞争力的零样本表现，体现了其强大的迁移能力，并暗示了可泛化运动估计的统一范式。项目页面地址：此https链接。
URL
https://arxiv.org/abs/2603.25739
PDF
-
LILAC: Language-Conditioned Object-Centric Optical Flow for Open-Loop Trajectory Generation
2026-03-26 14:21:22Motonari Kambara, Koki Seno, Tomoya Kaichi, Yanan Wang, Komei SugiuraAbstract
We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at this https URL.
Abstract (translated)
我们采用基于流式的轨迹生成方法来解决语言条件机器人操作问题，该方法支持利用人类操作和网络视频进行训练，且仅需极少量的具身数据。由于需要根据操作前图像和自然语言指令生成物体轨迹，并确保指令与流场的对齐，该任务具有挑战性。为应对这一挑战，我们提出了基于流式的语言指令引导开环动作生成器（LILAC）。这一基于流式的视觉-语言-动作模型（VLA）可从RGB图像和自然语言指令中生成以物体为中心的二维光流，并将光流转换为6自由度机械臂轨迹。LILAC包含两个关键组件：语义对齐损失（用于强化语言条件以生成与指令对齐的光流）和提示条件跨模态适配器（用于将学习到的视觉提示与图像及文本特征对齐，为流场生成提供丰富线索）。实验表明，我们的方法在多个基准测试中生成的流场质量优于现有方法。此外，在使用自由形式指令的物理物体操作实验中，LILAC的任务成功率也显著高于现有方法。项目页面可通过此链接访问。
URL
https://arxiv.org/abs/2603.25481
PDF
-
MoRGS: Efficient Per-Gaussian Motion Reasoning for Streamable Dynamic 3D Scenes
2026-03-26 05:25:11Wonjoon Lee, Sungmin Woo, Donghyeong Kim, Jungho Lee, Sangheon Park, Sangyoun LeeAbstract
Online reconstruction of dynamic scenes aims to learn from streaming multi-view inputs under low-latency constraints. The fast training and real-time rendering capabilities of 3D Gaussian Splatting have made on-the-fly reconstruction practically feasible, enabling online 4D reconstruction. However, existing online approaches, despite their efficiency and visual quality, fail to learn per-Gaussian motion that reflects true scene dynamics. Without explicit motion cues, appearance and motion are optimized solely under photometric loss, causing per-Gaussian motion to chase pixel residuals rather than true 3D motion. To address this, we propose MoRGS, an efficient online per-Gaussian motion reasoning framework that explicitly models per-Gaussian motion to improve 4D reconstruction quality. Specifically, we leverage optical flow on a sparse set of key views as lightweight motion cues that regularize per-Gaussian motion beyond photometric supervision. To compensate for the sparsity of flow supervision, we learn a per-Gaussian motion offset field that reconciles discrepancies between projected 3D motion and observed flow across views and time. In addition, we introduce a per-Gaussian motion confidence that separates dynamic from static Gaussians and weights Gaussian attribute residual updates, thereby suppressing redundant motion in static regions for better temporal consistency and accelerating the modeling of large motions. Extensive experiments demonstrate that MoRGS achieves state-of-the-art reconstruction quality and motion fidelity among online methods, while maintaining streamable performance.
Abstract (translated)
在线动态场景重建旨在低延迟约束下从流式多视角输入中学习。3D高斯泼溅（3D Gaussian Splatting）的快速训练与实时渲染能力使即时重建成为可能，从而支持在线4D重建。然而，现有在线方法尽管具备效率与视觉质量优势，却未能学习反映真实场景动态的每高斯运动。缺乏显式运动线索时，外观与运动仅通过光度损失进行优化，导致每高斯运动追逐像素残差而非真实3D运动。为解决此问题，我们提出MoRGS——一种高效的在线每高斯运动推理框架，通过显式建模每高斯运动来提升4D重建质量。具体而言，我们利用稀疏关键视角上的光流作为轻量级运动线索，以超越光度监督来约束每高斯运动。为弥补流监督的稀疏性，我们学习每高斯运动偏移场，以协调跨视角与时间上投影3D运动与观测流之间的差异。此外，我们引入每高斯运动置信度，区分动态与静态高斯，并对高斯属性残差更新进行加权，从而抑制静态区域的冗余运动以获得更佳时间一致性，并加速大运动建模。大量实验表明，MoRGS在在线方法中实现了最先进的重建质量与运动保真度，同时保持流式处理性能。
URL
https://arxiv.org/abs/2603.25042
PDF
-
Relaxed Rigidity with Ray-based Grouping for Dynamic Gaussian Splatting
2026-03-26 03:38:38Junoh Leea, Junmyeong Lee, Yeon-Ji Song, Inhwan Bae, Jisu Shin, Hae-Gon Jeon, Jin-Hwa KimAbstract
The reconstruction of dynamic 3D scenes using 3D Gaussian Splatting has shown significant promise. A key challenge, however, remains in modeling realistic motion, as most methods fail to align the motion of Gaussians with real-world physical dynamics. This misalignment is particularly problematic for monocular video datasets, where failing to maintain coherent motion undermines local geometric structure, ultimately leading to degraded reconstruction quality. Consequently, many state-of-the-art approaches rely heavily on external priors, such as optical flow or 2D tracks, to enforce temporal coherence. In this work, we propose a novel method to explicitly preserve the local geometric structure of Gaussians across time in 4D scenes. Our core idea is to introduce a view-space ray grouping strategy that clusters Gaussians intersected by the same ray, considering only those whose $\alpha$-blending weights exceed a threshold. We then apply constraints to these groups to maintain a consistent spatial distribution, effectively preserving their local geometry. This approach enforces a more physically plausible motion model by ensuring that local geometry remains stable over time, eliminating the reliance on external guidance. We demonstrate the efficacy of our method by integrating it into two distinct baseline models. Extensive experiments on challenging monocular datasets show that our approach significantly outperforms existing methods, achieving superior temporal consistency and reconstruction quality.
Abstract (translated)
利用3D高斯泼溅技术重建动态3D场景已展现出显著前景。然而，一个关键挑战仍在于模拟逼真的运动——大多数方法未能使高斯元的运动与真实物理动力学保持一致。这种偏差在单目视频数据集中尤为突出，因为无法维持连贯的运动会破坏局部几何结构，最终导致重建质量下降。因此，许多先进方法严重依赖光流或二维轨迹等外部先验信息来强制实现时间一致性。本研究提出一种新方法，旨在4D场景中显式保持高斯元跨时间的局部几何结构。核心思路是引入视图空间射线分组策略：将同一射线相交的高斯元进行聚类，仅考虑α混合权重超过阈值的高斯元，随后对这些分组施加约束以维持稳定的空间分布，从而有效保留其局部几何形态。该方法通过确保局部几何随时间保持稳定，构建了更符合物理规律的运动模型，并消除了对外部引导的依赖。我们将该方法集成到两个不同的基线模型中验证其有效性，在具有挑战性的单目数据集上的大量实验表明，本方案显著优于现有方法，在时间一致性和重建质量两方面均达到更优表现。
URL
https://arxiv.org/abs/2603.24994
PDF
-
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
2026-03-25 06:37:12Hieu Hoang, Dung Trung Tran, Hong Nguyen, Nam-Phong NguyenAbstract
Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.
Abstract (translated)
在线动作检测（OAD）系统面临两大主要挑战：计算成本高昂，以及对背景运动中的判别性时序动态建模不足。添加光流虽能提供强运动线索，但会带来巨大的计算开销。我们提出CAKE，一个基于光流蒸馏的OAD框架，用于将运动知识迁移至RGB模型。我们设计了动态运动适配器（DMA）以抑制静态背景噪声并突出像素变化，从而在无需显式计算的情况下有效近似光流。该框架还集成了浮动对比学习策略，以区分具有信息量的运动动态与时序背景。在TVSeries、THUMOS'14和Kinetics-400数据集上的多项实验表明了我们模型的有效性。在使用相同主干网络的情况下，CAKE实现了相较于SOTA方法的显著平均精度均值提升。我们的模型在单CPU上运行速度超过72 FPS，非常适合资源受限的系统。
URL
https://arxiv.org/abs/2603.23988
PDF
-
DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models
2026-03-24 17:59:13Jaewon Min, Jaeeun Lee, Yeji Choi, Paul Hyunbin Cho, Jin Hyeon Kim, Tae-Young Lee, Jongsik Ahn, Hwayeong Lee, Seonghyun Park, Seungryong KimAbstract
Optical flow models trained on high-quality data often degrade severely when confronted with real-world corruptions such as blur, noise, and compression artifacts. To overcome this limitation, we formulate Degradation-Aware Optical Flow, a new task targeting accurate dense correspondence estimation from real-world corrupted videos. Our key insight is that the intermediate representations of image restoration diffusion models are inherently corruption-aware but lack temporal awareness. To address this limitation, we lift the model to attend across adjacent frames via full spatio-temporal attention, and empirically demonstrate that the resulting features exhibit zero-shot correspondence capabilities. Based on this finding, we present DA-Flow, a hybrid architecture that fuses these diffusion features with convolutional features within an iterative refinement framework. DA-Flow substantially outperforms existing optical flow methods under severe degradation across multiple benchmarks.
Abstract (translated)
在高质量数据上训练的光流模型，在面对模糊、噪声和压缩伪影等真实世界退化时，性能往往会严重下降。为克服这一限制，我们提出了“退化感知光流”（Degradation-Aware Optical Flow）这一新任务，旨在从真实世界退化的视频中实现准确的稠密对应估计。我们的核心见解是：图像修复扩散模型的中间表示天然具备退化感知能力，但缺乏时序感知能力。为解决此问题，我们通过全时空注意力机制使模型能够跨相邻帧进行关注，并实证表明由此产生的特征具有零样本对应能力。基于这一发现，我们提出了DA-Flow——一种混合架构，它在迭代精炼框架内融合了这些扩散特征与卷积特征。在多个基准测试的严重退化场景下，DA-Flow的表现均显著优于现有光流方法。
URL
https://arxiv.org/abs/2603.23499
PDF
-
TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation
2026-03-24 17:53:41Jini Yang, Eunbeen Hong, Soowon Son, Hyunkoo Lee, Sunghwan Hong, Sunok Kim, Seungryong KimAbstract
Event cameras capture per-pixel brightness changes with microsecond resolution, offering continuous motion information lost between RGB frames. However, existing event-based motion estimators depend on large-scale synthetic data that often suffers from a significant sim-to-real gap. We propose TETO (Tracking Events with Teacher Observation), a teacher-student framework that learns event motion estimation from only $\sim$25 minutes of unannotated real-world recordings through knowledge distillation from a pretrained RGB tracker. Our motion-aware data curation and query sampling strategy maximizes learning from limited data by disentangling object motion from dominant ego-motion. The resulting estimator jointly predicts point trajectories and dense optical flow, which we leverage as explicit motion priors to condition a pretrained video diffusion transformer for frame interpolation. We achieve state-of-the-art point tracking on EVIMO2 and optical flow on DSEC using orders of magnitude less training data, and demonstrate that accurate motion estimation translates directly to superior frame interpolation quality on BS-ERGB and HQ-EVFI.
Abstract (translated)
事件相机以微秒级分辨率捕捉像素级亮度变化，提供RGB帧间丢失的连续运动信息。然而，现有基于事件的运动估计器依赖于大规模合成数据，常受显著的仿真到现实差距影响。我们提出TETO（通过教师观测追踪事件），一个师生框架，仅通过从预训练RGB追踪器进行知识蒸馏，就能从约25分钟未标注的真实世界记录中学习事件运动估计。我们提出的运动感知数据整理与查询采样策略，通过将物体运动与主导的自我运动解耦，最大化有限数据的学习效果。所得估计器联合预测点轨迹和密集光流，我们将其作为显式运动先验，用于调节预训练的视频扩散Transformer以实现帧插值。我们在EVIMO2上实现最先进的点追踪，在DSEC上实现最先进的光流，训练数据量减少数个数量级，并证明准确运动估计可直接提升BS-ERGB和HQ-EVFI上的帧插值质量。
URL
https://arxiv.org/abs/2603.23487
PDF
-
GenOpticalFlow: A Generative Approach to Unsupervised Optical Flow Learning
2026-03-23 17:55:52Yixuan Luo, Feng Qiao, Zhexiao Xiong, Yanjing Li, Nathan JacobsAbstract
Optical flow estimation is a fundamental problem in computer vision, yet the reliance on expensive ground-truth annotations limits the scalability of supervised approaches. Although unsupervised and semi-supervised methods alleviate this issue, they often suffer from unreliable supervision signals based on brightness constancy and smoothness assumptions, leading to inaccurate motion estimation in complex real-world scenarios. To overcome these limitations, we introduce \textbf{\modelname}, a novel framework that synthesizes large-scale, perfectly aligned frame--flow data pairs for supervised optical flow training without human annotations. Specifically, our method leverages a pre-trained depth estimation network to generate pseudo optical flows, which serve as conditioning inputs for a next-frame generation model trained to produce high-fidelity, pixel-aligned subsequent frames. This process enables the creation of abundant, high-quality synthetic data with precise motion correspondence. Furthermore, we propose an \textit{inconsistent pixel filtering} strategy that identifies and removes unreliable pixels in generated frames, effectively enhancing fine-tuning performance on real-world datasets. Extensive experiments on KITTI2012, KITTI2015, and Sintel demonstrate that \textbf{\modelname} achieves competitive or superior results compared to existing unsupervised and semi-supervised approaches, highlighting its potential as a scalable and annotation-free solution for optical flow learning. We will release our code upon acceptance.
Abstract (translated)
光流估计是计算机视觉中的基础问题，但依赖昂贵的真值标注限制了监督方法的可扩展性。尽管无监督与半监督方法缓解了这一问题，它们常因基于亮度恒定和平滑性假设的不可靠监督信号，在复杂真实场景中产生不准确的运动估计。为克服这些限制，我们提出\textbf{\modelname}，一种无需人工标注即可为监督光流训练合成大规模、精确对齐的帧-光流数据对的新框架。具体而言，我们的方法利用预训练深度估计网络生成伪光流，作为下一帧生成模型的输入条件，该模型经训练可生成高保真、像素对齐的后续帧。此过程能创建具有精确运动对应关系的丰富高质量合成数据。此外，我们提出\textit{不一致像素过滤}策略，识别并移除生成帧中的不可靠像素，有效提升在真实数据集上的微调性能。在KITTI2012、KITTI2015和Sintel上的大量实验表明，\textbf{\modelname}相比现有无监督与半监督方法取得了具有竞争力或更优的结果，凸显其作为可扩展、无标注解决方案的光流学习潜力。代码将在论文接收后公开。
URL
https://arxiv.org/abs/2603.22270
PDF
-
$x^2$-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space
2026-03-17 15:36:57Ruishan Guo, Ciyu Ruan, Haoyang Wang, Zihang Gong, Jingao Xu, Xinlei ChenAbstract
Estimating dense 2D optical flow and 3D scene flow is essential for dynamic scene understanding. Recent work combines images, LiDAR, and event data to jointly predict 2D and 3D motion, yet most approaches operate in separate heterogeneous feature spaces. Without a shared latent space that all modalities can align to, these systems rely on multiple modality-specific blocks, leaving cross-sensor mismatches unresolved and making fusion unnecessarily this http URL cameras naturally provide a spatiotemporal edge signal, which we can treat as an intrinsic edge field to anchor a unified latent representation, termed the Event Edge Space. Building on this idea, we introduce $x^2$-Fusion, which reframes multimodal fusion as representation unification: event-derived spatiotemporal edges define an edge-centric homogeneous space, and image and LiDAR features are explicitly aligned in this shared this http URL this space, we perform reliability-aware adaptive fusion to estimate modality reliability and emphasize stable cues under degradation. We further employ cross-dimension contrast learning to tightly couple 2D optical flow with 3D scene flow. Extensive experiments on both synthetic and real benchmarks show that $x^2$-Fusion achieves state-of-the-art accuracy under standard conditions and delivers substantial improvements in challenging scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16671
PDF
-
LenghuSky-8: An 8-Year All-Sky Cloud Dataset with Star-Aware Masks and Alt-Az Calibration for Segmentation and Nowcasting
2026-03-17 12:05:21Yicheng Rui, Xiao-Wei Duan, Licai Deng, Fan Yang, Zhengming Dang, Zhengjun Du, Junhao Peng, Wenhao Chu, Umut Mahmut, Kexin Li, Yiyun Wu, Fabo FengAbstract
Ground-based time-domain observatories require minute-by-minute, site-scale awareness of cloud cover, yet existing all-sky datasets are short, daylight-biased, or lack astrometric calibration. We present LenghuSky-8, an eight-year (2018-2025) all-sky imaging dataset from a premier astronomical site, comprising 429,620 $512 \times 512$ frames with 81.2% night-time coverage, star-aware cloud masks, background masks, and per-pixel altitude-azimuth (Alt-Az) calibration. For robust cloud segmentation across day, night, and lunar phases, we train a linear probe on DINOv3 local features and obtain 93.3% $\pm$ 1.1% overall accuracy on a balanced, manually labeled set of 1,111 images. Using stellar astrometry, we map each pixel to local alt-az coordinates and measure calibration uncertainties of approximately 0.37 deg at zenith and approximately 1.34 deg at 30 deg altitude, sufficient for integration with telescope schedulers. Beyond segmentation, we introduce a short-horizon nowcasting benchmark over per-pixel three-class logits (sky/cloud/contamination) with four baselines: persistence (copying the last frame), optical flow, ConvLSTM, and VideoGPT. ConvLSTM performs best but yields only limited gains over persistence, underscoring the difficulty of near-term cloud evolution. We release the dataset, calibrations, and an open-source toolkit for loading, evaluation, and scheduler-ready alt-az maps to boost research in segmentation, nowcasting, and autonomous observatory operations.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16429
PDF
-
Towards Generalizable Robotic Manipulation in Dynamic Environments
2026-03-16 17:59:57Heng Fang, Shangru Li, Shuhan Wang, Xuanyang Xi, Dingkang Liang, Xiang BaiAbstract
Vision-Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal reasoning capabilities. To address this, we introduce DOMINO, a large-scale dataset and benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15620
PDF
-
$text{F}^2text{HDR}$: Two-Stage HDR Video Reconstruction via Flow Adapter and Physical Motion Modeling
2026-03-16 07:27:17Huanjing Yue, Dawei Li, Shaoxiong Tu, Jingyu YangAbstract
Reconstructing High Dynamic Range (HDR) videos from sequences of alternating-exposure Low Dynamic Range (LDR) frames remains highly challenging, especially under dynamic scenes where cross-exposure inconsistencies and complex motion make inter-frame alignment difficult, leading to ghosting and detail loss. Existing methods often suffer from inaccurate alignment, suboptimal feature aggregation, and degraded reconstruction quality in motion-dominated regions. To address these challenges, we propose $\text{F}^2\text{HDR}$, a two-stage HDR video reconstruction framework that robustly perceives inter-frame motion and restores fine details in complex dynamic scenarios. The proposed framework integrates a flow adapter that adapts generic optical flow for robust cross-exposure alignment, a physical motion modeling to identify salient motion regions, and a motion-aware refinement network that aggregates complementary information while removing ghosting and noise. Extensive experiments demonstrate that $\text{F}^2\text{HDR}$ achieves state-of-the-art performance on real-world HDR video benchmarks, producing ghost-free and high-fidelity results under large motion and exposure variations.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14920
PDF
-
Zero-Shot Reconstruction of Animatable 3D Avatars with Cloth Dynamics from a Single Image
2026-03-16 03:12:27Joohyun Kwon, Geonhee Sim, Gyeongsik MoonAbstract
Existing single-image 3D human avatar methods primarily rely on rigid joint transformations, limiting their ability to model realistic cloth dynamics. We present DynaAvatar, a zero-shot framework that reconstructs animatable 3D human avatars with motion-dependent cloth dynamics from a single image. Trained on large-scale multi-person motion datasets, DynaAvatar employs a Transformer-based feed-forward architecture that directly predicts dynamic 3D Gaussian deformations without subject-specific optimization. To overcome the scarcity of dynamic captures, we introduce a static-to-dynamic knowledge transfer strategy: a Transformer pretrained on large-scale static captures provides strong geometric and appearance priors, which are efficiently adapted to motion-dependent deformations through lightweight LoRA fine-tuning on dynamic captures. We further propose the DynaFlow loss, an optical flow-guided objective that provides reliable motion-direction geometric cues for cloth dynamics in rendered space. Finally, we reannotate the missing or noisy SMPL-X fittings in existing dynamic capture datasets, as most public dynamic capture datasets contain incomplete or unreliable fittings that are unsuitable for training high-quality 3D avatar reconstruction models. Experiments demonstrate that DynaAvatar produces visually rich and generalizable animations, outperforming prior methods.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14772
PDF
-
RetimeGS: Continuous-Time Reconstruction of 4D Gaussian Splatting
2026-03-14 06:26:27Xuezhen Wang, Li Ma, Yulin Shen, Zeyu Wang, Pedro V. SanderAbstract
Temporal retiming, the ability to reconstruct and render dynamic scenes at arbitrary timestamps, is crucial for applications such as slow-motion playback, temporal editing, and post-production. However, most existing 4D Gaussian Splatting (4DGS) methods overfit at discrete frame indices but struggle to represent continuous-time frames, leading to ghosting artifacts when interpolating between timestamps. We identify this limitation as a form of temporal aliasing and propose RetimeGS, a simple yet effective 4DGS representation that explicitly defines the temporal behavior of the 3D Gaussian and mitigates temporal aliasing. To achieve smooth and consistent interpolation, we incorporate optical flow-guided initialization and supervision, triple-rendering supervision, and other targeted strategies. Together, these components enable ghost-free, temporally coherent rendering even under large motions. Experiments on datasets featuring fast motion, non-rigid deformation, and severe occlusions demonstrate that RetimeGS achieves superior quality and coherence over state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13783
PDF
-
Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
2026-03-12 15:31:32Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthaland, Martin MagnussonAbstract
We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras -- a setting that arises naturally when multiple observers capture a shared event. Prior approaches either handle only single-camera input or require rigidly mounted, pre-calibrated camera rigs, limiting their practical applicability. We propose a two-stage optimization framework that decouples the task into robust camera tracking and dense depth refinement. In the first stage, we extend single-camera visual SLAM to the multi-camera setting by constructing a spatiotemporal connection graph that exploits both intra-camera temporal continuity and inter-camera spatial overlap, enabling consistent scale and robust tracking. To ensure robustness under limited overlap, we introduce a wide-baseline initialization strategy using feed-forward reconstruction models. In the second stage, we refine depth and camera poses by optimizing dense inter- and intra-camera consistency using wide-baseline optical flow. Additionally, we introduce MultiCamRobolab, a new real-world dataset with ground-truth poses from a motion capture system. Finally, we demonstrate that our method significantly outperforms state-of-the-art feed-forward models on both synthetic and real-world benchmarks, while requiring less memory.
Abstract (translated)
我们解决了从多个自由移动的摄像机重建密集动态场景和估计相机姿态这一难题——这种情况自然地出现在多名观察者捕捉共享事件时。之前的解决方案要么只能处理单一摄像头输入，要么需要固定安装且预先校准好的多摄系统，这限制了它们的实际应用性。为此，我们提出了一种两阶段优化框架，将任务分解为稳健的摄像机跟踪和密集深度细化两个部分。 在第一阶段，我们将单摄视觉SLAM（同步定位与地图构建）扩展到多摄设置中，通过构建一个时空连接图来利用每个摄像头内的时间连续性和跨摄像头的空间重叠，从而使尺度一致且跟踪更加稳定。为了确保在有限重叠的情况下也能保持稳健性，我们引入了一种使用前馈重建模型的宽基线初始化策略。 第二阶段，我们通过对多摄和单摄内部的一致性进行优化来细化深度信息和摄像机姿态，具体而言就是通过宽基线光学流来实现密集的跨摄一致性优化。此外，为了验证我们的方法有效性，我们还引入了MultiCamRobolab，这是一个新的真实世界数据集，包含来自动作捕捉系统的地面真值位置。 最后，我们在合成和实际世界的基准测试中展示了我们的方法显著优于最新的前馈模型，并且在内存需求上也更为节省。
URL
https://arxiv.org/abs/2603.12064
PDF
