Weakly_Supervised
-
Tighnari v2: Mitigating Label Noise and Distribution Shift in Multimodal Plant Distribution Prediction via Mixture of Experts and Weakly Supervised Learning
2026-02-09 05:23:22Haixu Liu, Yufei Wang, Tianxiang Xu, Chuancheng Shi, Hongsheng XingAbstract
Large-scale, cross-species plant distribution prediction plays a crucial role in biodiversity conservation, yet modeling efforts in this area still face significant challenges due to the sparsity and bias of observational data. Presence-Absence (PA) data provide accurate and noise-free labels, but are costly to obtain and limited in quantity; Presence-Only (PO) data, by contrast, offer broad spatial coverage and rich spatiotemporal distribution, but suffer from severe label noise in negative samples. To address these real-world constraints, this paper proposes a multimodal fusion framework that fully leverages the strengths of both PA and PO data. We introduce an innovative pseudo-label aggregation strategy for PO data based on the geographic coverage of satellite imagery, enabling geographic alignment between the label space and remote sensing feature space. In terms of model architecture, we adopt Swin Transformer Base as the backbone for satellite imagery, utilize the TabM network for tabular feature extraction, retain the Temporal Swin Transformer for time-series modeling, and employ a stackable serial tri-modal cross-attention mechanism to optimize the fusion of heterogeneous modalities. Furthermore, empirical analysis reveals significant geographic distribution shifts between PA training and test samples, and models trained by directly mixing PO and PA data tend to experience performance degradation due to label noise in PO data. To address this, we draw on the mixture-of-experts paradigm: test samples are partitioned according to their spatial proximity to PA samples, and different models trained on distinct datasets are used for inference and post-processing within each partition. Experiments on the GeoLifeCLEF 2025 dataset demonstrate that our approach achieves superior predictive performance in scenarios with limited PA coverage and pronounced distribution shifts.
Abstract (translated)
大规模、跨物种的植物分布预测在生物多样性保护中起着关键作用，但在这一领域建模工作仍面临观测数据稀疏和偏差等重大挑战。存在-不存在（PA）数据提供准确且无噪声的标签，但获取成本高昂且数量有限；仅存在（PO）数据则提供了广泛的空间覆盖范围以及丰富的时间空间分布信息，但却因负样本中严重的标签噪声而受到影响。 为了解决这些实际限制，本文提出了一种多模态融合框架，该框架充分利用了PA和PO数据各自的优点。我们引入了一种基于卫星图像地理覆盖的创新伪标签聚合策略，从而实现标签空间与遥感特征空间之间的地理位置对齐。在模型架构方面，我们采用Swin Transformer Base作为卫星图像的基础网络，并使用TabM网络进行表格特征提取，保留时间序列建模中的Temporal Swin Transformer，同时利用可堆叠串行三模式交叉注意机制来优化异构模式的融合。 此外，实证分析显示PA训练和测试样本之间存在显著的地理分布偏移，直接混合PO和PA数据训练模型往往会因为PO数据中标签噪声而导致性能下降。为了应对这一问题，我们借鉴了专家混合范式：根据其与PA样本的空间接近程度对测试样本进行划分，并在每个分区使用不同数据集上训练的不同模型来进行推理和后处理。 通过GeoLifeCLEF 2025数据集的实验表明，我们的方法在PA覆盖范围有限且分布偏移显著的情况下达到了优越的预测性能。
URL
https://arxiv.org/abs/2602.08282
PDF
-
Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks
2026-02-08 17:02:55Yufei Wang, Haixu Liu, Tianxiang Xu, Chuancheng Shi, Hongsheng XingAbstract
To tackle the automatic recognition of "concealed emotions" in videos, this paper proposes a multimodal weak-supervision framework and achieves state-of-the-art results on the iMiGUE tennis-interview dataset. First, YOLO 11x detects and crops human portraits frame-by-frame, and DINOv2-Base extracts visual features from the cropped regions. Next, by integrating Chain-of-Thought and Reflection prompting (CoT + Reflection), Gemini 2.5 Pro automatically generates pseudo-labels and reasoning texts that serve as weak supervision for downstream models. Subsequently, OpenPose produces 137-dimensional key-point sequences, augmented with inter-frame offset features; the usual graph neural network backbone is simplified to an MLP to efficiently model the spatiotemporal relationships of the three key-point streams. An ultra-long-sequence Transformer independently encodes both the image and key-point sequences, and their representations are concatenated with BERT-encoded interview transcripts. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set for further gains. Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts accuracy from under 0.6 in prior work to over 0.69, establishing a new public benchmark. The study also validates that an "MLP-ified" key-point backbone can match - or even surpass - GCN-based counterparts in this task.
Abstract (translated)
为了应对视频中“隐蔽情绪”的自动识别问题，本文提出了一种多模态弱监督框架，并在iMiGUE网球访谈数据集上取得了最先进的成果。首先，YOLO 11x每帧检测并裁剪人物肖像，DINOv2-Base从裁剪区域提取视觉特征。接着，通过整合Chain-of-Thought和反思提示（CoT + Reflection），Gemini 2.5 Pro自动生成伪标签和推理文本，这些作为下游模型的弱监督信号。随后，OpenPose生成137维关键点序列，并加入帧间偏移特征；通常用于建模时空关系的关键点流图神经网络骨干被简化为MLP（多层感知机），以更高效地处理这一任务。一个超长序列Transformer独立编码图像和关键点序列，然后将它们的表示与BERT编码后的访谈记录拼接起来。每种模式首先单独预训练，之后共同微调，并将伪标签样本合并到训练集中以进一步提升性能。实验表明，在严重类不平衡的情况下，所提出的方法使准确性从先前研究中的不足0.6提高到了超过0.69的新水平，确立了一个新的公共基准标准。此外，该研究表明，“MLP化”的关键点骨干网络可以与基于GCN（图卷积网络）的同类方法相匹敌甚至超越。
URL
https://arxiv.org/abs/2602.08057
PDF
-
PANC: Prior-Aware Normalized Cut for Object Segmentation
2026-02-06 18:07:20Juan Guti\'errez, Victor Guti\'errez-Garcia, Jos\'e Luis Blanco-MurilloAbstract
Fully unsupervised segmentation pipelines naively seek the most salient object, should this be present. As a result, most of the methods reported in the literature deliver non-deterministic partitions that are sensitive to initialization, seed order, and threshold heuristics. We propose PANC, a weakly supervised spectral segmentation framework that uses a minimal set of annotated visual tokens to produce stable, controllable, and reproducible object masks. From the TokenCut approach, we augment the token-token affinity graph with a handful of priors coupled to anchor nodes. By manipulating the graph topology, we bias the spectral eigenspace toward partitions that are consistent with the annotations. Our approach preserves the global grouping enforced by dense self-supervised visual features, trading annotated tokens for significant gains in reproducibility, user control, and segmentation quality. Using 5 to 30 annotations per dataset, our training-free method achieves state-of-the-art performance among weakly and unsupervised approaches on standard benchmarks (e.g., DUTS-TE, ECSSD, MS COCO). Contrarily, it excels in domains where dense labels are costly or intra-class differences are subtle. We report strong and reliable results on homogeneous, fine-grained, and texture-limited domains, achieving 96.8% (+14.43% over SotA), 78.0% (+0.2%), and 78.8% (+0.37%) average mean intersection-over-union (mIoU) on CrackForest (CFD), CUB-200-2011, and HAM10000 datasets, respectively. For multi-object benchmarks, the framework showcases explicit, user-controllable semantic segmentation.
Abstract (translated)
完全无监督的分割流程通常会寻找最显著的对象，前提是该对象存在。因此，文献中报告的大多数方法提供了非确定性的分区结果，这些结果对初始化、种子顺序和阈值启发式非常敏感。我们提出了一种弱监督光谱分割框架PANC，它使用一组最小化的注释视觉标记来生成稳定、可控且可重复的对象掩码。从TokenCut方法出发，我们在令牌-令牌亲和图中添加了一些与锚节点耦合的先验知识。通过调整图的拓扑结构，我们偏向于让光谱特征空间倾向于那些与标注一致的分区结果。我们的方法保持了由密集自监督视觉特征强制执行的整体分组效果，并以注释标记为代价换取可重复性、用户控制和分割质量方面的显著提升。 在标准基准测试（如DUTS-TE、ECSSD、MS COCO）上，使用每个数据集5到30个标注的情况下，我们的无训练方法在弱监督与完全无监督的方法中达到了最佳性能。相反，在密集标签成本高或类内差异微妙的领域，该方法表现出色。我们在同质性、细粒度和纹理有限的领域报告了强大且可靠的结果，分别实现了CrackForest (CFD)上96.8%（比最先进方法提高14.43%），CUB-200-2011上78.0%（提高0.2%）以及HAM10000数据集上的78.8%（提高0.37%）的平均交并比（mIoU）。对于多对象基准测试，该框架展示了用户可控的语义分割。
URL
https://arxiv.org/abs/2602.06912
PDF
-
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
2026-02-05 08:55:26Budhaditya Mukhopadhyay, Chirag Mandal, Pavan Tummala, Naghmeh Mahmoodian, Andreas N\"urnberger, Soumick ChatterjeeAbstract
Liver tumour ablation presents a significant clinical challenge: whilst tumours are clearly visible on pre-operative MRI, they are often effectively invisible on intra-operative CT due to minimal contrast between pathological and healthy tissue. This work investigates the feasibility of cross-modality weak supervision for scenarios where pathology is visible in one modality (MRI) but absent in another (CT). We present a hybrid registration-segmentation framework that combines MSCGUNet for inter-modal image registration with a UNet-based segmentation module, enabling registration-assisted pseudo-label generation for CT images. Our evaluation on the CHAOS dataset demonstrates that the pipeline can successfully register and segment healthy liver anatomy, achieving a Dice score of 0.72. However, when applied to clinical data containing tumours, performance degrades substantially (Dice score of 0.16), revealing the fundamental limitations of current registration methods when the target pathology lacks corresponding visual features in the target modality. We analyse the "domain gap" and "feature absence" problems, demonstrating that whilst spatial propagation of labels via registration is feasible for visible structures, segmenting truly invisible pathology remains an open challenge. Our findings highlight that registration-based label transfer cannot compensate for the absence of discriminative features in the target modality, providing important insights for future research in cross-modality medical image analysis. Code an weights are available at: this https URL
Abstract (translated)
肝脏肿瘤消融在临床上是一个重大的挑战：尽管术前MRI图像上可以清晰地看到肿瘤，但在术中CT图像上由于病理组织和健康组织之间的对比度极低，这些肿瘤往往变得几乎不可见。这项研究探讨了跨模式弱监督方法的可行性，在这种方法中，一种成像方式（如MRI）能显示病理性结构，而另一种成像方式（如CT）则无法显示。我们提出了一种结合MSCGUNet进行跨模态图像配准与基于UNet的分割模块的混合框架，该框架能够生成注册辅助伪标签以供CT图像使用。 我们在CHAOS数据集上的评估表明，该流程可以成功地对健康肝脏解剖结构进行配准和分割，达到Dice分数为0.72的成绩。然而，在包含肿瘤的实际临床数据中应用时，性能显著下降（Dice分数为0.16），这揭示了现有注册方法在目标病理性特征缺乏相应视觉特性时的固有限制。 我们分析了“领域差距”和“特征缺失”的问题，表明尽管通过配准可以在可视结构间传播标签，但分割真正不可见的病理仍然是一个开放性的挑战。我们的研究结果强调，基于注册的标签转移不能弥补目标模态中缺乏区分性特征的问题，并为未来跨模式医学图像分析的研究提供了重要的见解。 代码和权重可在以下网址获得：[链接]（此处应插入具体URL）
URL
https://arxiv.org/abs/2602.05453
PDF
-
A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction
2026-02-04 08:38:55Marco Martinelli, Stefano Marchesin, Vanessa Bonato, Giorgio Maria Di Nunzio, Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani, Gianmaria SilvelloAbstract
Information Extraction (IE), encompassing Named Entity Recognition (NER), Named Entity Linking (NEL), and Relation Extraction (RE), is critical for transforming the rapidly growing volume of scientific publications into structured, actionable knowledge. This need is especially evident in fast-evolving biomedical fields such as the gut-brain axis, where research investigates complex interactions between the gut microbiota and brain-related disorders. Existing biomedical IE benchmarks, however, are often narrow in scope and rely heavily on distantly supervised or automatically generated annotations, limiting their utility for advancing robust IE methods. We introduce GutBrainIE, a benchmark based on more than 1,600 PubMed abstracts, manually annotated by biomedical and terminological experts with fine-grained entities, concept-level links, and relations. While grounded in the gut-brain axis, the benchmark's rich schema, multiple tasks, and combination of highly curated and weakly supervised data make it broadly applicable to the development and evaluation of biomedical IE systems across domains.
Abstract (translated)
信息抽取（IE）包括命名实体识别（NER）、命名实体链接（NEL）和关系提取（RE），对于将快速增长的科学出版物转化为结构化、可操作的知识至关重要。特别是在快速发展的生物医学领域，如肠-脑轴，研究复杂地探讨了肠道微生物群与大脑相关疾病之间的相互作用，这种需求尤为明显。然而，现有的生物医学IE基准测试通常范围狭窄，并且严重依赖于远程监督或自动产生的注释，这限制了它们在推进稳健IE方法方面的实用性。 我们引入了GutBrainIE，这是一个基于超过1,600篇PubMed摘要的基准测试，这些摘要由生物医学和术语学专家手动标注，包含了细粒度实体、概念级链接以及关系。虽然该基准测试建立在肠-脑轴的基础上，但其丰富的架构、多项任务设置及高度策划与弱监督数据的结合，使其能够广泛应用于跨领域生物医学IE系统的开发与评估。
URL
https://arxiv.org/abs/2602.04320
PDF
-
SPWOOD: Sparse Partial Weakly-Supervised Oriented Object Detection
2026-02-03 15:21:01Wei Zhang, Xiang Liu, Ningjing Liu, Mingxin Liu, Wei Liao, Chunyan Xu, Xue YangAbstract
A consistent trend throughout the research of oriented object detection has been the pursuit of maintaining comparable performance with fewer and weaker annotations. This is particularly crucial in the remote sensing domain, where the dense object distribution and a wide variety of categories contribute to prohibitively high costs. Based on the supervision level, existing oriented object detection algorithms can be broadly grouped into fully supervised, semi-supervised, and weakly supervised methods. Within the scope of this work, we further categorize them to include sparsely supervised and partially weakly-supervised methods. To address the challenges of large-scale labeling, we introduce the first Sparse Partial Weakly-Supervised Oriented Object Detection framework, designed to efficiently leverage only a few sparse weakly-labeled data and plenty of unlabeled data. Our framework incorporates three key innovations: (1) We design a Sparse-annotation-Orientation-and-Scale-aware Student (SOS-Student) model to separate unlabeled objects from the background in a sparsely-labeled setting, and learn orientation and scale information from orientation-agnostic or scale-agnostic weak annotations. (2) We construct a novel Multi-level Pseudo-label Filtering strategy that leverages the distribution of model predictions, which is informed by the model's multi-layer predictions. (3) We propose a unique sparse partitioning approach, ensuring equal treatment for each category. Extensive experiments on the DOTA and DIOR datasets show that our framework achieves a significant performance gain over traditional oriented object detection methods mentioned above, offering a highly cost-effective solution. Our code is publicly available at this https URL.
Abstract (translated)
在整个面向对象检测的研究中，一个持续的趋势是追求在使用较少和较弱标注的情况下保持相当的性能。这在遥感领域尤为重要，因为密集的对象分布和众多类别导致了高昂的成本。根据监督水平，现有的面向对象检测算法可以大致分为全监督、半监督和弱监督方法。在此工作中，我们进一步将它们分类为稀疏监督和部分弱监督方法。为了应对大规模标注的挑战，我们引入了第一个稀疏部分弱监督面向对象检测框架，旨在仅利用少量稀疏弱标记数据和大量未标记数据来高效运行。 我们的框架包含三个关键创新： 1. 我们设计了一种稀疏注释方向尺度感知的学生模型（SOS-学生模型），在稀疏标注的情况下将未标注的对象从背景中分离出来，并且能够从无方向性或无尺度性的弱标签中学习到方向和尺度信息。 2. 我们构建了一个新颖的多级伪标签过滤策略，该策略利用了模型预测分布的信息，这些信息来源于模型多层次的预测结果。 3. 我们提出了一种独特的稀疏分区方法，确保每个类别都能得到平等对待。 在DOTA和DIOR数据集上的大量实验表明，我们的框架比上述传统的面向对象检测方法获得了显著的性能提升，并提供了高度成本效益的解决方案。我们公开提供代码：[此链接]（请在此处插入实际的URL）。
URL
https://arxiv.org/abs/2602.03634
PDF
-
Finding Optimal Video Moment without Training: Gaussian Boundary Optimization for Weakly Supervised Video Grounding
2026-02-03 04:01:12Sunoh Kim, Kimin Yun, Daeho UmAbstract
Weakly supervised temporal video grounding aims to localize query-relevant segments in untrimmed videos using only video-sentence pairs, without requiring ground-truth segment annotations that specify exact temporal boundaries. Recent approaches tackle this task by utilizing Gaussian-based temporal proposals to represent query-relevant segments. However, their inference strategies rely on heuristic mappings from Gaussian parameters to segment boundaries, resulting in suboptimal localization performance. To address this issue, we propose Gaussian Boundary Optimization (GBO), a novel inference framework that predicts segment boundaries by solving a principled optimization problem that balances proposal coverage and segment compactness. We derive a closed-form solution for this problem and rigorously analyze the optimality conditions under varying penalty regimes. Beyond its theoretical foundations, GBO offers several practical advantages: it is training-free and compatible with both single-Gaussian and mixture-based proposal architectures. Our experiments show that GBO significantly improves localization, achieving state-of-the-art results across standard benchmarks. Extensive experiments demonstrate the efficiency and generalizability of GBO across various proposal schemes. The code is available at \href{this https URL}{this https URL}.
Abstract (translated)
弱监督视频时间定位的目标是在未裁剪的视频中仅使用视频-句子对来定位与查询相关的片段，而不需依赖明确指定具体时间边界的地面真实标注。最近的研究方法通过利用基于高斯分布的时间提案来表示与查询相关的片段。然而，这些方法的推理策略依赖于从高斯参数到段落边界之间的启发式映射，这导致了次优的时间定位性能。为了解决这个问题，我们提出了高斯边界优化（GBO），这是一种新的推断框架，它通过解决一个平衡提案覆盖范围和片段紧实度的原则性优化问题来预测片段边界。我们为该问题导出了闭合形式的解决方案，并严格分析了在不同惩罚制度下的最优条件。 除了其理论基础外，GBO还提供了许多实用优势：它是无须训练且与单个高斯和混合型提案架构兼容的。我们的实验表明，GBO显著提高了定位性能，在标准基准测试中取得了最先进的结果。广泛的实验证明了GBO在各种提案方案中的效率和泛化能力。代码可在[此处](this https URL)获取。 原文的意思是介绍了一种新的弱监督视频时间定位方法——高斯边界优化（Gaussian Boundary Optimization, GBO），这种方法通过解决一个平衡提案覆盖范围与片段紧实度的原则性优化问题，提高了定位的准确性，并且无须额外训练。这项技术在标准基准测试中取得了领先的成绩，并展示了其在多种提案方案中的高效性和泛化能力。
URL
https://arxiv.org/abs/2602.03071
PDF
-
CIEC: Coupling Implicit and Explicit Cues for Multimodal Weakly Supervised Manipulation Localization
2026-02-02 14:46:38Xinquan Yu, Wei Lu, Xiangyang LuoAbstract
To mitigate the threat of misinformation, multimodal manipulation localization has garnered growing attention. Consider that current methods rely on costly and time-consuming fine-grained annotations, such as patch/token-level annotations. This paper proposes a novel framework named Coupling Implicit and Explicit Cues (CIEC), which aims to achieve multimodal weakly-supervised manipulation localization for image-text pairs utilizing only coarse-grained image/sentence-level annotations. It comprises two branches, image-based and text-based weakly-supervised localization. For the former, we devise the Textual-guidance Refine Patch Selection (TRPS) module. It integrates forgery cues from both visual and textual perspectives to lock onto suspicious regions aided by spatial priors. Followed by the background silencing and spatial contrast constraints to suppress interference from irrelevant areas. For the latter, we devise the Visual-deviation Calibrated Token Grounding (VCTG) module. It focuses on meaningful content words and leverages relative visual bias to assist token localization. Followed by the asymmetric sparse and semantic consistency constraints to mitigate label noise and ensure reliability. Extensive experiments demonstrate the effectiveness of our CIEC, yielding results comparable to fully supervised methods on several evaluation metrics.
Abstract (translated)
URL
https://arxiv.org/abs/2602.02175
PDF
-
WS-IMUBench: Can Weakly Supervised Methods from Audio, Image, and Video Be Adapted for IMU-based Temporal Action Localization?
2026-02-02 09:22:35Pei Li, Jiaxi Yin, Lei Ouyang, Shihan Pan, Ge Wang, Han Ding, Fei WangAbstract
IMU-based Human Activity Recognition (HAR) has enabled a wide range of ubiquitous computing applications, yet its dominant clip classification paradigm cannot capture the rich temporal structure of real-world behaviors. This motivates a shift toward IMU Temporal Action Localization (IMU-TAL), which predicts both action categories and their start/end times in continuous streams. However, current progress is strongly bottlenecked by the need for dense, frame-level boundary annotations, which are costly and difficult to scale. To address this bottleneck, we introduce WS-IMUBench, a systematic benchmark study of weakly supervised IMU-TAL (WS-IMU-TAL) under only sequence-level labels. Rather than proposing a new localization algorithm, we evaluate how well established weakly supervised localization paradigms from audio, image, and video transfer to IMU-TAL under only sequence-level labels. We benchmark seven representative weakly supervised methods on seven public IMU datasets, resulting in over 3,540 model training runs and 7,080 inference evaluations. Guided by three research questions on transferability, effectiveness, and insights, our findings show that (i) transfer is modality-dependent, with temporal-domain methods generally more stable than image-derived proposal-based approaches; (ii) weak supervision can be competitive on favorable datasets (e.g., with longer actions and higher-dimensional sensing); and (iii) dominant failure modes arise from short actions, temporal ambiguity, and proposal quality. Finally, we outline concrete directions for advancing WS-IMU-TAL (e.g., IMU-specific proposal generation, boundary-aware objectives, and stronger temporal reasoning). Beyond individual results, WS-IMUBench establishes a reproducible benchmarking template, datasets, protocols, and analyses, to accelerate community-wide progress toward scalable WS-IMU-TAL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01850
PDF
-
Offline Discovery of Interpretable Skills from Multi-Task Trajectories
2026-02-01 05:03:58Chongyu Zhu, Mithun Vanniasinghe, Jiayu Chen, Chi-Guhn LeeAbstract
Hierarchical Imitation Learning is a powerful paradigm for acquiring complex robot behaviors from demonstrations. A central challenge, however, lies in discovering reusable skills from long-horizon, multi-task offline data, especially when the data lacks explicit rewards or subtask annotations. In this work, we introduce LOKI, a three-stage end-to-end learning framework designed for offline skill discovery and hierarchical imitation. The framework commences with a two-stage, weakly supervised skill discovery process: Stage one performs coarse, task-aware macro-segmentation by employing an alignment-enforced Vector Quantized VAE guided by weak task labels. Stage two then refines these segments at a micro-level using a self-supervised sequential model, followed by an iterative clustering process to consolidate skill boundaries. The third stage then leverages these precise boundaries to construct a hierarchical policy within an option-based framework-complete with a learned termination condition beta for explicit skill switching. LOKI achieves high success rates on the challenging D4RL Kitchen benchmark and outperforms standard HIL baselines. Furthermore, we demonstrate that the discovered skills are semantically meaningful, aligning with human intuition, and exhibit compositionality by successfully sequencing them to solve a novel, unseen task.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01018
PDF
-
Fire on Motion: Optimizing Video Pass-bands for Efficient Spiking Action Recognition
2026-01-30 07:47:23Shuhan Ye, Yuanbin Qian, Yi Yu, Chong Wang, Yuqi Xie, Jiazhen Xu, Kun Wang, Xudong JiangAbstract
Spiking neural networks (SNNs) have gained traction in vision due to their energy efficiency, bio-plausibility, and inherent temporal processing. Yet, despite this temporal capacity, most progress concentrates on static image benchmarks, and SNNs still underperform on dynamic video tasks compared to artificial neural networks (ANNs). In this work, we diagnose a fundamental pass-band mismatch: Standard spiking dynamics behave as a temporal low pass that emphasizes static content while attenuating motion bearing bands, where task relevant information concentrates in dynamic tasks. This phenomenon explains why SNNs can approach ANNs on static tasks yet fall behind on tasks that demand richer temporal this http URL remedy this, we propose the Pass-Bands Optimizer (PBO), a plug-and-play module that optimizes the temporal pass-band toward task-relevant motion bands. PBO introduces only two learnable parameters, and a lightweight consistency constraint that preserves semantics and boundaries, incurring negligible computational overhead and requires no architectural changes. PBO deliberately suppresses static components that contribute little to discrimination, effectively high passing the stream so that spiking activity concentrates on motion bearing content. On UCF101, PBO yields over ten percentage points improvement. On more complex multi-modal action recognition and weakly supervised video anomaly detection, PBO delivers consistent and significant gains, offering a new perspective for SNN based video processing and understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22675
PDF
-
Mining Forgery Traces from Reconstruction Error: A Weakly Supervised Framework for Multimodal Deepfake Temporal Localization
2026-01-29 09:35:27Midou Guo, Qilin Yin, Wei Lu, Xiangyang Luo, Rui YangAbstract
Modern deepfakes have evolved into localized and intermittent manipulations that require fine-grained temporal localization. The prohibitive cost of frame-level annotation makes weakly supervised methods a practical necessity, which rely only on video-level labels. To this end, we propose Reconstruction-based Temporal Deepfake Localization (RT-DeepLoc), a weakly supervised temporal forgery localization framework that identifies forgeries via reconstruction errors. Our framework uses a Masked Autoencoder (MAE) trained exclusively on authentic data to learn its intrinsic spatiotemporal patterns; this allows the model to produce significant reconstruction discrepancies for forged segments, effectively providing the missing fine-grained cues for localization. To robustly leverage these indicators, we introduce a novel Asymmetric Intra-video Contrastive Loss (AICL). By focusing on the compactness of authentic features guided by these reconstruction cues, AICL establishes a stable decision boundary that enhances local discrimination while preserving generalization to unseen forgeries. Extensive experiments on large-scale datasets, including LAV-DF, demonstrate that RT-DeepLoc achieves state-of-the-art performance in weakly-supervised temporal forgery localization.
Abstract (translated)
URL
https://arxiv.org/abs/2601.21458
PDF
-
Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy
2026-01-28 17:03:43Huanyu Tian, Martin Huber, Lingyun Zeng, Zhe Han, Wayne Bennett, Giuseppe Silvestri, Gerardo Mendizabal-Ruiz, Tom Vercauteren, Alejandro Chavez-Badiola, Christos BergelesAbstract
This paper rethinks steady-hand robotic manipulation by using a weakly supervised framework that fuses calibration-aware perception with admittance control. Unlike conventional automation that relies on labor-intensive 2D labeling, our framework leverages reusable warm-up trajectories to extract implicit spatial information, thereby achieving calibration-aware, depth-resolved perception without the need for external fiducials or manual depth annotation. By explicitly characterizing residuals from observation and calibration models, the system establishes a task-space error budget from recorded warm-ups. The uncertainty budget yields a lateral closed-loop accuracy of approx. 49 micrometers at 95% confidence (worst-case testing subset) and a depth accuracy of <= 291 micrometers at 95% confidence bound during large in-plane moves. In a within-subject user study (N=8), the learned agent reduces overall NASA-TLX workload by 77.1% relative to the simple steady-hand assistance baseline. These results demonstrate that the weakly supervised agent improves the reliability of microscope-guided biomedical micromanipulation without introducing complex setup requirements, offering a practical framework for microscope-guided intervention.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20776
PDF
-
Weakly supervised framework for wildlife detection and counting in challenging Arctic environments: a case study on caribou
2026-01-26 19:02:18Ghazaleh Serati, Samuel Foucher, Jerome TheauAbstract
Caribou across the Arctic has declined in recent decades, motivating scalable and accurate monitoring approaches to guide evidence-based conservation actions and policy decisions. Manual interpretation from this imagery is labor-intensive and error-prone, underscoring the need for automatic and reliable detection across varying scenes. Yet, such automatic detection is challenging due to severe background heterogeneity, dominant empty terrain (class imbalance), small or occluded targets, and wide variation in density and scale. To make the detection model (HerdNet) more robust to these challenges, a weakly supervised patch-level pretraining based on a detection network's architecture is proposed. The detection dataset includes five caribou herds distributed across Alaska. By learning from empty vs. non-empty labels in this dataset, the approach produces early weakly supervised knowledge for enhanced detection compared to HerdNet, which is initialized from generic weights. Accordingly, the patch-based pretrain network attained high accuracy on multi-herd imagery (2017) and on an independent year's (2019) test sets (F1: 93.7%/92.6%, respectively), enabling reliable mapping of regions containing animals to facilitate manual counting on large aerial imagery. Transferred to detection, initialization from weakly supervised pretraining yielded consistent gains over ImageNet weights on both positive patches (F1: 92.6%/93.5% vs. 89.3%/88.6%), and full-image counting (F1: 95.5%/93.3% vs. 91.5%/90.4%). Remaining limitations are false positives from animal-like background clutter and false negatives related to low animal density occlusions. Overall, pretraining on coarse labels prior to detection makes it possible to rely on weakly-supervised pretrained weights even when labeled data are limited, achieving results comparable to generic-weight initialization.
Abstract (translated)
URL
https://arxiv.org/abs/2601.18891
PDF
-
Context Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation
2026-01-21 07:12:23Yiyang Fu, Hui Li, Wangyu WuAbstract
Weakly Supervised Semantic Segmentation (WSSS), which relies only on image-level labels, has attracted significant attention for its cost-effectiveness and scalability. Existing methods mainly enhance inter-class distinctions and employ data augmentation to mitigate semantic ambiguity and reduce spurious activations. However, they often neglect the complex contextual dependencies among image patches, resulting in incomplete local representations and limited segmentation accuracy. To address these issues, we propose the Context Patch Fusion with Class Token Enhancement (CPF-CTE) framework, which exploits contextual relations among patches to enrich feature representations and improve segmentation. At its core, the Contextual-Fusion Bidirectional Long Short-Term Memory (CF-BiLSTM) module captures spatial dependencies between patches and enables bidirectional information flow, yielding a more comprehensive understanding of spatial correlations. This strengthens feature learning and segmentation robustness. Moreover, we introduce learnable class tokens that dynamically encode and refine class-specific semantics, enhancing discriminative capability. By effectively integrating spatial and semantic cues, CPF-CTE produces richer and more accurate representations of image content. Extensive experiments on PASCAL VOC 2012 and MS COCO 2014 validate that CPF-CTE consistently surpasses prior WSSS methods.
Abstract (translated)
弱监督语义分割（WSSS）依赖于图像级别的标签，因其成本效益和可扩展性而引起了广泛关注。现有的方法主要通过增强类间区分度并采用数据增强技术来减轻语义模糊性和减少虚假激活。然而，它们往往忽视了图块之间的复杂上下文依赖关系，导致局部表示不完整且分割精度受限。为解决这些问题，我们提出了带有类别标记增强（CPF-CTE）的上下文图块融合框架，该框架利用图块之间的上下文关系来丰富特征表示并提高分割性能。在核心部分，上下文融合双向长短时记忆模块（Contextual-Fusion BiLSTM, CF-BiLSTM）捕获图块间的空间依赖性，并实现双向信息流动，从而对空间相关性提供更全面的理解。这增强了特征学习和分割的鲁棒性。此外，我们引入了可学习的类别标记，这些标记能够动态编码并细化特定类别的语义信息，提高了判别能力。通过有效整合空间和语义线索，CPF-CTE生成了更加丰富且准确的图像内容表示。 在PASCAL VOC 2012及MS COCO 2014数据集上的大量实验验证表明，CPF-CTE方法始终优于先前的WSSS方法。
URL
https://arxiv.org/abs/2601.14718
PDF
-
Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging
2026-01-13 03:44:06Md. Faiyaz Abdullah Sayeedi, Rashedur Rahman, Siam Tahsin Bhuiyan, Sefatul Wasi, Ashraful Islam, Saadia Binte Alam, AKM Mahbubur RahmanAbstract
Medical image analysis increasingly relies on large vision-language models (VLMs), yet most systems remain single-pass black boxes that offer limited control over reasoning, safety, and spatial grounding. We propose R^4, an agentic framework that decomposes medical imaging workflows into four coordinated agents: a Router that configures task- and specialization-aware prompts from the image, patient history, and metadata; a Retriever that uses exemplar memory and pass@k sampling to jointly generate free-text reports and bounding boxes; a Reflector that critiques each draft-box pair for key clinical error modes (negation, laterality, unsupported claims, contradictions, missing findings, and localization errors); and a Repairer that iteratively revises both narrative and spatial outputs under targeted constraints while curating high-quality exemplars for future cases. Instantiated on chest X-ray analysis with multiple modern VLM backbones and evaluated on report generation and weakly supervised detection, R^4 consistently boosts LLM-as-a-Judge scores by roughly +1.7-+2.5 points and mAP50 by +2.5-+3.5 absolute points over strong single-VLM baselines, without any gradient-based fine-tuning. These results show that agentic routing, reflection, and repair can turn strong but brittle VLMs into more reliable and better grounded tools for clinical image interpretation. Our code can be found at: this https URL
Abstract (translated)
医学图像分析越来越依赖于大型视觉语言模型（VLM），然而，大多数系统仍然是一次性黑盒操作，对推理、安全和空间定位的控制有限。我们提出了R^4框架，这是一个代理框架，将医学成像工作流程分解为四个协调的代理：一个路由器，根据从图像、患者历史记录和元数据中提取的任务和专业信息来配置提示；一个检索器，使用范例记忆和pass@k抽样同时生成自由文本报告和边界框；一个反思者，批评每对草案-边框的关键临床错误模式（否定、侧向性、缺乏支持的主张、矛盾、遗漏发现以及定位错误）；以及一个修复者，在有针对性的约束下迭代修订叙述性和空间输出的同时为未来案例策划高质量范例。在胸部X光片分析中使用多个现代VLM后端，并通过报告生成和弱监督检测进行评估，R^4 在不进行任何基于梯度的微调的情况下，相对于强大的单一VLM基线，显著提高了LLM作为裁判（LLM-as-a-Judge）评分约1.7至2.5分，并将mAP50绝对提高了2.5至3.5个点。这些结果表明，代理路由、反思和修复可以将强大但脆弱的VLM转变为更可靠且更好地定位的临床图像解释工具。我们的代码可在以下链接找到：[此URL]
URL
https://arxiv.org/abs/2601.08192
PDF
-
Kidney Cancer Detection Using 3D-Based Latent Diffusion Models
2026-01-09 15:30:00Jen Dusseljee, Sarah de Boer, Alessa HeringAbstract
In this work, we present a novel latent diffusion-based pipeline for 3D kidney anomaly detection on contrast-enhanced abdominal CT. The method combines Denoising Diffusion Probabilistic Models (DDPMs), Denoising Diffusion Implicit Models (DDIMs), and Vector-Quantized Generative Adversarial Networks (VQ-GANs). Unlike prior slice-wise approaches, our method operates directly on an image volume and leverages weak supervision with only case-level pseudo-labels. We benchmark our approach against state-of-the-art supervised segmentation and detection models. This study demonstrates the feasibility and promise of 3D latent diffusion for weakly supervised anomaly detection. While the current results do not yet match supervised baselines, they reveal key directions for improving reconstruction fidelity and lesion localization. Our findings provide an important step toward annotation-efficient, generative modeling of complex abdominal anatomy.
Abstract (translated)
在这项工作中，我们提出了一种新颖的基于潜在扩散的管道，用于在增强腹部CT图像上进行3D肾脏异常检测。该方法结合了去噪扩散概率模型（DDPMs）、去噪扩散隐式模型（DDIMs）和向量量化生成对抗网络（VQ-GANs）。与之前的逐片处理方法不同，我们的方法直接针对整个图像体积进行操作，并利用仅基于病例级别的伪标签的弱监督。我们将该方法与最先进的有监督分割和检测模型进行了基准测试。这项研究表明了在弱监督下使用3D潜在扩散技术进行异常检测的可能性和前景。尽管目前的结果尚未达到有监督基线的标准，但它们揭示了提高重建准确性和病灶定位的关键方向。我们的研究为高效注释、复杂腹部解剖结构的生成建模提供了一个重要的步骤。
URL
https://arxiv.org/abs/2601.05852
PDF
-
SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization
2026-01-09 07:25:55Xinghao Wang, Changtao Miao, Dianmo Sheng, Tao Gong, Qi Chu, Nenghai Yu, Quanchen Zou, Deyue Zhang, Xiangzheng ZhangAbstract
Malicious image manipulation threatens public safety and requires efficient localization methods. Existing approaches depend on costly pixel-level annotations which make training expensive. Existing weakly supervised methods rely only on image-level binary labels and focus on global classification, often overlooking local edge cues that are critical for precise localization. We observe that feature variations at manipulated boundaries are substantially larger than in interior regions. To address this gap, we propose Semantic-Agnostic Prompt Learning (SAPL) in CLIP, which learns text prompts that intentionally encode non-semantic, boundary-centric cues so that CLIPs multimodal similarity highlights manipulation edges rather than high-level object semantics. SAPL combines two complementary modules Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL) to exploit edge information in both textual and visual spaces. The proposed ECPL leverages edge-enhanced image features to generate learnable textual prompts via an attention mechanism, embedding semantic-irrelevant information into text features, to guide CLIP focusing on manipulation edges. The proposed HECL extract genuine and manipulated edge patches, and utilize contrastive learning to boost the discrimination between genuine edge patches and manipulated edge patches. Finally, we predict the manipulated regions from the similarity map after processing. Extensive experiments on multiple public benchmarks demonstrate that SAPL significantly outperforms existing approaches, achieving state-of-the-art localization performance.
Abstract (translated)
恶意图像篡改威胁到公共安全，需要高效的定位方法。现有方法依赖于昂贵的像素级标注，使得训练成本高昂。现有的弱监督方法仅依靠图级别二进制标签，并专注于全局分类，往往忽视了局部边缘线索对于精确定位至关重要的作用。我们观察到，在被篡改边界上的特征变化远大于内部区域的变化。为解决这一问题，我们在CLIP模型中提出了语义不可知提示学习（SAPL），该方法通过有意编码非语义的边界中心线索来学习文本提示，使CLIP的多模态相似性更能突出篡改边缘而不是高层次的对象语义。SAPL结合了两个互补模块：边缘感知上下文提示学习（ECPL）和分层边缘对比学习（HECL），用于在文本空间和视觉空间中利用边缘信息。 所提出的ECPL通过增强图像特征的边缘，生成可学习的文字提示，并借助注意力机制将语义无关的信息嵌入到文字特征中，引导CLIP关注篡改边缘。而所提出的HECL则提取真实边界片和被篡改边界片，并利用对比学习来提高对真实边界片与被篡改边界片之间的区分能力。 最后，在处理完相似性地图后，我们预测出被篡改的区域。在多个公开基准上的广泛实验表明，SAPL显著优于现有方法，实现了最先进的定位性能。
URL
https://arxiv.org/abs/2601.06222
PDF
-
PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understanding
2026-01-06 11:11:06I\~naki Erregue, Kamal Nasrollahi, Sergio EscaleraAbstract
Video Anomaly Understanding (VAU) extends traditional Video Anomaly Detection (VAD) by not only localizing anomalies but also describing and reasoning about their context. Existing VAU approaches often rely on fine-tuned multimodal large language models (MLLMs) or external modules such as video captioners, which introduce costly annotations, complex training pipelines, and high inference overhead. In this work, we introduce PrismVAU, a lightweight yet effective system for real-time VAU that leverages a single off-the-shelf MLLM for anomaly scoring, explanation, and prompt optimization. PrismVAU operates in two complementary stages: (1) a coarse anomaly scoring module that computes frame-level anomaly scores via similarity to textual anchors, and (2) an MLLM-based refinement module that contextualizes anomalies through system and user prompts. Both textual anchors and prompts are optimized with a weakly supervised Automatic Prompt Engineering (APE) framework. Extensive experiments on standard VAD benchmarks demonstrate that PrismVAU delivers competitive detection performance and interpretable anomaly explanations -- without relying on instruction tuning, frame-level annotations, and external modules or dense processing -- making it an efficient and practical solution for real-world applications.
Abstract (translated)
视频异常理解（Video Anomaly Understanding，简称VAU）在传统的视频异常检测（Video Anomaly Detection，简称VAD）基础上更进一步，不仅定位了异常事件，还描述并解释了这些异常的上下文。现有的VAU方法往往依赖于微调过的多模态大规模语言模型（Multimodal Large Language Models，简称MLLMs）或外部模块如视频字幕生成器，这引入了昂贵的数据标注、复杂的训练流程和高推理成本。 在此项工作中，我们提出了PrismVAU，这是一个轻量级且高效的系统，适用于实时的VAU任务。该系统利用单一现成的多模态大规模语言模型（MLLM）来完成异常评分、解释以及提示优化。PrismVAU通过两个互补阶段实现其功能：(1) 一个粗略的异常评分模块，它计算基于与文本锚点相似度的帧级异常分数；(2) 一个基于多模态大规模语言模型（MLLM）的细化模块，该模块通过系统和用户提示来上下文化异常。这些文本锚点和提示都经过弱监督自动提示工程（Automatic Prompt Engineering, APE）框架进行优化。 在标准VAD基准上的广泛实验表明，PrismVAU能够提供竞争力强的检测性能以及可解释性的异常解释——并且无需依赖指令调优、帧级标注或外部模块及密集处理流程。这使得PrismVAU成为实际应用中一个高效且实用的解决方案。
URL
https://arxiv.org/abs/2601.02927
PDF
-
HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures
2026-01-05 13:32:37Yating Wang, Yuan Sun, Xuan Wang, Ran Yi, Boyao Zhou, Yipengjing Sun, Hongyu Liu, Yinuo Wang, Lizhuang MaAbstract
Recent 3D-aware head generative models based on 3D Gaussian Splatting achieve real-time, photorealistic and view-consistent head synthesis. However, a fundamental limitation persists: the deep entanglement of illumination and intrinsic appearance prevents controllable relighting. Existing disentanglement methods rely on strong assumptions to enable weakly supervised learning, which restricts their capacity for complex illumination. To address this challenge, we introduce HeadLighter, a novel supervised framework that learns a physically plausible decomposition of appearance and illumination in head generative models. Specifically, we design a dual-branch architecture that separately models lighting-invariant head attributes and physically grounded rendering components. A progressive disentanglement training is employed to gradually inject head appearance priors into the generative architecture, supervised by multi-view images captured under controlled light conditions with a light stage setup. We further introduce a distillation strategy to generate high-quality normals for realistic rendering. Experiments demonstrate that our method preserves high-quality generation and real-time rendering, while simultaneously supporting explicit lighting and viewpoint editing. We will publicly release our code and dataset.
Abstract (translated)
近期基于3D高斯点阵的三维感知头部生成模型实现了实时、逼真且视角一致的头部合成。然而，这些模型仍存在一个基本限制：光照与内在外观之间的深度纠缠阻碍了可控重光照的能力。现有的脱耦方法依赖于强烈的假设来实现弱监督学习，这限制了它们处理复杂照明场景的能力。 为了应对这一挑战，我们引入了一种名为HeadLighter的新框架，该框架在头部生成模型中学习了一个物理上合理的外观和光照分解。具体来说，我们设计了一种双分支架构，分别对光照不变的头部属性和基于物理原理的渲染组件进行建模。通过使用多视角图像（这些图像是在具有灯光舞台设置的受控照明条件下捕获的）监督下逐步注入头部外观先验，采用渐进式脱耦训练方法。 此外，我们还引入了一种蒸馏策略来生成高质量的法线图以支持逼真的渲染。实验结果表明，我们的方法在保持高质量生成和实时渲染的同时，也能够支持明确的光照编辑和视角调整。我们将公开发布代码和数据集。
URL
https://arxiv.org/abs/2601.02103
PDF
