Weakly_Supervised
-
VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagation
2025-08-09 04:34:54Ayaan Nooruddin Siddiqui, Mahnoor Zaidi, Ayesha Nazneen Shahbaz, Priyadarshini Chatterjee, Krishnan Menon IyerAbstract
Accurate segmentation of subcutaneous vessels from clinical images is hampered by scarce, expensive ground truth and by low contrast, noisy appearance of vessels across patients and modalities. We present a novel weakly supervised training framework tailored for subcutaneous vessel segmentation that leverages inexpensive sparse annotations (e.g., centerline traces, dot markers, or short scribbles). Sparse labels are expanded into dense, probabilistic supervision via a differentiable random walk label propagation model whose transition weights incorporate image driven vesselness cues and tubular continuity priors. The propagation yields per-pixel hitting probabilities together with calibrated uncertainty estimates; these are incorporated into an uncertainty weighted loss to avoid over fitting to ambiguous regions. Crucially, the label-propagator is learned jointly with a CNN based segmentation predictor, enabling the system to discover vessel edges and continuity constraints without explicit edge supervision. We further introduce a topology aware regularizer that encourages centerline connectivity and penalizes spurious branches, improving clinical usability. In experiments on clinical subcutaneous imaging datasets, our method consistently outperforms naive training on sparse labels and conventional dense pseudo-labeling, producing more complete vascular maps and better calibrated uncertainty for downstream decision making. The approach substantially reduces annotation burden while preserving clinically relevant vessel topology.
Abstract (translated)
从临床图像中精确分割皮下血管受到稀缺且昂贵的真值标注和患者间及模态间的低对比度、噪声外观的影响。我们提出了一种新颖的弱监督训练框架，专门用于皮下血管分割，该框架利用廉价稀疏注释（例如，中心线轨迹、点标记或简短涂鸦）。这些稀疏标签通过一种可微分随机游走标签传播模型扩展为密集且概率化的监督信号，该模型在传输权重中整合了图像驱动的血管特征和管状连续性先验。这种传播产生了每个像素的命中概率以及校准过的不确定性估计；这些信息被纳入一个不确定性加权损失函数中以避免过度拟合到模棱两可区域。关键在于标签传播器与基于 CNN 的分割预测器共同学习，从而使系统能够发现血管边缘和连续性约束而无需明确的边缘监督。此外，我们引入了一种拓扑感知正则化器，鼓励中心线连通并惩罚虚假分支，从而提高临床实用性。在对临床皮下成像数据集进行实验时，我们的方法始终优于基于稀疏标签的简单训练以及传统的密集伪标注，生成了更完整的血管图，并且为下游决策提供了更好的校准不确定性估计。该方法大幅减少了注释负担，同时保留了临床上相关的血管拓扑结构。
URL
https://arxiv.org/abs/2508.06819
PDF
-
WGAST: Weakly-Supervised Generative Network for Daily 10 m Land Surface Temperature Estimation via Spatio-Temporal Fusion
2025-08-08 17:49:46Sofiane Bouaziz, Adel Hafiane, Raphael Canals, Rachid NedjaiAbstract
Urbanization, climate change, and agricultural stress are increasing the demand for precise and timely environmental monitoring. Land Surface Temperature (LST) is a key variable in this context and is retrieved from remote sensing satellites. However, these systems face a trade-off between spatial and temporal resolution. While spatio-temporal fusion methods offer promising solutions, few have addressed the estimation of daily LST at 10 m resolution. In this study, we present WGAST, a Weakly-Supervised Generative Network for Daily 10 m LST Estimation via Spatio-Temporal Fusion of Terra MODIS, Landsat 8, and Sentinel-2. WGAST is the first end-to-end deep learning framework designed for this task. It adopts a conditional generative adversarial architecture, with a generator composed of four stages: feature extraction, fusion, LST reconstruction, and noise suppression. The first stage employs a set of encoders to extract multi-level latent representations from the inputs, which are then fused in the second stage using cosine similarity, normalization, and temporal attention mechanisms. The third stage decodes the fused features into high-resolution LST, followed by a Gaussian filter to suppress high-frequency noise. Training follows a weakly supervised strategy based on physical averaging principles and reinforced by a PatchGAN discriminator. Experiments demonstrate that WGAST outperforms existing methods in both quantitative and qualitative evaluations. Compared to the best-performing baseline, on average, WGAST reduces RMSE by 17.18% and improves SSIM by 11.00%. Furthermore, WGAST is robust to cloud-induced LST and effectively captures fine-scale thermal patterns, as validated against 33 ground-based sensors. The code is available at this https URL.
Abstract (translated)
城市化、气候变化和农业压力正在增加对精确且及时的环境监测需求。地表温度（Land Surface Temperature，LST）是这一背景下的关键变量，并通过遥感卫星获取数据。然而，这些系统在空间分辨率与时间分辨率之间存在权衡。尽管时空融合方法提供了有前景的解决方案，但鲜少有关于如何估算每日10米分辨率LST的研究。在这项研究中，我们提出了WGAST——一种基于Terra MODIS、Landsat 8和Sentinel-2卫星数据时空融合的弱监督生成网络，用于估计每日10米分辨率的地表温度。WGAST是第一个为这一任务设计的端到端深度学习框架。它采用了一种条件生成对抗架构，其中生成器包含四个阶段：特征提取、融合、LST重建和噪声抑制。第一阶段利用一组编码器从输入中提取多层潜在表示，然后在第二阶段使用余弦相似度、归一化以及时间注意力机制进行融合。第三阶段将融合的特征解码为高分辨率LST，并通过高斯滤波来消除高频噪声。训练采用基于物理平均原理的弱监督策略，并由PatchGAN鉴别器加强。实验表明，WGAST在定量和定性评估中都优于现有方法。相比最佳基准线，在平均情况下，WGAST使RMSE降低了17.18%，SSIM提升了11%。此外，WGAST对云引起的LST具有鲁棒性，并有效捕捉到了细尺度的热模式，通过33个地面传感器进行了验证。代码可在以下网址获取：[提供链接]。
URL
https://arxiv.org/abs/2508.06485
PDF
-
DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition
2025-08-07 17:22:33Haijing Liu, Tao Pu, Hefeng Wu, Keze Wang, Liang LinAbstract
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image, requiring both precise intra-class localization to pinpoint objects and effective inter-class reasoning to model complex category dependencies. While Vision-Language Pre-training (VLP) models offer a strong open-vocabulary foundation, they often struggle with fine-grained localization under weak supervision and typically fail to explicitly leverage structured relational knowledge beyond basic semantics, limiting performance especially for unseen classes. To overcome these limitations, we propose the Dual Adaptive Refinement Transfer (DART) framework. DART enhances a frozen VLP backbone via two synergistic adaptive modules. For intra-class refinement, an Adaptive Refinement Module (ARM) refines patch features adaptively, coupled with a novel Weakly Supervised Patch Selecting (WPS) loss that enables discriminative localization using only image-level labels. Concurrently, for inter-class transfer, an Adaptive Transfer Module (ATM) leverages a Class Relationship Graph (CRG), constructed using structured knowledge mined from a Large Language Model (LLM), and employs graph attention network to adaptively transfer relational information between class representations. DART is the first framework, to our knowledge, to explicitly integrate external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision for OV-MLR. Extensive experiments on challenging benchmarks demonstrate that our DART achieves new state-of-the-art performance, validating its effectiveness.
Abstract (translated)
开放词汇多标签识别（OV-MLR）的目标是在图像中识别多种已知和未知的对象类别，这需要精确的类内定位以确定对象的位置，并有效地进行跨类推理以建模复杂的类别依赖关系。虽然视觉语言预训练（VLP）模型提供了强大的开放词汇基础，但它们在弱监督下的细粒度定位方面常常遇到困难，并且通常无法显式地利用超出基本语义的结构化关联知识，这限制了其在未见过类别的表现能力。 为了克服这些局限性，我们提出了双自适应细化传输（DART）框架。DART通过两个协同作用的自适应模块增强了冻结的VLP骨干网络。对于类内精炼，自适应精炼模块（ARM）能够根据图像级别的标签自适应地精炼块特征，并且结合了一种新颖的弱监督块选择损失（WPS），仅使用图像级别标签即可实现有区分性的定位。与此同时，在跨类别迁移方面，自适应传输模块（ATM）利用了一个基于大型语言模型（LLM）挖掘出的结构化知识构建的类关系图（CRG），并通过图形注意力网络来适应性地在类表示之间传递关联信息。 据我们所知，DART是第一个显式集成外部由LLM产生的关系知识进行自适应跨类别迁移的同时，在弱监督条件下执行自适应内类别精炼以实现OV-MLR的框架。在具有挑战性的基准测试中进行的一系列实验表明，我们的DART达到了新的最先进性能水平，验证了其有效性。 这个介绍展示了DART框架如何通过结合视觉语言预训练模型和大型语言模型的知识来改进开放词汇多标签识别任务中的定位精度和类别关系理解能力。
URL
https://arxiv.org/abs/2508.05585
PDF
-
Skin-SOAP: A Weakly Supervised Framework for Generating Structured SOAP Notes
2025-08-07 04:12:43Sadia Kamal, Tim Oates, Joy WanAbstract
Skin carcinoma is the most prevalent form of cancer globally, accounting for over $8 billion in annual healthcare expenditures. Early diagnosis, accurate and timely treatment are critical to improving patient survival rates. In clinical settings, physicians document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, manually generating these notes is labor-intensive and contributes to clinician burnout. In this work, we propose skin-SOAP, a weakly supervised multimodal framework to generate clinically structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Our approach reduces reliance on manual annotations, enabling scalable, clinically grounded documentation while alleviating clinician burden and reducing the need for large annotated data. Our method achieves performance comparable to GPT-4o, Claude, and DeepSeek Janus Pro across key clinical relevance metrics. To evaluate this clinical relevance, we introduce two novel metrics MedConceptEval and Clinical Coherence Score (CCS) which assess semantic alignment with expert medical concepts and input features, respectively.
Abstract (translated)
皮肤癌是全球最常见的癌症形式，每年的医疗保健支出超过80亿美元。早期诊断、准确及时的治疗对于提高患者生存率至关重要。在临床环境中，医生通过详细的SOAP（主诉、客观检查、评估和计划）记录来记录患者的就诊情况。然而，手动生成这些记录既费时又增加了医务人员的工作负担，导致职业倦怠。为此，我们提出了skin-SOAP，这是一种弱监督多模态框架，可以从有限的输入（包括病变图像和稀疏的临床文本）生成结构化的SOAP记录。我们的方法减少了对人工注释的依赖，使文档制作更具规模性和临床实用性，同时减轻了医务人员的工作负担，并减少了对大规模标注数据的需求。我们提出的方法在关键临床相关性指标上达到了与GPT-4o、Claude和DeepSeek Janus Pro相当的性能水平。为了评估其临床相关性，我们引入了两个新的评价指标：MedConceptEval 和临床连贯性评分（CCS），分别用于评估语义与专家医学概念及输入特征的一致性。
URL
https://arxiv.org/abs/2508.05019
PDF
-
TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring
2025-08-07 00:17:45Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang liuAbstract
Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1)Relation-aware knowledge mining: we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at this https URL.
Abstract (translated)
动态场景图生成（DSGG）的目标是通过检测物体并预测它们之间的关系，为视频中的每一帧创建一个场景图。弱监督的动态场景图生成（WS-DSGG）方法通过使用来自每个视频单个帧的未定位的场景图来减少标注工作量。现有的WS-DSGG方法依赖于现成的外部对象检测器来生成用于后续DSGG训练的伪标签。然而，基于静态、以物体为中心的图像进行训练的对象检测器在动态关系感知场景下表现不佳，导致定位不准确和低置信度建议。 为了解决WS-DSGG中外部对象检测器带来的挑战，我们提出了一种时间增强的关系感知知识迁移（TRKT）方法，该方法利用知识来提高在关系感知动态场景中的检测性能。TRKT建立在两个关键组件上： 1. **关系感知的知识挖掘**：首先使用物体和关系类解码器生成特定于类别的注意图以突出显示对象区域及其互动区。然后我们提出了一种跨帧注意力增强策略，利用光流来提高相邻帧中的注意图，使其能够感知运动，并且在面对运动模糊时更为鲁棒。这一步产生了适用于WS-DSGG的关系和运动感知的知识挖掘。 2. **双流融合模块**：此模块将特定于类别的注意图集成到外部检测结果中，以细化物体定位并提高对象建议的置信度得分。 广泛的实验表明，TRKT在Action Genome数据集上达到了最先进的性能。我们的代码可以在这里访问（原文中的URL链接）。
URL
https://arxiv.org/abs/2508.04943
PDF
-
JanusNet: Hierarchical Slice-Block Shuffle and Displacement for Semi-Supervised 3D Multi-Organ Segmentation
2025-08-06 01:08:02Zheng Zhang, Tianzhuzi Tan, Guanchun Yin, Bo Zhang, Xiuzhuang ZhouAbstract
Limited by the scarcity of training samples and annotations, weakly supervised medical image segmentation often employs data augmentation to increase data diversity, while randomly mixing volumetric blocks has demonstrated strong performance. However, this approach disrupts the inherent anatomical continuity of 3D medical images along orthogonal axes, leading to severe structural inconsistencies and insufficient training in challenging regions, such as small-sized organs, etc. To better comply with and utilize human anatomical information, we propose JanusNet}, a data augmentation framework for 3D medical data that globally models anatomical continuity while locally focusing on hard-to-segment regions. Specifically, our Slice-Block Shuffle step performs aligned shuffling of same-index slice blocks across volumes along a random axis, while preserving the anatomical context on planes perpendicular to the perturbation axis. Concurrently, the Confidence-Guided Displacement step uses prediction reliability to replace blocks within each slice, amplifying signals from difficult areas. This dual-stage, axis-aligned framework is plug-and-play, requiring minimal code changes for most teacher-student schemes. Extensive experiments on the Synapse and AMOS datasets demonstrate that JanusNet significantly surpasses state-of-the-art methods, achieving, for instance, a 4% DSC gain on the Synapse dataset with only 20% labeled data.
Abstract (translated)
由于训练样本和标注的稀缺性，弱监督下的医学图像分割通常采用数据增强来增加数据多样性。随机混合体积块的方法已经展示出了强大的性能，但这种方法会破坏3D医学影像在正交轴上的内在解剖连续性，导致严重的结构不一致，并且对于如小型器官等具有挑战性的区域训练不足。为了更好地符合和利用人体解剖信息，我们提出了JanusNet，这是一个针对3D医疗数据的数据增强框架，它全局建模解剖连续性，同时局部关注难以分割的区域。 具体来说，我们的Slice-Block Shuffle步骤沿随机轴对不同体积中的相同索引切片块进行对齐打乱操作，并且保持了与扰动轴垂直的方向上的解剖背景。同时，Confidence-Guided Displacement步骤使用预测可靠性来替换每个切片内的区块，从而放大来自难以分割区域的信号。 这种双阶段、沿轴线的数据增强框架是即插即用的，对于大多数师生方案几乎不需要代码改动。在Synapse和AMOS数据集上的广泛实验表明，JanusNet显著超越了现有的最佳方法，在只有20%标注数据的情况下，例如在Synapse数据集中达到了4%的DSC增益。
URL
https://arxiv.org/abs/2508.03997
PDF
-
AlignCAT: Visual-Linguistic Alignment of Category and Attributefor Weakly Supervised Visual Grounding
2025-08-05 08:16:35Yidan Wang, Chenyi Zhuang, Wutao Liu, Pan Gao, Nicu SebeAbstract
Weakly supervised visual grounding (VG) aims to locate objects in images based on text descriptions. Despite significant progress, existing methods lack strong cross-modal reasoning to distinguish subtle semantic differences in text expressions due to category-based and attribute-based ambiguity. To address these challenges, we introduce AlignCAT, a novel query-based semantic matching framework for weakly supervised VG. To enhance visual-linguistic alignment, we propose a coarse-grained alignment module that utilizes category information and global context, effectively mitigating interference from category-inconsistent objects. Subsequently, a fine-grained alignment module leverages descriptive information and captures word-level text features to achieve attribute consistency. By exploiting linguistic cues to their fullest extent, our proposed AlignCAT progressively filters out misaligned visual queries and enhances contrastive learning efficiency. Extensive experiments on three VG benchmarks, namely RefCOCO, RefCOCO+, and RefCOCOg, verify the superiority of AlignCAT against existing weakly supervised methods on two VG tasks. Our code is available at: this https URL.
Abstract (translated)
弱监督视觉接地（VG）旨在根据文本描述在图像中定位对象。尽管已经取得了显著进展，现有的方法仍缺乏强大的跨模态推理能力，难以区分基于类别和属性的文本表达中的细微语义差异。为了解决这些挑战，我们提出了AlignCAT，这是一种新颖的查询基语义匹配框架，用于弱监督视觉接地（VG）。 为了增强视觉-语言对齐，我们提出了一种粗粒度对齐模块，该模块利用类别信息和全局上下文，有效减轻了来自类不一致对象的干扰。随后，细粒度对齐模块通过利用描述性信息并捕捉单词级别的文本特征来实现属性一致性。通过充分利用语言线索，我们的AlignCAT逐步过滤掉不对齐的视觉查询，并提高对比学习效率。 在三个VG基准数据集（RefCOCO、RefCOCO+ 和 RefCOCOg）上进行的大量实验验证了AlignCAT在两项VG任务中优于现有的弱监督方法。我们的代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2508.03201
PDF
-
RefineSeg: Dual Coarse-to-Fine Learning for Medical Image Segmentation
2025-08-04 19:14:30Anghong Du, Nay Aung, Theodoros N. Arvanitis, Stefan K. Piechnik, Joao A C Lima, Steffen E. Petersen, Le ZhangAbstract
High-quality pixel-level annotations of medical images are essential for supervised segmentation tasks, but obtaining such annotations is costly and requires medical expertise. To address this challenge, we propose a novel coarse-to-fine segmentation framework that relies entirely on coarse-level annotations, encompassing both target and complementary drawings, despite their inherent noise. The framework works by introducing transition matrices in order to model the inaccurate and incomplete regions in the coarse annotations. By jointly training on multiple sets of coarse annotations, it progressively refines the network's outputs and infers the true segmentation distribution, achieving a robust approximation of precise labels through matrix-based modeling. To validate the flexibility and effectiveness of the proposed method, we demonstrate the results on two public cardiac imaging datasets, ACDC and MSCMRseg, and further evaluate its performance on the UK Biobank dataset. Experimental results indicate that our approach surpasses the state-of-the-art weakly supervised methods and closely matches the fully supervised approach.
Abstract (translated)
高质量的像素级医学图像标注对于监督分割任务至关重要，但获取这样的标注既耗时又需要医疗专业知识。为了解决这一挑战，我们提出了一种新颖的粗到细分割框架，该框架完全依赖于粗略级别的标注（包括目标和补充绘制），即使这些标注中存在固有的噪声。此框架通过引入转换矩阵来建模粗略注释中的不准确和不完整区域。通过在多个粗略注释集上联合训练，它逐步细化网络输出，并推断出真正的分割分布，从而通过基于矩阵的模型实现对精确标签的稳健近似。 为了验证所提出方法的灵活性和有效性，我们在两个公开的心脏成像数据集ACDC和MSCMRseg上展示了结果，并进一步在UK Biobank数据集中评估了其性能。实验结果显示，我们的方法超过了现有的弱监督方法，在某些指标上接近于完全监督的方法。
URL
https://arxiv.org/abs/2508.02844
PDF
-
Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
2025-08-04 08:22:39Wenbo Xu, Wei Lu, Xiangyang LuoAbstract
The spread of Deepfake videos has caused a trust crisis and impaired social stability. Although numerous approaches have been proposed to address the challenges of Deepfake detection and localization, there is still a lack of systematic research on the weakly supervised multimodal fine-grained temporal forgery localization (WS-MTFL). In this paper, we propose a novel weakly supervised multimodal temporal forgery localization via multitask learning (WMMT), which addresses the WS-MTFL under the multitask learning paradigm. WMMT achieves multimodal fine-grained Deepfake detection and temporal partial forgery localization using merely video-level annotations. Specifically, visual and audio modality detection are formulated as two binary classification tasks. The multitask learning paradigm is introduced to integrate these tasks into a multimodal task. Furthermore, WMMT utilizes a Mixture-of-Experts structure to adaptively select appropriate features and localization head, achieving excellent flexibility and localization precision in WS-MTFL. A feature enhancement module with temporal property preserving attention mechanism is proposed to identify the intra- and inter-modality feature deviation and construct comprehensive video features. To further explore the temporal information for weakly supervised learning, an extensible deviation perceiving loss has been proposed, which aims to enlarge the deviation of adjacent segments of the forged samples and reduce the deviation of genuine samples. Extensive experiments demonstrate the effectiveness of multitask learning for WS-MTFL, and the WMMT achieves comparable results to fully supervised approaches in several evaluation metrics.
Abstract (translated)
深度伪造视频的传播已经导致了信任危机，并影响到了社会稳定性。尽管已提出了许多方法来解决深度伪造检测和定位所面临的挑战，但弱监督下的多模态精细时间伪造定位（WS-MTFL）的研究仍存在系统性不足的问题。本文提出了一种基于多任务学习的新颖弱监督多模态时序伪造定位方法（WMMT），该方法在多任务学习范式下解决了WS-MTFL问题。WMMT仅通过视频级标注就能实现多模态精细深度伪造检测和时间局部伪造的定位。具体来说，视觉和音频模式检测被构造成两个二分类任务。引入了多任务学习框架将这些任务整合成一个多模态任务。此外，WMMT采用了混合专家（Mixture-of-Experts）结构，能够自适应地选择合适的特征及定位头，在WS-MTFL中实现了出色的灵活性和定位精度。为了识别出同一模式内的特征偏差以及跨模式间的特征差异，并构建全面的视频特征，本文提出了一种具有时间特性保持注意机制的功能增强模块。为进一步探索弱监督学习中的时序信息，引入了一种可扩展的偏差感知损失函数，其目标是扩大伪造样本相邻段落之间的偏差并减小真实样本的偏差。 大量的实验表明，多任务学习对于WS-MTFL的有效性，并且WMMT在多个评价指标中实现了与完全监督方法相当的结果。
URL
https://arxiv.org/abs/2508.02179
PDF
-
Weakly-Supervised Image Forgery Localization via Vision-Language Collaborative Reasoning Framework
2025-08-02 12:14:29Ziqi Sheng, Junyan Wu, Wei Lu, Jiantao ZhouAbstract
Image forgery localization aims to precisely identify tampered regions within images, but it commonly depends on costly pixel-level annotations. To alleviate this annotation burden, weakly supervised image forgery localization (WSIFL) has emerged, yet existing methods still achieve limited localization performance as they mainly exploit intra-image consistency clues and lack external semantic guidance to compensate for weak supervision. In this paper, we propose ViLaCo, a vision-language collaborative reasoning framework that introduces auxiliary semantic supervision distilled from pre-trained vision-language models (VLMs), enabling accurate pixel-level localization using only image-level labels. Specifically, ViLaCo first incorporates semantic knowledge through a vision-language feature modeling network, which jointly extracts textual and visual priors using pre-trained VLMs. Next, an adaptive vision-language reasoning network aligns textual semantics and visual features through mutual interactions, producing semantically aligned representations. Subsequently, these representations are passed into dual prediction heads, where the coarse head performs image-level classification and the fine head generates pixel-level localization masks, thereby bridging the gap between weak supervision and fine-grained localization. Moreover, a contrastive patch consistency module is introduced to cluster tampered features while separating authentic ones, facilitating more reliable forgery discrimination. Extensive experiments on multiple public datasets demonstrate that ViLaCo substantially outperforms existing WSIFL methods, achieving state-of-the-art performance in both detection and localization accuracy.
Abstract (translated)
图像伪造定位旨在精确识别图片中的篡改区域，但通常依赖于昂贵的像素级标注。为减轻这种标注负担，弱监督下的图像伪造定位（WSIFL）应运而生，然而现有方法仍因主要利用内部一致性线索并缺乏外部语义指导来弥补弱监督的不足，从而导致有限的定位性能。本文中我们提出了一种视觉-语言协作推理框架ViLaCo，它引入了从预训练的视觉-语言模型（VLM）蒸馏出的辅助语义监督，仅使用图像级标签即可实现准确的像素级定位。 具体来说，ViLaCo首先通过视觉-语言特征建模网络整合语义知识，该网络利用预先训练好的VLM联合提取文本和视觉先验。接下来，自适应视觉-语言推理网络通过相互作用对齐文本语义与视觉特征，生成语义一致的表示。随后，这些表示被传递到双预测头中：粗略头部执行图像级分类而精细头部产生像素级定位掩模，从而弥合了弱监督和细粒度定位之间的差距。此外，引入了一种对比式补丁一致性模块来聚类篡改特征并分离真实特征，以便更可靠地进行伪造鉴别。 在多个公共数据集上的广泛实验表明，ViLaCo显著优于现有的WSIFL方法，在检测和定位精度方面均达到了最先进的性能。
URL
https://arxiv.org/abs/2508.01338
PDF
-
DisFaceRep: Representation Disentanglement for Co-occurring Facial Components in Weakly Supervised Face Parsing
2025-08-02 08:02:06Xiaoqin Wang, Xianxu Hou, Meidan Ding, Junliang Chen, Kaijun Deng, Jinheng Xie, Linlin ShenAbstract
Face parsing aims to segment facial images into key components such as eyes, lips, and eyebrows. While existing methods rely on dense pixel-level annotations, such annotations are expensive and labor-intensive to obtain. To reduce annotation cost, we introduce Weakly Supervised Face Parsing (WSFP), a new task setting that performs dense facial component segmentation using only weak supervision, such as image-level labels and natural language descriptions. WSFP introduces unique challenges due to the high co-occurrence and visual similarity of facial components, which lead to ambiguous activations and degraded parsing performance. To address this, we propose DisFaceRep, a representation disentanglement framework designed to separate co-occurring facial components through both explicit and implicit mechanisms. Specifically, we introduce a co-occurring component disentanglement strategy to explicitly reduce dataset-level bias, and a text-guided component disentanglement loss to guide component separation using language supervision implicitly. Extensive experiments on CelebAMask-HQ, LaPa, and Helen demonstrate the difficulty of WSFP and the effectiveness of DisFaceRep, which significantly outperforms existing weakly supervised semantic segmentation methods. The code will be released at \href{this https URL}{\textcolor{cyan}{this https URL}}.
Abstract (translated)
面部解析的目标是将面部图像分割成关键组成部分，如眼睛、嘴唇和眉毛。现有的方法依赖于密集的像素级注释，但这些注释获取起来成本高昂且费时费力。为了降低标注成本，我们提出了弱监督面部解析（WSFP），这是一种新任务设定，它仅使用诸如图形单标签和自然语言描述之类的弱监督信息来进行稠密的面部组件分割。由于面部组件之间的高度共现性和视觉相似性，WSFP带来了独特的挑战，这会导致模糊激活并降低解析性能。 为了解决这些问题，我们提出了DisFaceRep框架，该框架设计用于通过显式和隐式的机制来分离共存的面部组件。具体来说，我们引入了一种共发生组件解缠策略以减少数据集级别的偏差，并提出了一种文本引导组件解缠损失函数，使用语言监督隐式地指导组件分离。在CelebAMask-HQ、LaPa和Helen等数据集上的大量实验表明了WSFP的难度以及DisFaceRep的有效性，该方法显著优于现有的弱监督语义分割方法。 代码将在[此处](this https URL)发布。
URL
https://arxiv.org/abs/2508.01250
PDF
-
Tobler's First Law in GeoAI: A Spatially Explicit Deep Learning Model for Terrain Feature Detection Under Weak Supervision
2025-08-01 21:47:50Wenwen Li, Chia-Yu Hsu, Maosheng HuAbstract
Recent interest in geospatial artificial intelligence (GeoAI) has fostered a wide range of applications using artificial intelligence (AI), especially deep learning, for geospatial problem solving. However, major challenges such as a lack of training data and the neglect of spatial principles and spatial effects in AI model design remain, significantly hindering the in-depth integration of AI with geospatial research. This paper reports our work in developing a deep learning model that enables object detection, particularly of natural features, in a weakly supervised manner. Our work makes three contributions: First, we present a method of object detection using only weak labels. This is achieved by developing a spatially explicit model based on Tobler's first law of geography. Second, we incorporate attention maps into the object detection pipeline and develop a multistage training strategy to improve performance. Third, we apply this model to detect impact craters on Mars, a task that previously required extensive manual effort. The model generalizes to both natural and human-made features on the surfaces of Earth and other planets. This research advances the theoretical and methodological foundations of GeoAI.
Abstract (translated)
最近对地理空间人工智能（GeoAI）的兴趣推动了广泛的应用，这些应用利用人工智能（AI），尤其是深度学习技术来解决地理空间问题。然而，诸如训练数据不足以及在AI模型设计中忽视空间原理和空间效应等重大挑战仍然存在，这严重阻碍了AI与地理研究的深度融合。本文报告了我们开发的一种深度学习模型的工作成果，该模型能够在弱监督条件下实现目标检测，特别是自然特征的目标检测。我们的工作主要有三个贡献： 首先，我们提出了一种仅使用弱标签进行对象检测的方法。这一方法通过基于托布勒（Tobler）地理学第一定律的空间显式模型得以实现。 其次，我们将注意图引入到物体检测管道中，并开发出一种多阶段训练策略以提高性能。 第三，我们将该模型应用于火星撞击坑的检测任务上，这一任务之前需要大量的手动工作。我们的模型可以推广到地球及其他行星表面自然和人造特征的目标检测任务中。 这项研究推进了GeoAI在理论和方法论基础方面的进展。
URL
https://arxiv.org/abs/2508.03745
PDF
-
Weakly Supervised Virus Capsid Detection with Image-Level Annotations in Electron Microscopy Images
2025-08-01 12:02:25Hannah Kniesel, Leon Sick, Tristan Payer, Tim Bergner, Kavitha Shaga Devan, Clarissa Read, Paul Walther, Timo RopinskiAbstract
Current state-of-the-art methods for object detection rely on annotated bounding boxes of large data sets for training. However, obtaining such annotations is expensive and can require up to hundreds of hours of manual labor. This poses a challenge, especially since such annotations can only be provided by experts, as they require knowledge about the scientific domain. To tackle this challenge, we propose a domain-specific weakly supervised object detection algorithm that only relies on image-level annotations, which are significantly easier to acquire. Our method distills the knowledge of a pre-trained model, on the task of predicting the presence or absence of a virus in an image, to obtain a set of pseudo-labels that can be used to later train a state-of-the-art object detection model. To do so, we use an optimization approach with a shrinking receptive field to extract virus particles directly without specific network architectures. Through a set of extensive studies, we show how the proposed pseudo-labels are easier to obtain, and, more importantly, are able to outperform other existing weak labeling methods, and even ground truth labels, in cases where the time to obtain the annotation is limited.
Abstract (translated)
当前最先进的物体检测方法依赖于大型数据集上的标注边界框进行训练。然而，获取这些注释非常昂贵，并且可能需要数百小时的人工劳动。这一挑战尤其严重，因为这类注释只能由具备特定科学领域知识的专家提供。为了解决这个问题，我们提出了一种基于图像级标签的领域特定弱监督物体检测算法，这种标签更容易获得。我们的方法通过一个预训练模型的知识蒸馏过程，在预测图像中是否存在病毒的任务上获取一组伪标签，这些伪标签可以用于后续训练最先进的物体检测模型。为此，我们采用了一个优化方法，并使用了收缩感受野的方法来直接提取病毒颗粒，而无需特定的网络架构。通过对一系列广泛的实验研究显示，所提出的伪标签更容易获得，并且在注释时间有限的情况下，能够优于其他现有的弱标注方法，甚至可以超越真实的地面实况标签。
URL
https://arxiv.org/abs/2508.00563
PDF
-
Weakly Supervised Intracranial Aneurysm Detection and Segmentation in MR angiography via Multi-task UNet with Vesselness Prior
2025-08-01 00:45:46Erin Rainville, Amirhossein Rasoulian, Hassan Rivaz, Yiming XiaoAbstract
Intracranial aneurysms (IAs) are abnormal dilations of cerebral blood vessels that, if ruptured, can lead to life-threatening consequences. However, their small size and soft contrast in radiological scans often make it difficult to perform accurate and efficient detection and morphological analyses, which are critical in the clinical care of the disorder. Furthermore, the lack of large public datasets with voxel-wise expert annotations pose challenges for developing deep learning algorithms to address the issues. Therefore, we proposed a novel weakly supervised 3D multi-task UNet that integrates vesselness priors to jointly perform aneurysm detection and segmentation in time-of-flight MR angiography (TOF-MRA). Specifically, to robustly guide IA detection and segmentation, we employ the popular Frangi's vesselness filter to derive soft cerebrovascular priors for both network input and an attention block to conduct segmentation from the decoder and detection from an auxiliary branch. We train our model on the Lausanne dataset with coarse ground truth segmentation, and evaluate it on the test set with refined labels from the same database. To further assess our model's generalizability, we also validate it externally on the ADAM dataset. Our results demonstrate the superior performance of the proposed technique over the SOTA techniques for aneurysm segmentation (Dice = 0.614, 95%HD =1.38mm) and detection (false positive rate = 1.47, sensitivity = 92.9%).
Abstract (translated)
颅内动脉瘤（IAs）是脑血管的异常扩张，如果破裂，则可能导致生命危险。然而，由于其体积小且在放射学扫描中的对比度较低，准确和高效的检测及形态分析变得颇具挑战性，而这些对于该疾病的临床护理至关重要。此外，缺乏带有像素级别专家注释的大规模公开数据集也给开发深度学习算法以解决这些问题带来了困难。 因此，我们提出了一种新颖的弱监督3D多任务UNet模型，该模型结合了血管优先级信息，能够在时间飞跃磁共振血管成像（TOF-MRA）中同时执行动脉瘤检测和分割。具体来说，为了可靠地指导IA的检测和分割，我们采用了流行的Frangi血管滤波器来为网络输入和注意力模块生成软脑血管先验知识，并通过解码器进行分割，辅助分支进行检测。 我们在Lausanne数据集上使用粗略的地面真实分割对模型进行了训练，并在带有改进标签的数据库测试集中对其进行了评估。为了进一步评估我们模型的泛化能力，还在外部ADAM数据集上对其进行验证。我们的结果显示，所提出的动脉瘤分割技术（Dice = 0.614，95%HD = 1.38mm）和检测技术（假阳性率 = 1.47，灵敏度 = 92.9%）在最先进的方法中表现出了优越的性能。
URL
https://arxiv.org/abs/2508.00235
PDF
-
From Waveforms to Pixels: A Survey on Audio-Visual Segmentation
2025-07-29 22:20:51Jia Li, Yapeng TianAbstract
Audio-Visual Segmentation (AVS) aims to identify and segment sound-producing objects in videos by leveraging both visual and audio modalities. It has emerged as a significant research area in multimodal perception, enabling fine-grained object-level understanding. In this survey, we present a comprehensive overview of the AVS field, covering its problem formulation, benchmark datasets, evaluation metrics, and the progression of methodologies. We analyze a wide range of approaches, including architectures for unimodal and multimodal encoding, key strategies for audio-visual fusion, and various decoder designs. Furthermore, we examine major training paradigms, from fully supervised learning to weakly supervised and training-free methods. Notably, we provide an extensive comparison of AVS methods across standard benchmarks, highlighting the impact of different architectural choices, fusion strategies, and training paradigms on performance. Finally, we outline the current challenges, such as limited temporal modeling, modality bias toward vision, lack of robustness in complex environments, and high computational demands, and propose promising future directions, including improving temporal reasoning and multimodal fusion, leveraging foundation models for better generalization and few-shot learning, reducing reliance on labeled data through selfand weakly supervised learning, and incorporating higher-level reasoning for more intelligent AVS systems.
Abstract (translated)
音频-视觉分割（AVS）旨在通过利用视觉和听觉两种模式来识别并划分视频中产生声音的对象。它已经成为多模态感知领域的一个重要研究方向，能够实现对物体层面的精细理解。在这篇综述中，我们全面概述了AVS领域的现状，涵盖其问题定义、基准数据集、评估指标以及方法论的发展历程。本文分析了一系列广泛的方法，包括单模和跨模编码架构设计，音频-视觉融合的关键策略，以及各种解码器的设计方案。此外，本文还考察了主要的训练范式，从完全监督学习到弱监督及无训练方法。尤为重要的是，我们对标准基准上的AVS方法进行了详尽比较，突出了不同架构选择、融合策略和训练范式对性能的影响。 最后，本文概述了当前面临的主要挑战，如有限的时间建模能力、偏向视觉的模式偏差、复杂环境下的鲁棒性不足以及计算需求高。同时提出了有前景的发展方向，包括提升时间推理与多模态融合的能力，利用基础模型以提高泛化能力和少量样本学习能力，通过自我监督及弱监督学习减少对标注数据的依赖，并引入高级别推理以构建更为智能的AVS系统。
URL
https://arxiv.org/abs/2508.03724
PDF
-
Bridging Synthetic and Real-World Domains: A Human-in-the-Loop Weakly-Supervised Framework for Industrial Toxic Emission Segmentation
2025-07-29 16:53:00Yida Tao, Yen-Chia HsuAbstract
Industrial smoke segmentation is critical for air-quality monitoring and environmental protection but is often hampered by the high cost and scarcity of pixel-level annotations in real-world settings. We introduce CEDANet, a human-in-the-loop, class-aware domain adaptation framework that uniquely integrates weak, citizen-provided video-level labels with adversarial feature alignment. Specifically, we refine pseudo-labels generated by a source-trained segmentation model using citizen votes, and employ class-specific domain discriminators to transfer rich source-domain representations to the industrial domain. Comprehensive experiments on SMOKE5K and custom IJmond datasets demonstrate that CEDANet achieves an F1-score of 0.414 and a smoke-class IoU of 0.261 with citizen feedback, vastly outperforming the baseline model, which scored 0.083 and 0.043 respectively. This represents a five-fold increase in F1-score and a six-fold increase in smoke-class IoU. Notably, CEDANet with citizen-constrained pseudo-labels achieves performance comparable to the same architecture trained on limited 100 fully annotated images with F1-score of 0.418 and IoU of 0.264, demonstrating its ability to reach small-sampled fully supervised-level accuracy without target-domain annotations. Our research validates the scalability and cost-efficiency of combining citizen science with weakly supervised domain adaptation, offering a practical solution for complex, data-scarce environmental monitoring applications.
Abstract (translated)
工业烟雾分割对于空气质量监测和环境保护至关重要，但由于实际场景中像素级标注的成本高昂且稀缺，这一过程往往受到限制。我们提出了CEDANet，这是一个人类参与的、基于类别感知领域的自适应框架，该框架独特地整合了由公民提供的弱视频级别标签与对抗性特征对齐技术。具体来说，我们通过使用公民投票来优化源训练分割模型生成的伪标签，并利用特定类别的领域判别器将丰富的源域表示转移到工业领域。 在SMOKE5K和定制的IJmond数据集上的全面实验表明，CEDANet在有公民反馈的情况下取得了0.414的F1分数和0.261的烟雾类别交并比（IoU），大大超过了基础模型的成绩，该模型分别得分0.083和0.043。这代表了F1分数五倍的增长以及烟雾类别IoU六倍的增长。 值得注意的是，在仅依靠公民约束伪标签的情况下，CEDANet能够达到与在有限的100张完全标注图像上训练的相同架构相似的表现水平，其F1得分为0.418，IoU为0.264。这表明CEDANet能够在不进行目标域注释的情况下，以小样本充分监督级别的精度完成任务。 我们的研究验证了结合公民科学与弱监督领域适应方法在规模和成本效益上的可行性，并提供了一种实用的解决方案来应对复杂且数据稀缺的环境监测应用挑战。
URL
https://arxiv.org/abs/2507.22002
PDF
-
Mitigating Spurious Correlations in Weakly Supervised Semantic Segmentation via Cross-architecture Consistency Regularization
2025-07-29 16:09:15Zheyuan Zhang, Yen-chia HsuAbstract
Scarcity of pixel-level labels is a significant challenge in practical scenarios. In specific domains like industrial smoke, acquiring such detailed annotations is particularly difficult and often requires expert knowledge. To alleviate this, weakly supervised semantic segmentation (WSSS) has emerged as a promising approach. However, due to the supervision gap and inherent bias in models trained with only image level labels, existing WSSS methods suffer from limitations such as incomplete foreground coverage, inaccurate object boundaries, and spurious correlations, especially in our domain, where emissions are always spatially coupled with chimneys. Previous solutions typically rely on additional priors or external knowledge to mitigate these issues, but they often lack scalability and fail to address the model's inherent bias toward co-occurring context. To address this, we propose a novel WSSS framework that directly targets the co-occurrence problem without relying on external supervision. Unlike prior methods that adopt a single network, we employ a teacher-student framework that combines CNNs and ViTs. We introduce a knowledge transfer loss that enforces cross-architecture consistency by aligning internal representations. Additionally, we incorporate post-processing techniques to address partial coverage and further improve pseudo mask quality.
Abstract (translated)
像素级标签的稀缺性在实际场景中是一个重大挑战。特别是在工业烟雾等特定领域，获取这些详细标注尤为困难，并且通常需要专家知识。为了解决这一问题，弱监督语义分割（WSSS）作为一种有前景的方法应运而生。然而，由于仅使用图像级别标签进行训练时模型中存在的监督差距和固有偏差，现有的WSSS方法面临着诸如前景覆盖不全、对象边界不准确以及虚假相关等问题的限制，特别是在我们的领域中，排放总是与烟囱空间上耦合。先前的解决方案通常依赖于额外先验知识或外部信息来缓解这些问题，但它们往往缺乏可扩展性且无法解决模型对共现上下文的固有偏差问题。 为了解决这个问题，我们提出了一种新的WSSS框架，该框架直接针对共现问题，并不依赖于外部监督。不同于以往采用单一网络的方法，我们使用了一个教师-学生框架结合CNN和ViT。我们引入了知识迁移损失，通过对齐内部表示来强制跨架构的一致性。此外，我们还采用了后处理技术以解决部分覆盖问题并进一步提高伪标签掩码的质量。 这种方法旨在克服现有WSSS方法的局限性，并提供一个更具可扩展性和鲁棒性的解决方案，特别适用于工业烟雾等空间共现特征明显的场景。
URL
https://arxiv.org/abs/2507.21959
PDF
-
Emerging Trends in Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation with Image-Level Supervision
2025-07-29 08:41:58Zheyuan Zhang, Wang ZhangAbstract
Unlike fully supervised semantic segmentation, weakly supervised semantic segmentation (WSSS) relies on weaker forms of supervision to perform dense prediction tasks. Among the various types of weak supervision, WSSS with image level annotations is considered both the most challenging and the most practical, attracting significant research attention. Therefore, in this review, we focus on WSSS with image level annotations. Additionally, this review concentrates on mainstream research directions, deliberately omitting less influential branches. Given the rapid development of new methods and the limitations of existing surveys in capturing recent trends, there is a pressing need for an updated and comprehensive review. Our goal is to fill this gap by synthesizing the latest advancements and state-of-the-art techniques in WSSS with image level labels. Basically, we provide a comprehensive review of recent advancements in WSSS with image level labels, categorizing existing methods based on the types and levels of additional supervision involved. We also examine the challenges of applying advanced methods to domain specific datasets in WSSS,a topic that remains underexplored. Finally, we discuss the current challenges, evaluate the limitations of existing approaches, and outline several promising directions for future research. This review is intended for researchers who are already familiar with the fundamental concepts of WSSS and are seeking to deepen their understanding of current advances and methodological innovations.
Abstract (translated)
与完全监督的语义分割相比，弱监督语义分割（WSSS）依赖于较弱形式的监督来执行密集预测任务。在各种类型的弱监督中，带有图像级别注释的WSSS被认为是最具挑战性和最实用的，因此吸引了大量研究关注。因此，在这篇综述中，我们将重点放在带有图像级别注释的WSSS上。此外，该综述专注于主流的研究方向，并有意省略了影响力较小的方向。鉴于新方法迅速发展以及现有调查在捕捉最新趋势方面的局限性，迫切需要更新和全面的回顾。我们的目标是通过综合最新的进展和最先进的技术来填补这一空白，特别是在带有图像级别标签的WSSS领域。 基本上，我们提供了最近在带有图像级别标签的WSSS领域的综合回顾，并根据额外监督的类型和程度对现有方法进行了分类。此外，我们将探讨将高级方法应用于特定领域数据集时所面临的挑战，在该领域中此话题尚未得到充分研究。最后，我们将讨论当前的挑战、评估现有方法的局限性，并概述未来有前景的研究方向。 这篇综述旨在为已经熟悉WSSS基本概念并希望深入了解最新进展和方法创新的研究人员提供帮助。
URL
https://arxiv.org/abs/2507.21587
PDF
-
Post-Disaster Affected Area Segmentation with a Vision Transformer -based EVAP Model using Sentinel-2 and Formosat-5 Imagery
2025-07-21 07:48:07Yi-Shan Chu, Hsuan-Cheng WeiAbstract
We propose a vision transformer (ViT)-based deep learning framework to refine disaster-affected area segmentation from remote sensing imagery, aiming to support and enhance the Emergent Value Added Product (EVAP) developed by the Taiwan Space Agency (TASA). The process starts with a small set of manually annotated regions. We then apply principal component analysis (PCA)-based feature space analysis and construct a confidence index (CI) to expand these labels, producing a weakly supervised training set. These expanded labels are then used to train ViT-based encoder-decoder models with multi-band inputs from Sentinel-2 and Formosat-5 imagery. Our architecture supports multiple decoder variants and multi-stage loss strategies to improve performance under limited supervision. During the evaluation, model predictions are compared with higher-resolution EVAP output to assess spatial coherence and segmentation consistency. Case studies on the 2022 Poyang Lake drought and the 2023 Rhodes wildfire demonstrate that our framework improves the smoothness and reliability of segmentation results, offering a scalable approach for disaster mapping when accurate ground truth is unavailable.
Abstract (translated)
我们提出了一种基于视觉变换器（ViT）的深度学习框架，旨在通过遥感图像来优化受灾地区分割，以支持并提升台湾太空机构（TASA）开发的紧急增值产品（EVAP）。该过程从一小部分手动标注区域开始。然后应用基于主成分分析（PCA）的特征空间分析，并构建置信指数（CI），以此扩大这些标签，生成一个弱监督训练集。接着使用扩展后的标签来训练多频段输入来自Sentinel-2和Formosat-5图像的ViT编码器-解码器模型。 我们的架构支持多种解码器变体和多阶段损失策略，在有限监督下提高性能。在评估过程中，将模型预测与高分辨率EVAP输出进行比较，以评估空间一致性及分割的一致性。2022年鄱阳湖干旱以及2023年罗德岛野火的案例研究表明，我们的框架提高了分割结果的平滑度和可靠性，在缺乏准确地面真实情况的情况下为灾害制图提供了一种可扩展的方法。
URL
https://arxiv.org/abs/2507.16849
PDF
-
Personalized 4D Whole Heart Geometry Reconstruction from Cine MRI for Cardiac Digital Twins
2025-07-21 03:01:33Xiaoyue Liu, Xicheng Sheng, Xiahai Zhuang, Vicente Grau, Mark YY Chan, Ching-Hui Sia, Lei LiAbstract
Cardiac digital twins (CDTs) provide personalized in-silico cardiac representations and hold great potential for precision medicine in cardiology. However, whole-heart CDT models that simulate the full organ-scale electromechanics of all four heart chambers remain limited. In this work, we propose a weakly supervised learning model to reconstruct 4D (3D+t) heart mesh directly from multi-view 2D cardiac cine MRIs. This is achieved by learning a self-supervised mapping between cine MRIs and 4D cardiac meshes, enabling the generation of personalized heart models that closely correspond to input cine MRIs. The resulting 4D heart meshes can facilitate the automatic extraction of key cardiac variables, including ejection fraction and dynamic chamber volume changes with high temporal resolution. It demonstrates the feasibility of inferring personalized 4D heart models from cardiac MRIs, paving the way for an efficient CDT platform for precision medicine. The code will be publicly released once the manuscript is accepted.
Abstract (translated)
心脏数字孪生（CDTs）提供个性化的体内心脏表示，并在心脏病学的精准医学中具有巨大潜力。然而，目前全心CDT模型仍受限于难以模拟所有四个心室的整体器官规模电机械特性。在这项工作中，我们提出了一种弱监督学习模型，可以从多视角2D心脏动态MRI（cine MRIs）直接重建4D（3D+t）心脏网格。通过学习动态MRI与4D心脏网格之间的自监督映射关系，该方法能够生成与输入动态MRI紧密对应的个性化心脏模型。由此产生的4D心脏网格可以自动提取关键心脏变量，包括射血分数和心室容积的动态变化，并具有高时间分辨率。这表明从心脏磁共振图像中推断个性化的4D心脏模型是可行的，为精准医学中的高效CDT平台铺平了道路。一旦论文被接受，代码将公开发布。
URL
https://arxiv.org/abs/2507.15203
PDF
