Abstract
Tumor synthesis can generate examples that AI often misses or over-detects, improving AI performance by training on these challenging cases. However, existing synthesis methods, which are typically unconditional -- generating images from random variables -- or conditioned only by tumor shapes, lack controllability over specific tumor characteristics such as texture, heterogeneity, boundaries, and pathology type. As a result, the generated tumors may be overly similar or duplicates of existing training data, failing to effectively address AI's weaknesses. We propose a new text-driven tumor synthesis approach, termed TextoMorph, that provides textual control over tumor characteristics. This is particularly beneficial for examples that confuse the AI the most, such as early tumor detection (increasing Sensitivity by +8.5%), tumor segmentation for precise radiotherapy (increasing DSC by +6.3%), and classification between benign and malignant tumors (improving Sensitivity by +8.2%). By incorporating text mined from radiology reports into the synthesis process, we increase the variability and controllability of the synthetic tumors to target AI's failure cases more precisely. Moreover, TextoMorph uses contrastive learning across different texts and CT scans, significantly reducing dependence on scarce image-report pairs (only 141 pairs used in this study) by leveraging a large corpus of 34,035 radiology reports. Finally, we have developed rigorous tests to evaluate synthetic tumors, including Text-Driven Visual Turing Test and Radiomics Pattern Analysis, showing that our synthetic tumors is realistic and diverse in texture, heterogeneity, boundaries, and pathology.
Abstract (translated)
肿瘤合成能够生成人工智能常会忽略或过度检测的例子，通过训练这些具有挑战性的案例可以提高AI的性能。然而，现有的合成方法通常是无条件的——从随机变量中生成图像——或者仅由肿瘤形状进行条件控制，缺乏对特定肿瘤特征（如纹理、异质性、边界和病理类型）的可控性。因此，生成的肿瘤可能过于相似或重复现有训练数据，无法有效解决AI的弱点。我们提出了一种新的基于文本驱动的肿瘤合成方法，称为TextoMorph，它可以在文本层面对肿瘤特性进行控制。这对于那些最容易使AI困惑的例子特别有益，比如早期肿瘤检测（提高灵敏度+8.5%）、用于精确放疗的肿瘤分割（增加DSC +6.3%）以及良性与恶性肿瘤分类（提高灵敏度+8.2%）。通过将从放射学报告中提取的文本融入合成过程，我们增加了合成肿瘤的变化性和可控性，以更精准地针对AI的失败案例。此外，TextoMorph利用不同的文本和CT扫描进行对比学习，显著减少了对稀缺图像-报告配对（本研究仅使用了141对）的依赖，并通过一个包含34,035份放射学报告的大语料库来实现这一点。最后，我们开发了一系列严格的测试来评估合成肿瘤，包括基于文本驱动的视觉图灵测试和放射组学模式分析，结果表明我们的合成肿瘤在纹理、异质性、边界和病理方面是现实且多样的。
URL
https://arxiv.org/abs/2412.18589
