M 2 FTrans:模态屏蔽融合变压器,用于不完整的多模态脑肿瘤分割。
M 2 FTrans: Modality-Masked Fusion Transformer for Incomplete Multi-Modality Brain Tumor Segmentation.
发表日期:2023 Oct 20
作者:
Junjie Shi, Li Yu, Qimin Cheng, Xin Yang, Kwang-Ting Cheng, Zengqiang Yan
来源:
IEEE Journal of Biomedical and Health Informatics
摘要:
脑肿瘤分割是一项基本任务,现有方法通常依赖多模态磁共振成像(MRI)图像来进行精确分割。然而,临床实践中常见的模态缺失/不完整问题会严重降低其分割性能,并且现有的不完整多模态脑肿瘤分割融合策略远非理想。在这项工作中,我们提出了一种名为 M 2 FTrans 的新颖框架,用于在各种不完整的多模态设置下通过模态屏蔽融合变压器探索和融合跨模态特征。考虑到普通的自注意力对丢失的标记/输入很敏感,引入了可学习的融合标记和屏蔽的自注意力,以稳定地构建跨模态的远程依赖,同时更灵活地从不完整的模态中学习。此外,为了避免偏向某些主导模态,通过空间权重注意力和通道融合变换器进一步重新加权模态特定特征,以减少特征冗余和模态重新平衡。通过这种方式,M 2 FTrans 中的融合策略对于缺失模态更加稳健。广泛使用的 BraTS2018、BraTS2020 和 BraTS2021 数据集上的实验结果证明了 M 2 FTrans 的有效性,在各种不完整的脑肿瘤分割模式下,其性能明显优于最先进的方法。代码可在 https://github.com/Jun-Jie-Shi/M2FTrans 获取。
Brain tumor segmentation is a fundamental task and existing approaches usually rely on multi-modality magnetic resonance imaging (MRI) images for accurate segmentation. However, the common problem of missing/incomplete modalities in clinical practice would severely degrade their segmentation performance, and existing fusion strategies for incomplete multi-modality brain tumor segmentation are far from ideal. In this work, we propose a novel framework named M 2 FTrans to explore and fuse cross-modality features through modality-masked fusion transformers under various incomplete multi-modality settings. Considering vanilla self-attention is sensitive to missing tokens/inputs, both learnable fusion tokens and masked self-attention are introduced to stably build long-range dependency across modalities while being more flexible to learn from incomplete modalities. In addition, to avoid being biased toward certain dominant modalities, modality-specific features are further re-weighted through spatial weight attention and channel- wise fusion transformers for feature redundancy reduction and modality re-balancing. In this way, the fusion strategy in M 2 FTrans is more robust to missing modalities. Experimental results on the widely-used BraTS2018, BraTS2020, and BraTS2021 datasets demonstrate the effectiveness of M 2 FTrans, outperforming the state-of-the-art approaches with large margins under various incomplete modalities for brain tumor segmentation. Code is available at https://github.com/Jun-Jie-Shi/M2FTrans.