研究动态
Articles below are published ahead of final publication in an issue. Please cite articles in the following format: authors, (year), title, journal, DOI.

使用统一的超网络学习多模态MR图像合成和肿瘤分割技术,并应对缺失模态问题。

Learning Unified Hyper-network for Multi-modal MR Image Synthesis and Tumor Segmentation with Missing Modalities.

发表日期:2023 Aug 04
作者: Heran Yang, Jian Sun, Zongben Xu
来源: Brain Structure & Function

摘要:

在临床评估和治疗规划中,精确地分割脑肿瘤至关重要,这需要多个提供互补信息的MR模态。然而,由于实际限制,在现实情况下可能会缺少一个或多个模态。为了解决这个问题,现有的方法需要训练多个网络或一个统一但固定的网络来处理各种可能的缺失模态情况,这会导致高计算负担或次优的性能。在本文中,我们提出了一种统一和自适应多模态MR图像合成方法,并进一步将其应用于缺失模态的肿瘤分割。基于多模态MR图像的分解成公共特征和模态特定特征,我们设计了一个共享的超编码器来将每个可用模态嵌入到特征空间中,一个基于图注意力的融合块来聚合可用模态的特征到融合特征中,以及一个共享的超解码器用于图像重构。我们还提出了一个对抗性公共特征约束来强制融合特征处于一个公共空间中。至于缺失模态分割,我们首先使用我们的合成方法对特征级和图像级进行填充,然后根据完成的MR图像和提取的公共特征来分割肿瘤。此外,我们设计了一个基于超网络的调制模块来自适应地利用真实和合成的模态。实验结果表明,我们的方法不仅可以合成合理的多模态MR图像,而且在缺失模态的脑肿瘤分割上取得了最先进的性能。
Accurate segmentation of brain tumors is of critical importance in clinical assessment and treatment planning, which requires multiple MR modalities providing complementary information. However, due to practical limits, one or more modalities may be missing in real scenarios. To tackle this problem, existing methods need to train multiple networks or a unified but fixed network for various possible missing modality cases, which leads to high computational burdens or sub-optimal performance. In this paper, we propose a unified and adaptive multi-modal MR image synthesis method, and further apply it to tumor segmentation with missing modalities. Based on the decomposition of multi-modal MR images into common and modality-specific features, we design a shared hyper-encoder for embedding each available modality into the feature space, a graph-attention-based fusion block to aggregate the features of available modalities to the fused features, and a shared hyper-decoder for image reconstruction. We also propose an adversarial common feature constraint to enforce the fused features to be in a common space. As for missing modality segmentation, we first conduct the feature-level and image-level completion using our synthesis method and then segment the tumors based on the completed MR images together with the extracted common features. Moreover, we design a hypernet-based modulation module to adaptively utilize the real and synthetic modalities. Experimental results suggest that our method can not only synthesize reasonable multi-modal MR images, but also achieve state-of-the-art performance on brain tumor segmentation with missing modalities.