乳腺癌显着影响女性健康，超声波对于病变评估至关重要。为了提高诊断准确性，计算机辅助检测（CAD）系统引起了人们的极大兴趣。这项研究引入了一种名为“多模态多任务网络”（3MT-Net）的前瞻性深度学习架构。 3MT-Net 结合了临床数据、B 型和彩色多普勒超声。我们设计了 AM-CapsNet 网络，专门用于从超声中提取关键的肿瘤特征。为了合并 3MT-Net 中的临床数据，我们采用了级联交叉注意力来融合来自三个不同来源的信息。为了确保在高维和低维数据融合过程中保留相关信息，我们采用集成学习的思想，设计了一种优化算法来为不同模态分配权重。最终，3MT-Net进行良恶性病变的二元分类以及病理亚型分类。此外，我们回顾性地收集了九个医疗中心的数据。为了确保 3MT-Net 的广泛适用性，我们创建了两个独立的测试集并进行了广泛的实验。此外，还对3MT-Net与工业级CAD产品S-detect进行了对比分析。 3MT-Net的AUC超过S-Detect 1.4%至3.8%。

Breast cancer significantly impacts women's health, with ultrasound being crucial for lesion assessment. To enhance diagnostic accuracy, computer-aided detection (CAD) systems have attracted considerable interest. This study introduces a prospective deep learning architecture called "Multi-modal Multi-task Network" (3MT-Net). 3MT-Net utilizes a combination of clinical data, B-mode, and color Doppler ultrasound. We have designed the AM-CapsNet network, specifically tailored to extract crucial tumor features from ultrasound. To combine clinical data in 3MT-Net, we have employed a cascaded cross-attention to fuse information from three distinct sources. To ensure the preservation of pertinent information during the fusion of high-dimensional and low-dimensional data, we adopt the idea of ensemble learning and design an optimization algorithm to assign weights to different modalities. Eventually, 3MT-Net performs binary classification of benign and malignant lesions as well as pathological subtype classification. In addition, we retrospectively collected data from nine medical centers. To ensure the broad applicability of the 3MT-Net, we created two separate testsets and conducted extensive experiments. Furthermore, a comparative analysis was conducted between 3MT-Net and the industrial-grade CAD product S-detect. The AUC of 3MT-Net surpasses S-Detect by 1.4% to 3.8%.