乳腺癌是女性第二大常见癌症，被认为是最危险的癌症类型之一，并且在全球范围内呈上升趋势。定期筛查对于早期治疗至关重要。数字乳房X线摄影（DM）是最受认可和广泛使用的乳腺癌筛查技术。对比增强光谱乳腺 X 线摄影（CESM 或 CM）与 DM 结合使用来检测和识别隐藏的异常，特别是在致密的乳腺组织中，单独使用 DM 可能效果不佳。在这项工作中，我们探索了每种模式（CM、DM 或两者）在使用深度学习方法检测乳腺癌病变方面的有效性。我们引入了一种用于在头尾 (CC) 和内外侧斜 (MLO) 视图中检测和分类 DM 和 CM 图像中乳腺癌病变的架构。所提出的架构（JointNet）由用于提取局部特征的卷积模块、用于提取远程特征的变换器模块以及用于融合局部特征、全局特征和基于局部特征加权的全局特征的特征融合层组成。这显着提高了将 DM 和 CM 图像分类为正常或异常类别以及将病变分类为良性或恶性的准确性。以我们的架构为骨干，引入了三个病变分类管道，利用专注于病变形状、纹理和整体乳房纹理的注意力机制，检查有效病变分类的关键特征。结果表明，我们提出的方法在将图像分类为正常或异常方面优于其组件，并减轻了独立使用变换器模块或卷积模块的限制。还引入了集成模型来探索每种模式和每种视图的效果，以提高基线架构的准确性。与其他类似作品相比，结果显示出优越的性能。 DM 图像上的最佳性能是通过半自动 AOL 病变分类管道实现的，准确度为 98.85%，AUROC 为 0.9965，F1 分数为 98.85%，精确度为 98.85%，特异性为 98.85%。对于 CM 图像，使用自动 AOL 病变分类管道获得了最高结果，准确度为 97.47%，AUROC 为 0.9771，F1 分数为 97.34%，精确度为 94.45%，特异性为 97.23%。使用 DM 和 CM 图像时，半自动集成 AOL 分类管道提供了最佳的整体性能，准确度为 94.74%，F1 分数为 97.67%，特异性为 93.75%，灵敏度为 95.45%。此外，我们还探讨了 CM 和 DM 图像在深度学习模型中的比较有效性，表明虽然 CM 图像为人眼提供了更清晰的洞察，但我们在 DM 图像上训练的模型使用病变关注 (AOL) 技术产生了更好的结果。该研究还表明，使用 DM 和 CM 图像以及集成学习的多模态方法可以提供更稳健的分类结果。版权所有 © 2024。由 Elsevier Ltd 出版。

Breast cancer ranks as the second most prevalent cancer in women, recognized as one of the most dangerous types of cancer, and is on the rise globally. Regular screenings are essential for early-stage treatment. Digital mammography (DM) is the most recognized and widely used technique for breast cancer screening. Contrast-Enhanced Spectral Mammography (CESM or CM) is used in conjunction with DM to detect and identify hidden abnormalities, particularly in dense breast tissue where DM alone might not be as effective. In this work, we explore the effectiveness of each modality (CM, DM, or both) in detecting breast cancer lesions using deep learning methods. We introduce an architecture for detecting and classifying breast cancer lesions in DM and CM images in Craniocaudal (CC) and Mediolateral Oblique (MLO) views. The proposed architecture (JointNet) consists of a convolution module for extracting local features, a transformer module for extracting long-range features, and a feature fusion layer to fuse the local features, global features, and global features weighted based on the local ones. This significantly enhances the accuracy of classifying DM and CM images into normal or abnormal categories and lesion classification into benign or malignant. Using our architecture as a backbone, three lesion classification pipelines are introduced that utilize attention mechanisms focused on lesion shape, texture, and overall breast texture, examining the critical features for effective lesion classification. The results demonstrate that our proposed methods outperform their components in classifying images as normal or abnormal and mitigate the limitations of independently using the transformer module or the convolution module. An ensemble model is also introduced to explore the effect of each modality and each view to increase our baseline architecture's accuracy. The results demonstrate superior performance compared with other similar works. The best performance on DM images was achieved with the semi-automatic AOL Lesion Classification Pipeline, yielding an accuracy of 98.85 %, AUROC of 0.9965, F1-score of 98.85 %, precision of 98.85 %, and specificity of 98.85 %. For CM images, the highest results were obtained using the automatic AOL Lesion Classification Pipeline, with an accuracy of 97.47 %, AUROC of 0.9771, F1-score of 97.34 %, precision of 94.45 %, and specificity of 97.23 %. The semi-automatic ensemble AOL Classification Pipeline provided the best overall performance when using both DM and CM images, with an accuracy of 94.74 %, F1-score of 97.67 %, specificity of 93.75 %, and sensitivity of 95.45 %. Furthermore, we explore the comparative effectiveness of CM and DM images in deep learning models, indicating that while CM images offer clearer insights to the human eye, our model trained on DM images yields better results using Attention on Lesion (AOL) techniques. The research also suggests a multimodal approach using both DM and CM images and ensemble learning could provide more robust classification outcomes.Copyright © 2024. Published by Elsevier Ltd.