背景 在癌症诊断中事前识别人工智能（AI）失败风险的患者将有助于更安全地将诊断算法整合到临床中。目的 在MRI和CT中评估AI预测变异性作为确定癌症诊断AI失败风险的不确定性量化（UQ）指标，跨不同癌症类型、数据集和算法。 材料和方法 回顾性分析多中心数据集和三个先前研究中评估增强CT图像中胰腺癌检测、MRI扫描中前列腺癌检测和低剂量CT图像中肺结节恶性预测的公开可用AI算法。每个任务的算法被扩展以生成基于集成预测变异性的不确定性分数。使用排列检验对百分位阈值范围（10%-90%）的不确定性分数下的确定和不确定患者组进行AI准确度百分比和接受者操作特征曲线下部分面积（pAUC）进行比较，以确定统计学显著性。将肺结节恶性预测算法与11个临床读者的确定组（CG）和不确定组（UG）进行比较。 结果 共使用18,022张图像进行训练，使用838张图像进行测试。对于所有任务，CG中的AI诊断准确率较高（P < .001）。在80%的确定预测阈值下，CG中的准确率比UG中的高出21%-29%，比整体测试数据集中的高出4%-6%。CG中的病变级别pAUC比UG中的高出0.25-0.39，比整体测试数据集中的高出0.05-0.08（P < .001）。对于肺结节恶性预测，AI的准确率与临床医生在CG中持平（AI结果与临床结果相比，80% [95% CI: 76，85] vs. 78% [95%CI: 70，87]; P = .07），但在UG中较差（AI结果与临床结果相比，50% [95% CI: 37, 64] vs. 68% [95% CI: 60, 76]; P < .001）。 结论 AI预测UQ指标始终能够确定癌症诊断中AI性能的下降。©RSNA，2023。本文提供了补充材料。请参阅本期编辑Babyn的社论。

Background A priori identification of patients at risk of artificial intelligence (AI) failure in diagnosing cancer would contribute to the safer clinical integration of diagnostic algorithms. Purpose To evaluate AI prediction variability as an uncertainty quantification (UQ) metric for identifying cases at risk of AI failure in diagnosing cancer at MRI and CT across different cancer types, data sets, and algorithms. Materials and Methods Multicenter data sets and publicly available AI algorithms from three previous studies that evaluated detection of pancreatic cancer on contrast-enhanced CT images, detection of prostate cancer on MRI scans, and prediction of pulmonary nodule malignancy on low-dose CT images were analyzed retrospectively. Each task's algorithm was extended to generate an uncertainty score based on ensemble prediction variability. AI accuracy percentage and partial area under the receiver operating characteristic curve (pAUC) were compared between certain and uncertain patient groups in a range of percentile thresholds (10%-90%) for the uncertainty score using permutation tests for statistical significance. The pulmonary nodule malignancy prediction algorithm was compared with 11 clinical readers for the certain group (CG) and uncertain group (UG). Results In total, 18 022 images were used for training and 838 images were used for testing. AI diagnostic accuracy was higher for the cases in the CG across all tasks (P < .001). At an 80% threshold of certain predictions, accuracy in the CG was 21%-29% higher than in the UG and 4%-6% higher than in the overall test data sets. The lesion-level pAUC in the CG was 0.25-0.39 higher than in the UG and 0.05-0.08 higher than in the overall test data sets (P < .001). For pulmonary nodule malignancy prediction, accuracy of AI was on par with clinicians for cases in the CG (AI results vs clinician results, 80% [95% CI: 76, 85] vs 78% [95% CI: 70, 87]; P = .07) but worse for cases in the UG (AI results vs clinician results, 50% [95% CI: 37, 64] vs 68% [95% CI: 60, 76]; P < .001). Conclusion An AI-prediction UQ metric consistently identified reduced performance of AI in cancer diagnosis. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Babyn in this issue.