DALL-E 2 对放射学了解多少?
What Does DALL-E 2 Know About Radiology?
发表日期:2023 Mar 16
作者:
Lisa C Adams, Felix Busch, Daniel Truhn, Marcus R Makowski, Hugo J W L Aerts, Keno K Bressem
来源:
JOURNAL OF MEDICAL INTERNET RESEARCH
摘要:
生成模型(如OpenAI的DALL-E 2)可能成为放射学人工智能研究中潜在的未来工具,用于图像生成、增强和操作,前提是这些模型具有足够的医学领域知识。我们在此展示,DALL-E 2已经学习了相关的X光图像表示,具有零-shot文本到图像生成的新图像、图像边界的持续以及元素去除的有希望的能力;然而,其生成带有病理异常(如肿瘤、骨折和炎症)或计算机断层扫描、磁共振成像或超声成像的图像的能力仍然有限。因此,使用生成模型增强和生成放射学数据似乎是可行的,即使首先需要进一步微调和适应这些模型到各自的领域。©Lisa C Adams,Felix Busch,Daniel Truhn,Marcus R Makowski,Hugo J W L Aerts,Keno K Bressem。最初发表于《医学互联网研究杂志》(https://www.jmir.org),2023年3月16日。
Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.©Lisa C Adams, Felix Busch, Daniel Truhn, Marcus R Makowski, Hugo J W L Aerts, Keno K Bressem. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 16.03.2023.