TOPIC 454 - SOFTWARE TO EVALUATE ARTIFICIAL INTELLIGENCE/MACHINE LEARNING MEDICAL DEVICES IN ONCOLOGY SETTINGS

Artificial intelligence (AI)-based software devices are starting to play key roles in healthcare with more and more regulatory approval and clinical adoptions. Segmentation is a key task in oncology settings including radiation treatment planning and volumetric assessment to assist cancer diagnosis and treatment monitoring. In radiation oncology, multiple AI segmentation devices have obtained FDA 510(k) clearance and are adopted. However, there is a lack of standardized evaluation tool that can conveniently estimate and report the robustness of the AI performance, especially in real-world setting. For segmentation devices, the challenges for such a tool include: 1) the ground-truthing can be subjective and institution specific; 2) the evaluation criteria that are focused on overlap with the ground truth may not accurately reflect the real-world performance and safety concerns; and 3) the comparison and statistical analysis can be complicated by the variations of these devices as each one may define a different set of output segmentations. These creates obstacles for device manufacturers, regulatory, and clinical users. To address these challenges and benefit all parties, in this proposal we aim to develop a convenient tool that potentially qualify for the FDA Medical Device Development Tools (MDDT) Program to provide standardized evaluation for AI segmentation.