Evaluation Testbed for ATD/T Performance Prediction (ETAPP)

Currently, common techniques for evaluating automatic target detection/tracking (ATD/T) performance include use simple models, such as quick-look models, or detailed exhaustive simulation. Simple models cannot accurately quantify the performance of candidate algorithms, while the detailed simulation is time consuming, requiring evaluation over each operating condition. A need exists for ATD/T performance prediction based on more accurate models. We propose a method that predicts performance based on image measures quantifying the intrinsic difficulty of ATD/T on the image, as well as using relevant metadata. The propose measures are based on the National Imagery Interpretability Rating Scales (NIIRS) Measure, the Constant False Alarm Rate Measure (CFAR), and the Free Response Operating Characteristic (FROC). We propose a two-phase approach: a learning phase, where the image measures are computed over a set of test images, and the performance of each algorithm is computed; and a performance prediction phase. From the learning phase a model of predicted performance is learned that can be applied across several algorithms and over the various operating conditions, as represented by the test image suite. This model can be used for subsequent rapid evaluation of future algorithms. The testbed has plug-in capability to allow rapid evaluation of new ATD/T algorithms.