Scoring, Truth and Registration Toolkit (START)

Evaluation is a crucial part of Automatic Target Recognition (ATR) algorithm development. Without standardized metrics and software implementing those metrics, accurate assessment of an algorithm's performance cannot be made. Comparing different ATRalgorithms on the same dataset is impossible unless the same metrics are computed in exactly the same manner. Our Scoring, Truthing And Registration Toolkit (START) offers a consistent set of post-data collection tools for ATR algorithm performanceevaluation. Our primary design goals are to decrease operator effort when entering registration and truth information, increase accuracy of truth information, and develop standardized scoring metrics for assessing ATR algorithm performance. Using START,all imagery for one scene is registered to a common reference frame (geo-registered if needed) such that target locations, identity, orientation, etc. need only be specified once per scene. This reduces operator error since registration requires less skillthan identifying targets. START then computes a variety of standard scoring metrics suggested by the ATR community (and ATRWG in particular).Our Phase II effort will focus on 3D geo-registration capabilities, automatic placement of registration tie-points, scoring difficult ATR tasks (moving targets, change detection, sensor fusion, multi-look data), sensitivity analysis, and a human factorsanalysis optimizing the usability of START.