Low-shot Automated Performance Prediction via Transfer Learning

Low-shot objection recognition has become an area of active research in recent years, with advances dramatically improving performance when only a few samples are available, nominally fewer than 20. These technologies are a focus of the intelligence community (IC) because this challenge pertains to many intelligence problems, e.g., objects of interest are rare due to their use, sensitive nature, or context.  Published works artificially subsample a class to create a low-shot environment in training and retain the larger sample size for the test environment; however, how do we determine performance when only a few samples truly exist? The traditional cross-validation approach with training/test separation breaks down when only a few exemplars are available. This problem is further exacerbated by correlation amongst the example images (e.g., derived from the same location, time, relative aspect, etc.).  Furthermore, traditional approaches do not provide either performance estimates or an indication of uncertainty (especially relevant measures given the limited training data).  Without an understanding of the performance of such a system, it's impossible to abide by the DoD ethical principles of Artificial Intelligence. \n\n Our proposed Low-shot Transfer Performance Predictor (LowTraPP) first models the performance of well-sampled classes, a process that is agnostic to the underlying recognition system.  The resulting learned distributions then serve as priors for modeling low-shot classes' performance as a function of critical operating parameters. LowTraPP satisfies the following requirements levied by the IC community: \n\n\n\t LowTraPP is agnostic to the recognition system \n\t LowTraPP provides an estimate of performance with confidence bounds \n\t LowTraPP is a function of operating conditions of interest \n\t LowTraPP operates solely on available measured data  \n