Graphical Methods for Discovering Structure and Context in Large Datasets

In this proposed Phase II effort we will implement a software framework that will reduce the time required to annotate large image/video datasets by a factor of 100x while also reducing the data necessary to train state of the art computer vision models by up to 80%. Our strategy combines several elements and ideas. First, we have developed a sub-tile based a priori theory of how and why CNNs can overcome the curse of dimensionality. This theory makes several testable hypotheses that have been verified experimentally. Second, we have developed a framework to exploit the vast amounts of publicly available labeled data sets to build CNNs that are fantastically good at detecting small targets in overhead videos and imagery (both EO and IR). And third, Mayachitra has built an active/continuous learning framework called VisionForge, that requires orders of magnitude fewer annotations from analysts to teach CNNs to detect (bounding box) and classify small targets in EO and IR full motion video (FMV). We will leverage Mayachitra's active learning framework, VisionForge, in building our prototype framework. We note that Mayachitra is currently in the process of transitioning some of our technology in VisionForge to contractors in the Navy. Currently, image analysts painstakingly draw highly accurate bounding boxes around each target in every frame of full motion video (FMV) and then assign individual target specific labels to each bounding box. A single minute of FMV can take two whole days of an analyst's time. This observation is based on input from Navy contractors who are actually performing such annotations for training machine learning methods. Moreover data scarcity is a significant problem in most DoD applications, and accurate pixel level annotation is important to extract accurate machine learning models from the available data. In Phase I we demonstrated that there is very good support for a mathematical theory of sub-tiles for deep learning and that it is applicable to image classification, object detection and localization. In Phase II we will develop, based on these theoretical ideas, a fast and accurate tile-based active learning annotation system. In particular we will extend VisionForge with the tile based theory to allow rapid pixel level annotation of objects. The final goal is to deliver a rapid annotation and deep learning system for detecting and classifying small targets on overhead imagery and FMV from both EO and IR sensors. The deliverables, in addition to monthly updates and detailed interim reports, include software prototypes at 6 month intervals that culminate in a prototype that meets the efficiency gains listed above.