Hierarchical Dynamic Exploitation of FMV (HiDEF)

Video collected by aerial platforms can be a significant source of situational awareness. Recent progress in video capture and storage technology makes it possible to scale the gathering and storage of video to greater levels, but the ability to extract semantic meaning has remained a bottleneck because of its dependence on human participation. Previous attempts to automate this review relied on image segmentation approaches that ignored valuable contextual information. VLADE uses deep learning to analyze patterns in training data to enable the generation of descriptions of new videos. For the Phase I system, we built an algorithm to select candidate video segments from longer videos, an indexing scheme for fast look-up, and models for embedding video segments, for generating text descriptions, and for embedding sentences. Phase II will improve model speed and accuracy, incorporate both synthetic and Air Force data, and implement training on a distributed cloud for greater speed and scalability. It will also enhance the model to leverage long-term temporal dependencies in long videos, enable streaming input and user feedback, and integrate all of these components into an end-to-end system that can efficiently ingest and analyze a video and then index it for search.