TALON: Multi-INT Metadata Extraction for Threat Detection

Despite a large and growing set of available data sources across many domains, identifying and tracking developing threats in operational domains remains a critical unsolved problem. Extraction of relevant threat information and correlation across multiple modalities is required to take full advantage of the data available. VSI proposes TALON, a system for leveraging natural language descriptions of multi-int data, which are semantically rich, extremely flexible, and can be readily generated and processed thanks to recent advances in machine learning, computer vision, and natural language processing. Combined with geo-position and high-value individual (HVI) identifiers, natural language metadata can express a detailed description of virtually any activity or event with its surrounding context. The ability to generate semantically rich natural language-based metadata from multi-modal input source data provides a massive opportunity to leverage higher level reasoning and trend analysis tools originally designed for streams of text. The proposed threat identification system includes a suite of metadata generation algorithms based on state-of-the-art computer vision, machine learning, and natural language processing technology designed to extract detailed content descriptions from multimedia source data. Potential threats are identified within the generated metadata stream by the proposed anomalous activity detection system based on deviations from dynamically learned normalcy models. In addition to developing algorithms for automatic metadata generation and threat analysis, significant effort will be devoted to the curation of rich multi-modal datasets and quantitative evaluation metrics. These datasets and evaluation metrics will be leveraged to establish feasibility of the proposed approach and identify key challenges to be addressed in Phase II. VSI and STR are uniquely suited to successfully execute this project thanks to their experience and expertise in computer vision, remote sensing, natural language processing, and data fusion. VSI and STR have an extensive track record of delivering innovative systems capable of extracting and fusing intelligence from raw data sources such as satellite imagery, news reports, social media, and structured data in challenging real-world problem domains.