TALON: Multi-INT Metadata Extraction for Threat Detection

Identifying and tracking developing threats across a large and growing set of data sources remains a critical unsolved problem. Currently, multi-modal data (images, video, audio clips, satellite images, etc.) as well as OSINT data (social media platforms like Twitter, Telegram, etc.) must be manually inspected by an analyst to extract relevant metadata, such as ōwhoö, ōwhatö, ōwhereö, ōwhenö, and ōwhyö. Subsequent correlation across multiple modalities, required to identify and track developing threats in operational domains, is also a manual process. Recent advances in computer vision, machine learning (ML), and natural language processing (NLP) technology have resulted in the creation of innovative algorithms capable of assisting an analyst in understanding, cataloging, and characterizing multi-modal content automatically. However, these advances have seen limited application to defense and intelligence problems. Vision Systems Inc. (VSI) with support from STR proposes TALON, an innovative multi-INT analysis framework designed to generate information-rich metadata across multi-INT data streams and use the context provided by this metadata to detect emerging threats. In Phase I, TALON leveraged advances in ML and NLP to discover events from publicly available social media and online sources and produce flexible and semantically rich natural-language descriptions of associated multi-modal data. The proposed SBIR Phase II program will build on Phase I results, enhancing and expanding metadata generation capabilities, analyzing metadata using the latest in state-of-the-art NLP systems such as Large Language Models (LLMs) models, and considering operational data sources of interest to potential stakeholders across the defense and intelligence community. The VSI and STR team is uniquely suited to this project, offering unparalleled expertise in machine learning, natural language processing, and remote sensing; extensive experience in the development of innovative software solutions for challenging defense and intelligence problems; and a thorough understanding of the needs of image analysts across the defense and intelligence community.