SANDMAN: Synthesizing Activities and Narrative Descriptions from MOVINT through Active iNference

Advances in location acquisition, remote sensing, and wearable technologies have resulted in explosive growth of spatial trajectory data, which capture the mobility of diverse entities. This information offers us the unprecedented opportunity to understand their capabilities, activities, and even intents. For example, spatiotemporal data from radar sensors and wide-area imagery platforms have been extensively used by intelligence analysts for area observation, object classification, and activity detection. When analysts evaluate movement data in the context of available sociocultural information, their reporting insights become much more significant and meaningful. Unfortunately, with today’s overabundance of trajectory data, humans manually performing these tasks becomes infeasible. Intelligence analysts need technologies that can provide automated assistance regarding the detection, understanding, and sharing of salient, culturally informed activities from mover intelligence (MOVINT). One major technical challenge that must be overcome to produce such technology is the heterogeneity gap, wherein a system must identify mappings between content in differing modality formats (motion vs. text). Furthermore, for such a system to be useful to intelligence analysts, it must “speak the same language” as the analysts so as not to add extra work. \n\n Aptima and PatchPlus Consulting propose to develop a system for Synthesizing Activities and Narrative Descriptions from MOVINT through Active iNference (SANDMAN). The objective of SANDMAN will be to develop a proof-of-concept tool capable of generating narrative descriptions and structured summaries of events, activities, and anomalies associated with locations and data from MOVINT, GIS, and cultural contexts to both facilitate and augment the intelligence-analysis process. SANDMAN will utilize a unified framework for synthesizing activity text narratives and recognizing activities from MOVINT data that expand the theory of active inference to the multi-modal case. This theory considers perception and action under one universal imperative: maximize the evidence for (generative) models of the world. By mapping both text and trajectory data into a common latent knowledge representation, SANDMAN will be able to close the heterogeneity gap and perform inference (identifying activity given observed data), generation (constructing texts and tracks given a model), and adaptation/learning (modifying models that explain and encode activities of interest). Critically, SANDMAN will allow users to input their needs and feedback via free text and explore and edit the individual components of the system’s models. Since model structures will encode entities, relations, and attributes using a common lexical vocabulary, these components will correspond to logical rules and constraints, which would be easy to interpret and modify, reducing analyst workload.