DIG-SOF: A Next Generation Toolset for Data Aggregation and Analysis

As the Web has become an increasingly important communication network, it has also become a resource for terrorists, criminals, insurgents and other adversaries. For this reason, military and intelligence analysts are increasingly interested in technology for monitoring and analyzing information on Web sites, social media and other sources.However, there are many difficult technical challenges involved in this endeavor. Data needs to be harvested from heterogeneous sources without revealing our own identity and operating procedures. Relevant information needs to be extracted from each source, the information needs to cleaned and normalized, and data about entities described in multiple sources needs to be resolved and merged.Finally, the aggregated data must be analyzed to uncover and link relevant information, and the results presented in an easy-to-digest manner.In this project, our goal is to develop a system, DIG-SOF, that addresses these needs.There are many existing components that address pieces of the problem, and we propose to evaluate which components can be integrated to produce a highly capable and flexible system that will meet a wide spectrum of SOCOMs needs.