BSM-KD: BMDS Simulation Mining for Knowledge Discovery

The Missile Defense Agency (MDA) is currently producing hundreds of terabytes of simulation data per year and will soon be producing petabytes every year. A system is needed to ingest, host, and analyze both batch and streaming simulation data sources. These simulations could represent tests at a variety of locations, using a diverse set of hardware, and at a significant range of scales. Analysis across these simulations, including correlation, causation, and prediction, will require implementation of established and the development of novel data mining methodologies. The volume, variety, veracity, and velocity of the data and the complexity of the mining algorithms preclude a relational database approach. IAI proposes a data lake, BSM-KD, utilizing open big data technology. The data lake will leverage NoSQL database technology to handle the frequently unstructured simulation data that could derive from simulation input parameters, run-time logs, and simulation output and end states. Our experience with the Hadoop Ecosystem and Big Data technology will enable at scale analysis across the BMDS Simulation landscape. To demonstrate feasibility during Phase 1, we will design and implement a data lake architecture and data mining methodologies that can handle simulation data at scale utilizing the public and internal datasets. Approved for Public Release | 18-MDA-9817 (23 Oct 18)