FishEye Intelligent Dynamic Data Logging – Automated Metadata Integration

Connecting data silos and extracting information is a ubiquitous challenge for the Missile Defense Agency (MDA). MDA operates many subsystems from diverse suppliers built with multi-generation technology. These subsystems emit complex and wide-ranging data flows at petabyte scale that makes data conversion impractical. There is a desire for unified data access for analysis, and for Artificial Intelligence (AI) Machine Learning (ML) model development and insight. Impediments to these mission needs are constantly evolving data interfaces, diverse data formats, the need to move and analyze petabyte-scale-data, and difficulty in accessing data. The proposed approach automatically extracts meta-data from subsystems to dynamically link evolving data into a common data repository or “Data Lake” for routine and impromptu analysis. Metadata enables a process that maximizes performance by limiting data manipulation, and ensures integrity by keeping data in its original form. Metadata is used to create views into the repository so that the pools of heterogenous data appear homogeneous to a diverse analysis community and their custom and commercial tools. Approved for Public Release | 22-MDA-11339 (13 Dec 22)