DMAGs Diffusion Map Augmented Grammars for High-Dimensional Data Representation, Analysis, and Visualization

The Navy and DOD are faced with the mounting problem of analyzing larger and larger corpora of unstructured data. Petabytes of unstructured data are generated from a slew of Navy initiatives (LIDAR images electro-optical videos, sonar images) that are completely intractable to analyze by hand. Ontologies and clustering approaches have been proposed to organize the data and mine it for structure, but hierarchical representations require a lot of human interaction while clustering methods have trouble finding compositional structures in the data. UtopiaCompression"s (UC) diffusion map augmented grammars (DMAGs) solve these problems by automatically organizing the data into a hierarchy of low-dimensional manifolds. DMAGs capture the contextual relationships between subspaces of the data and can identify patterns that would take operators hours or days to find. DMAGs are learned in a statistically rigorous framework (diffusion geometry + minimax entropy) and capture far more variability than diffusion maps alone. Moreover, DMAGs can be used to analyze, visualize, or classify new data at different levels of abstraction. The DMAG approach streamlines Navy and DOD efforts to find patterns in large high-dimensional datasets and leverages a novel knowledge-based hierarchy to quickly and efficiently represent unstructured data from a multitude of different sources.