Defense Against National Vulnerabilities in Public Data

One of the Biggest sources of Data is Public Data. Its data about us, for us, and paid by us. And all this public data is mined and utilized by disparate concerns, both public and private, to gain actionable insight. And with Open Data gaining traction in the US at all levels of government, a lot of this data is now becoming available at a much larger scale in machine-readable format. This project proposes to create a prototype that will crawl representative public data catalogs using the two major open data publishing platforms (i.e. CKAN and Socrata) to extract metadata, derive additional metadata, sample and normalize the data, characterize it statistically and link concepts across catalogs. The aim is to create a semantic metacatalog spanning all the registered data catalogs. By using a graph database, we can systematically create "federated risk graphs" that can be quickly modified to model different risk scenarios across disparate domains. As an option, we also want to explore creating a prototype for an automated "white-hat" mechanism that leverages this semantic metacatalog to proactively investigate and monitor public data catalogs.