An Intelligence Discovery Portal based on Corpus-Level Information Extraction and Text Mining

This proposal addresses two major enhancements to current information extraction (IE) technology. The first concerns the development of higher levels of IE, at the corpus level, and finally across corpora including structured data. The second objective concerns text mining from a rich IE repository assimilated from multiple corpora. IE is only a means to an end, which is the discovery of hidden trends and patterns that are implicit in large volumes of text. These objectives are based on a core IE system, InfoXtract. The core engine will be adapted to handle diverse, case-insensitive information, e.g. broadcast transcripts, HUMINT documents. The fusion system will assimilate information extracted across multiple documents with structured knowledge bases. The resulting knowledge repository will be mined using unsupervised, machine-learning techniques for significant information. Finally, an intelligence discovery portal (IDP) to be used by analysts that incorporates the new technology into an intuitive web-based application is proposed. The final deliverable is a functional IDP prototype reflecting: (i) user-driven threaded search and browsing, (ii) data-driven search and browsing based on text mining technology, and (iii) visualization trends analysis in support of situational awareness.