An Intelligence Discovery Portal based on Corpus-Level Information Extraction and Text Mining

This proposal addresses two major enhancements to current information extraction (IE) technology. The first concerns the development of higher levels of IE, at the corpus level, and finally across corpora including structured data. T he second objectiveconcerns text mining from a rich IE repository assimilated from multiple corpora. IE is only a means to an end, which is the discovery of hidden trends and patterns that are implicit in large volumes of text. These objectives are based on a core IEsystem, InfoXtract. The core engine will be adapted to handle diverse, case-insensitive information, e.g. broadcast transcripts, HUMINT documents. The fusion system will assimilate information extracted across multiple documents with structured knowledgebases. The resulting rich, knowledge repository will be mined using unsupervised, machine-learning techniques for significant information. Finally, the design of an intelligence discovery portal (IDP) to be used by analysts that consolidates the newtechnology into an intuitive web-based application is proposed. This calls for natural language processing, corpus-based statistical NLP, and advanced machine learning. Final deliverables include: (i) prototype system for corpus-level IE, demonstratedusing the Foreign Broadcast Information Services (FBIS) corpus (ii) prototype for text mining on above repository, and (iii) design of the IDP. Information analysts and decision makers will benefit since the proposed effort extends the utility of currentinformation extraction technology. The IDP represents the goal of advanced IE systems. A commercially viable solution has many applications. Business intelligence systems use large knowledge-bases on companies, products, people and projects. Updatingthese knowledge-bases from diverse text sources as well as deriving key business intelligence in an automatic or semi-automatic manner would be valuable. Our commercial application focused on brand perception and monitoring will benefit. Militaryapplications stem from the fact that analysts need assistance in detecting hidden trends and patterns from large volumes of text.