Consolidating Entity Information from Heterogeneous Text Sources for Multi-INT Fusion

Machine Learning has proven to be a useful approach for developing entity consolidation systems that can be tuned to a particular domain. However, the accuracy of learning algorithms often suffers in applications where there are “special cases” that learning algorithms systematically misclassify. To address this, we propose a new twist to the machine learning approach to entity consolidation in which a domain expert provides knowledge to the system about special cases. This knowledge enables the system to create models that explicitly handle these types of cases. In a sense, our approach combines the best aspects of machine learning with ideas from the older “rule-based” approach to entity consolidation. The objective is to take advantage of the statistical characteristics of data sets, while making it easy to create commonsense inference models that can utilize these statistics for high accuracy matching. We believe this approach will significantly improve upon the state-of-the-art accuracy of entity consolidation, particularly in applications that involve heterogeneous data, where special cases are more common. This includes applications where entity extractors harvest data from very different types of documents.