Modeling Topic Pragmatics for Improved First Story Detection

We propose an innovative combination of machine learning techniques and pragmatics modeling in pursuit of high accuracy, domain- and language-independent First Story Detection. Our Topic-Oriented Pragmatics and Invariant Chaining (TOPIC) system willautomatically construct topic category models that explicitly capture the pragmatics of how topic cohesion is maintained across the stories that constitute a news topic. These models will be used to generate predictions about the types of stories expectedon diverse news channels, providing a hitherto untapped source of novelty discrimination. Central to our pragmatics-based approach is the idea that the invariant entities and events that constitute a topic are better determinants of topicality thanfull-text similarity measures. Our model-based techniques combine with statistical text-similarity algorithms to provide independent perspectives on story topicality. Committee-based methods arbitrate these multiple classification viewpoints. TOPIC alsoexploits redundancy in pragmatics expression across news channels and source languages. Phase I research and development of a proof-of-concept limited prototype will demonstrate the feasibility and utility of TOPIC's First Story Detection capability andwill lay the groundwork for its Phase II implementation and eventual commercialization. Besides being an important tool for intelligence analysis, TOPIC would also prove useful in numerous commercial domains (e.g., competitive market analysis andepidemiology) where the identification and monitoring of topical news articles is a requirement. Another exciting application is in the creation of individually customized news reports.