New Event Detection

Intelligence organizations want to know when an unprecedented event or new information is reported. While there is good technology for searching, tracking, and filtering on known topics, current methods do poorly at detecting something new. The chiefmechanism of search and topic tracking, spotting important words, is innapropriate-new stories are not ones with no important words. Because the degree of difference of new and old is different for different topics, uniform thresholds for overlap, asused in current filtering technologies, are also inapproriate.This project approaches the problem in three new ways. First, it applies Latent Semantic Analysis (LSA), a machine-learning technology that simulates human understanding of discourse. After automatic training on a large body of representative text, LSAaccurately measures amount of meaning similarity between two passages using all the words in both. Texts with a few words in common are not judged similar if their meaning is different, but are, even if they use entirely different terminology, if theirmeaning is the same. Second, the system interacts with human users to adapt its criteria to their interests and the characteristics of the data. Third, it uses novel LSA-based storage and retrieval techniques to increase efficiency and capacity. The target innovative new event and new information technology will be of benefit to intelligence agencies and government and private organizations needing to keep abreast of news and developments in commerce, science, and world affairs.