Multimodal Knowledge Acquisition and Management

Automated techniques for harvesting knowledge from documents will make tasks like intelligence gathering significantly faster and more reliable. Information in documents is often distributed between text and non-textual components, which rely upon each other to create a comprehensive picture of what is being conveyed to the reader. The Friedland Group, working with our partner, The University of Rochester, is developing a multimodal knowledge acquisition and management (MKAM) technology specifically to be able to harvest knowledge that is distributed among different modalities in documents. For example, an image containing people could provide details like the age, race, gender, hair and eye color and relative positions of the individuals in the photo. The addition of a caption, e.g."The Adams Family at their daughter's graduation."provides a deeper context - specifically, who the people are, how they relate to each other and what event are they participating in. Taken individually, text and images each provide only a piece of the puzzle, but adding them together allows us to recover not only the broader context, but important specifics that would otherwise have been missed.