Achieving a High Level of Scalability in Federated Information Retrieval

At the present time, no federated search engine exists that is capable of searching, aggregating, and ranking more than a small fraction of the scientific content produced by the research community at large and by DOE researchers in particular. Although thousands of sources of valuable content exist, a solution has not been developed that ensures that scientific discoveries already made can be easily found and leveraged to further advance science. This project will identify and implement an architecture to enable the access and processing of massive numbers of documents from thousands of distributed heterogeneous resources. The approach will involve the creation of nested virtual collections ¿ comprised of individual content sources, which can be accessed in a cascading fashion. Phase I will identify and specify an architecture and set of requirements for a distributed computing solution for implementing the high-performance, scalable Information Retrieval system. Simple prototypes ¿ which employ federation, dynamic workflow management, and support of custom document processing filters ¿ will be built and used to prove the feasibility of the proposed architecture. In Phase II, the architecture will be implemented so that documents can be easily accessed and so that customizable, meaningful extraction functions can be applied to these documents. Commercial Applications And Other Benefits as described by the Applicant: The high-performance, scalable information-retrieval solution should find use in organizations that need to access vast numbers of distributed collections and documents, and apply custom filtering to the text, in order to extract meaning from it. In addition to national laboratories and research universities, potential customers include pharmaceutical and biotech companies, oil and gas companies, legal research firms, and financial institutions.