Extrieva - A Low Cost Scalable Archive Storage Management System

Locating, summarizing and presenting large collections of Earth science data in a highly distributed and networked environment is critical in NASA's mission for Earth Sciences. Technologies supporting management, storage, search and retrieval of very large, distributed, geo-spatial earth science data volumes are urgently needed to cope with the impending data survivability crisis. For instance, the EOSDIS archive data growth rate is currently about 1 petabyte/year. NRC's Committee on Coping with Increasing Demands on Government Data Centers recently made a series of recommendations on which emerging technologies can help data centers meet user needs and build and maintain the long-term record of environmental change. In this proposal, we propose to design, develop and prototype Extrieva - a low cost scalable Archive Storage Management System innovation, which embraces several of the NRC technology recommendations. In particular, Extrieva is a disk-based solution as assessed by NRC to be now competitive with tape for long-term, archival-class storage. Moreover, with its self-management and automation features implemented over commodity Linux clusters using distributed indexing and load balancing algorithms, Extrieva offers a low cost scalable solution with unprecedented ease of management addressing the needs of EOSDIS' diverse global users base.