Large Volume, High Speed, Data Management Technology

One of the major problems facing data management today is that collection capabilities far outstrip analysis capabilities. This is particularly true of scientific and engineering data sets. Analysis methods require access that cross file/data granule boundaries. We propose to address this problem by integrating unified data models of scientific data as embodied by the widely used data formats CDF, NetCDF, and HDF through the use of a commercially available data management system, METEOR. We propose to customize METEOR to handle the access requirements as imposed by the CDF, NetCDF, and HDF scientific data models. In particular this will involve clustering techniques to optimize access to large multidimensional datasets through temporal, spatial, or special sampling ordering rules. In Phase I we propose to demonstrate a prototype system utilizing one of the above data formats integrated with the METEOR data management system. This will involve a set of extraction and composition filters to populate the METEOR databases with metadata from the scientific datasets against which standard spatial and temporal, or attribute based queries can be formulated. The datasets retrieved from the data management system using the composition filters, can be merged into a single dataset for analysis. Phase II would involve specific ordering rules, and building a robust system for integration into a commercial product.