Management and Comparative Analysis of Dataset Ensembles

Simulation and experimental data is growing in size and complexity. Furthermore, the number of datasets that must be analyzed and compared with one another is increasing. These increases in data size and complexity are negatively impacting the use of modern computer simulation methods, since scientists do not have the necessary software and data management tools to process such data effectively. Since DOE researchers are now producing datasets approaching petascale, this is problem is becoming particularly acute and hindering their ability to perform advanced analysis. To address this problem, we are developing a data-centric framework for managing, comparing, analyzing, and statistically processing such data ensembles. The framework includes robust tools for importing, browsing, processing, and visualizing multiple datasets. Automated techniques will also be developed for processing hundreds or thousands of datasets, including tools for configuring and applying processing pipelines. Such automation techniques, when coupled with modern database technology, will enable scientists and engineers to rapidly evaluate alternative designs or explore experimentally measured physical phenomena. For example, thousands of computer simulations that model variations of an engine design can be run concurrently, from which the user can select the optimal configuration. The framework will be open and hence accessible by a variety of standard tools such as Python, VTK, ParaView and Visit. The simulation and analysis market is large measured in the billions of dollars. Visual analysis will enable these techniques to be used more effectively, therefore the market potential for this technology is measured in the hundreds of millions of dollars. As an open source company Commercial Applications and other Benefits as described by the awardee , Kitware will contribute to the public good, including DOE, by delivering the core technology in open source form. However, we will commercialize value added products that integrate the technology into the workflow of an organization