Interactive Visualization of Multi-Terabyte Datasets on Commodity Hardware

Researchers are entering a new era in scientific discovery enabled by Big Data and E- Science. A growing, but largely untapped resource, is large-scale image analytics. To-date, images were only viewed, but advanced analytics now allow scientists to extract quantitative data from images. New imaging techniques now routinely reveal everything from crystal lattices to fully cleared brains in ever- increasing levels of detail. The high resolution and imaging throughput requires significant innovation to visualize, validate and analyze these data. Furthermore, as data sizes grow, it is ever more critical to have the necessary computational power close to the data. More importantly, the pervasiveness of web-based platforms allows for collaboration with colleagues and business associates. In Phase I of the project, a scalable, web-based viewer and query system for large scientific image and tabular data was developed. It was tested on a number of real datasets from materials (X-ray tomography) and life science (high-resolution microscopy). In order to convert this prototype into a hardened product ready for use in real scientific discovery workflows, additional R&D effort is needed. In Phase II, novel query and analysis mechanisms will be added to handle large, diverse collections of scientific datasets. Furthermore, the visualization capabilities will be extended with scalable web viewers for other common scientific data types and with novel ways to analyze large datasets directly in the viewer. Support for scientific workflows and versioning will be added to guarantee repeatability of analyses. Finally, the platform will be hardened by including security, auditing, archival, and paywall features. The main outcome of this project will be a web-based platform that enables ubiquitous access to data, collaboration, and scientific workflows. It will be deployed with a number of beta users to understand the viability for a variety of domains. The final platform has the potential to reshape scientific collaborative work in data-intensive domains and may lead to new insights and discoveries in materials science, life sciences, and other related disciplines. Commercial Applications: The software-as-a-service platform developed in this proposal will pro- vide an important element to any industry dealing with massive data analytics requiring data sharing and delivery. Pharmaceutical and biotech companies are increasingly outsourcing R&D to external contract research organizations boosting the need for inter-organization collaboration over large life science datasets such as full-brain scans. Specifically, the work undertaken in this project has the unique ability to expedite the drug discovery process and potentially reduce the current time-to-market for the development of novel treatments. Similarly, other industries such as materials science, oil and gas exploration, construction, surveillance, and agribusiness will benefit from the proposed image sharing and analysis service.