Advancing Hyperspectral Data Cube Query Capabilities through Apache Spark DataFrame Abstraction

The goal of this research is to accelerate the use of NASArsquo;s hyperspectral data by combining powerful open source tools into an easily understood workflow that will distribute the computation of hyperspectral data across clusters of machines hosted in public cloud infrastructure, while also leveraging low cost object storage in order to maximize accessibility and use.nbsp; Hyperspectral data collects and processes information across the electromagnetic spectrum, dividing the spectrum into many more bands than are visible to the human eye.nbsp; The collected images are combined to form a three-dimensional data cube, where two spatial dimensions of the same scene are joined by a third dimension comprised of a range of spectral wavelengths.nbsp; These data cubes are capable of supporting many surface biology and geology applications, with particular potential for improving the discovery and management of energy, mineral, and soil resources.nbsp; That said, the ability to efficiently process hyperspectral data is currently limited by the complexity of the data itself, and the inability to present and store it in formats and public cloud environments useful to data scientists.nbsp;nbsp;In addition to building standalone tools that can be used independently, this project will enable efficient consumption, reformatting, and processing of hyperspectral datasets in the NASA-funded Raster Foundry platform, an open source solution for finding, analyzing, and publishing geospatial imagery on the web.nbsp; In so doing, the proposed research will build on previous agency investments that are making additional remotely sensed imagery from NASA and other public resources more broadly accessible for global application to contemporary geospatial challenges.nbsp; Furthermore, it will provide access to a rich ecosystem of image processing and scientific computing libraries that will support data science studies in industries ranging from precision agriculture to security and defense.