A Stream Processing Engine Approach to Earth Science Data Processing

Timely processing of raw Earth science data for calibration and validation in a highly distributed and networked environment, and its storage at Distributed Active Archive Centers (DAACs) for presentation to the global scientific community is critical in NASA's mission for Earth Sciences. Here we propose to develop a stream processing engine approach to earth science data processing. Our innovation is based leveraging the emerging stream processing engine technology. Traditionally stream processing applications have been built using customized DBMS., which tend to be costly, and hard to change by non-specialist end-users. Our proposed architecture offers several significant benefits. First, an SPE developed application enables the refinement of stream filtering, the rapid development of new stream filtering capability faster than any other database or middleware based solution using StreamSQL , thus improving the maintainability and adaptivity of the system especially by non-specialist end-users. Second, by design, SPE technology offers inherent fault tolerance against asynchronous data input with attendant drop-outs.