A Distributed and Standard based Data Warehouse and Mining System for Large-Scale Sensor Data

The sensor data in the area of Navy Multi-INT could be multi-modal, including acoustic images, ocean profiles, infrared data, biometric data, vehicle tracking information, and facial profiles. Textual information, including intelligence reports or chat logs, will also be considered as sensor data. It is important to develop highly scalable and effective data warehousing and mining methods for in-depth exploitation and analysis of such data so that various information consumers can gain intelligence from different perspectives. To address this importance, IAI is developing a distributed and standards based data cubing, mining system, SensorCube, for large amounts of geospatial sensor data. The SensorCube consists of three layers: data integration, data cubing and mining, and semantic search. In the data integration layer, an XML driven ETL (Extraction, Transformation, and Loading) process is developed to integrate complex sensor data. In the data cubing and mining layer, a powerful distributed computational engine is developed to model multi-dimensional sensor data. In the semantic search layer, the search and query capability is developed to connect the meta data and information summarized in data cubes and mining models. Furthermore, the semantic search and cubing visualization can be accessible across different computational platform such as smart phones and tablets.