UnifiedDL: Secure Metadata-enabled Intelligent Data Integration and Management Lake by AI-powered Graph Canonization and Parallel Processing

The Navy has started integrating Battle Management Aid systems (BMA) on the ship platform to share heterogonous superset data from disparate data sources through a centralized data repository like Data Lake (DL). However, the existing approaches like a common data delivery occur asynchronously and take up available bandwidth for intra-application sharing multiple times. The optimization of shared data from the DL for reuse across BMAs has been limited without a unified data management strategy. For this issue, InfoBeyond advocates UnifiedDL (Secure Metadata-enabled Intelligent Data Integration and Management Lake by AI-powered Graph Canonization and Parallel Processing) for providing a secure and unified data management approach in the backend DL to effectively support microservices across BMAs. UnifiedDL utilizes graph canonization with an attention-based neural learning network for integrating heterogeneous datasets into a unified data model for microservices. Such a unified data model enables common data field search by microservices and further enhances the efficiency of data reuse. UnifiedDL also tackles the effectiveness of the multiple-request-format searches from microservices by developing a parallel-processing runtime-optimized aggregated query approach through the cluster-based multithreading execution. Meanwhile, the proposed multi-zone YANG-specified metadata strategy integrates the ingestion-time access control schema for enhancing data reuse, management, and security/privacy. In such a way, UnifiedDL provides effective data lifecycle management within the DL that minimizes the bandwidth of duplicate data traversing the network backbone.