Designing Large Data Handling Architectures

We focus on a solution for ultra-scale data handling, into the terabyte range and beyond. Our solution leverages the latest advances in storage technology, and can be used in a cloud, to exploit cloud-native features. We address the following functional requirements: discoverability of data sources, federated source handling for data originating in domains with different admin policies, heterogeneity of data, retrievable data (both push and pull), classifiable data, labeled data as to classification, searchable data using metadata for semantic searching, transformable to extract, shape, and analyze data streams, shareable to allow consumption by stakeholders with different permissions, management of the overall architecture, and analyzable to analyze data in the context of one or more ontologies. We apply ultra-scale columnar oriented data handling, grid computing, semantic tagging of data, and related concepts to achieve the functional requirements.