Scalable perfSONAR Extensions using Software-Defined Measurements and Cloud-hosted Archival

In today’s enterprise networks supporting data-intensive applications, there is an apparent isolation between the ‘application’ and ‘measurement’ planes. Network operators use system and network measurements to infer application-level performance; concurrently, they also use application performance independently to validate system and network performance. Even with the wide adoption of perfSONAR, a multi-domain measurement and monitoring framework in high-performance computing and networking communities, the performance transparency of application and measurement planes is still opaque. Consequently, the ‘control’ plane actions are not being confidently orchestrated, and existing systems still continue to be manually configured with data silos and are slow to adapt to changes. As a result, there are significant challenges to: (i) provide awareness to applications regarding when/where measurements are running or need to be run, (ii) co-schedule application and measurement infrastructure to reap the mutual gains in their joint orchestration and correlation analysis, and (iii) exploit correlation in application and measurement planes to effectively analyze and visualize performance for intelligent adaptation of Big Data supporting resources by control frameworks. The overall objective of this SBIR proposal is to investigate and develop a set of perfSONAR-compatible extensions that provide a cloud-hosted “Global Measurement Archive” that will decrease measurement overlap, increase network-wide performance intelligence. It will also allow operators and Big Data application users to interface with each other effectively and customize application’s needs of measurement awareness. The extensions will be based on our experiences -- through a series of research experiments -- where The Samraksh Company discovered that the opaqueness is due to the fact that much of the intelligence to be used from measurement plane needs to be centrally managed and carefully customized based on application-specific needs. In addition, the extensions will address the market need to closely couple ‘software-defined monitoring’ with ‘software-defined networking’ in particular, but also harmonize the needs of other resource dimensions (e.g., CPU/GPU, storage) for a ‘software-defined data center’. In order to fully support this endeavor Samraksh is reorganizing to recognize Narada Metrics as new product division in which PI Alex Berryman is the full time CEO. Our project will build upon the Narada Metrics “measurement operating system” developed by Samraksh in collaboration with MU, and will result in a cloud-hosted Global Measurement Archive and an ‘App catalog’. Multi-domain perfSONAR-based network monitoring Archive and App catalog that aids Big Data enterprises to gain performance visibility of cyber resources. Commercial Applications and Benefits: Data-intensive science related DOE labs and Universities, as well as Big Data related network operators of commercial enterprises, will be the audience for Phase 1 trials of our measurement App catalog. The Measurement Archive and App catalog will enable wider-scale enterprise use of performance monitoring data that is tangible to be used for orchestration of network resources using widely used standards such as OpenFlow, and OpenStack – in compliance with enterprise security directives. Building upon the early trials of these users, The Samraksh Company will develop a commercialization strategy that will target the Measurement Archive and App catalog to be widely integrated into enterprises that monitor end-to-end multi- domain paths supporting Big Data traffic, and cloud-to-edge performance monitoring.