Toolkit for Managing Evaluation and Testing for Red Team Investigations of Cyber Security (METRICS)

Adversaries have become increasingly proficient at cyber attacks against our military"s command and control (C2) infrastructure. Maintaining security requires high-fidelity assessments of software services, often implemented as cyber Red Team exercises in which Systems Under Test (SUTs) are subjected to attacks designed to evaluate defensive capabilities. These tests produce massive amounts of data with subtle patterns and effects that can be difficult to interpret post-experimentally, let alone in real time where these effects would enable more thorough, dynamic, and realistic testing of SUT security. To address this need, we propose to design and demonstrate a toolkit for Managing Evaluation and Testing for Red Team Investigations of Cyber Security (METRICS). The METRICS toolkit includes four key components: (1) a library of contextualized metrics that incorporate a full understanding of system and attack implications to support real-time assessment of the SUT cyber defense; (2) intuitive authoring tools for customizing and developing metrics for evolving SUTs, attacks, and experiment needs; (3) adaptable and adaptive visualizations that present analysis results in manner that ensures observability of critical patterns during ongoing experiments; and (4) a collection harness that employs COTS packet sniffers and agent-based data collection tools to non-intrusively collect data for analysis.