Automated OMEN Performance Assessment and Analytics Capability (A-OPAAC)

Over the last three years, CPG has built, tested, and deployed a digitized assessment and analytics capability for measuring analyst and planner thinking and problem-solving skills in support of operations in the information environment (OIE). This work has been part of the Operational Mastery of the Information Environment (OMEN) program sponsored by ONR. The developing tool set is known as the OMEN Performance Assessment and Analytics Capability (O-PAAC), a self-contained system built around the OMEN Behaviorally Anchored Rating Scale (O-BARS) that units can use to self-assess individual and team performance across OIE domains, training events, and exercise contexts. As the OMEN program has progressed, the combined capabilities and technologies being developed by various performers are advancing closer to readiness for operational transition to the Fleet. One key area requiring continued focus is the measurement and evaluation component of the OMEN program. Specifically, the theoretical construct underpinning many of the software applications being developed is the BEND Framework (Carley, 2020). BEND is a theory describing the types of influence strategies that go beyond the widely known 4Ds – distract, dismay, distort, disrupt (Nimmo, 2015) – to describe information maneuvers about shaping the narrative and the community of actors. While the current O-BARS applies to any OIE exercise context and is technology agnostic (it can evaluate analyst performance regardless of the specific tools and platforms they are using), a more focused way of assessing BEND performance is needed to support ONR’s objectives for technology transfer. In addition, a stronger, more cohesive, and efficient measurement and assessment capability is needed that blends the power of human-in-the-loop adjudication with the efficiency of automated collection and aggregation of measures of performance in near-real time via automated means. To realize these goals, three lines of research and development must occur. First, the research team must engage with BEND SMEs to build a BEND-specific assessment to gauge analyst and planner performance understanding, using, and applying BEND in their work. Second, the research team must clearly understand, delineate, and articulate the different ways analyst performance in BEND mastery and application can be detected when using OMEN software. This understanding will feed into the construction and validation of an integrated framework that describes what good performance looks like and how development and progression towards mastery is likely to occur when engaging with OMEN software. Third, the research team aims to build prototypes for automating the collection and summarization of analyst and planner BEND performance during OMEN training and exercise contexts. Together, these efforts will advance multiple components of the OMEN program goals while moving towards a self-contained system for measurement and evaluation.