ASIM-TECC: ASI Measures for Test and Evaluation of Cooperative Communication

Artificial intelligence (AI) is approaching a turning point in which it briskly evolves from a tool for information lookup and processing in knowledge-based tasks to a teammate that monitors the team and its members; and accurately infers beliefs, predicts actions, and executes cognitive work with humans. This will transform individual taskwork, teamwork, and human-machine dynamics on a wide variety of tasks. It will require a capability to assess the extent to which machines consider the current beliefs (mental models) and likely future actions (theory of mind) of human teammates, in much the same way that good human team members build a theory of mind of their partners. The factors that enable teamwork among humans have been well-explored by organizational psychologists, but those that enable teamwork in hybrid, human-AI teams are only emerging. Standardized frameworks to assess an artificial social intelligence’s (ASI’s) capabilities do not yet exist in the way they do for human teammates. Their absence is limiting ASI developers’ ability to quickly understand the consequences of their design choices on the functioning of hybrid teams and is slowing the cycle of development as a result. We propose to develop a system of ASI Measures for Test and Evaluation of Cooperative Communication (ASIM-TECC) to address this current measurement gap for ASI developers. ASIM-TECC consists of (1) a framework defining measures and experimental manipulations of hybrid (human-AI) teams grounded in social and computer science research; (2) software to help select and operationalize measures and manipulations to assess ASI; (3) a baseline ASI that is grounded in a mathematical model of cooperative communication (i.e., how people communicate effectively) and can be configured with varying ASI capabilities; and (4) empirical data validating the measures, manipulations, and model in a domain of DoD interest to include urban search and rescue (USAR), humanitarian assistance and disaster response (HA/DR), or similar domains. Aptima, Inc., and the Cognitive and Data Science Lab (CoDaS) of Rutgers University Department Of Mathematics and Computer Science will work in close coordination to develop the measure and manipulation frameworks and implementation (1 and 2, led by Aptima) and the modeling (3, led by Rutgers). The Rutgers model is an instance of a potentially powerful ASI that can be used diagnostically to assess the state of a team, and prognostically to estimate the impact of an ASI intervention on the team. Thus, it is an appropriate ASI model to use in validating the measure and manipulation framework (4, by Aptima). The products associated with this effort can be used by developers in combination to understand the behavior and performance of their ASI relative to a validated and modular benchmark ASI, which can help ASI developers address specific needs throughout the design cycle.