Multimodal Interaction Technologies to Support Small Unit Leaders

The war in Ukraine has shown that aerial vehicles are a significant part of modern warfare serving in tasks from aerial surveillance to missile defense. However, the current state of drone control requires an operator to have their head down and hands tied to a manual vehicle controller at all times which affects situational awareness (SA) and ultimately operator security. New forms of Human Machine Interfaces (HMIs) and human-machine teaming technology provided in this SBIR could greatly free up battlespace awareness of UxS operators of the future who will need to control drones swarms, complex behaviors, and dynamic retasking. The primary objective of the proposed Phase II effort is to prototype, conduct testing, and evaluate (T&E) a unified HMI which is able to integrate a wide variety of wearable controllers and 3D visualization displays on a diverse number of Ground Control Stations (GCS) in order to enhance the UxS warfighter experience and improve CONOP success rates. The solution will provide a clear path for wearable companies to integrate with various GCS which are used by MARSOC and NSW, starting with Tomahawk Robotics Kinesis and QGroundControl (QGC). The primary effort is to develop consistent, synergistic displays which run on Head Mounted Displays (HMDs) and End User Devices (EUDs) for UxS Command and Control (C2). Market research, end user evaluations, and demonstrations will be shown of the Graphic User Interfaces (GUIs). This differentiation of the GUI is that it can be contextualized to specific CONOPs and wearable combinations for ISR CONOPs with a roadmap for intelligent automation to enhance the UxS operator experience. For this reason we are calling this middleware software solution “MISMATCH” which stands for Multimodal Interface System for Man-Autonomy, Teaming, Control, and Handoffs. Phase II efforts will use state-of-the-art HCI COTS technologies wherever possible to inform what the intended MISMATCH kit should look like, keeping a steady on technology maturation which could be field ready by the end of the Phase II PoP. Both virtual and live T&E efforts will be led by our teammates at Rogue Wolf Advanced Research (RWAR) and Stone Solutions and Research Collective (S2RC) to inform the design and population of the foundational multimodal UxS HCI matrix that will inform the end MISMATCH solution.