A Smart Interaction Device for Multi-Modal Supervisory Control of Multiple RPAs

ABSTRACT: Demand for Remotely Piloted Aircraft (RPA) within the U.S. Military has significantly increased over the past decade, including a demand for Micro-Air Vehicles (MAVs). However, the increased use of these systems has been hindered by the lack of natural interfaces for controlling them. Currently multiple operators are required to manage all aspects of a single aircraft and operators interact with joystick or point-and-click interfaces that require almost their full attention. The work proposed here seeks to enable one operator to control multiple RPAs in a supervisory control fashion using multiple modes of interaction in order to reduce the burden on the operator and enable multiple RPAs to be managed in parallel. To accomplish this, SoarTech, in consultation with the University of Michigan, will design a multi-modal Smart Interaction Device (SID) for multiple RPA control, based on a proven Intelligent User Interface system that we have used for supervisory control of tactical simulation entities. We will extend this prior work to multiple input modalities and a new plug-in to interface with existing RPA control systems. This approach will enable supervisory control of one or more RPAs, and greatly reduce the workload of the RPA operator. BENEFIT: A natural multi-modal interaction system for supervisory control of RPAs has the potential to change the face of RPA systems. Current battlefield RPAs effectively require multiple people to operate, including navigation, system health monitoring, and payload and mission management. Changing the way in which an operator can interact with each RPA, including feedback from that RPA in natural modalities, frees the operator to work at a higher level of interaction. This in turn will allow for a single operator to more easily manage multiple RPAs, much like a human team lead can manage multiple subordinates. The SID framework will also be capable of managing diverse RPAs though a single natural control interface. Beyond RPA applications in reconnaissance, surveillance, and target acquisition (RSTA) missions, supervisory control of autonomous entities has a great deal of potential in a range of fields: managing all kinds of ground and airborne unmanned systems; managing entities in large-scale simulation environments, where hundreds or even thousands of entities play a part; and managing intelligent virtual characters in immersive trainers.