A Smart Interaction Device for Multi-Modal Supervisory Control of Multiple RPAs

ABSTRACT: Demand for Remotely Piloted Aircraft (RPA) within the U.S. Military has significantly increased over the past decade, including Micro-Air Vehicles (MAVs). However, effective use of these systems has been hindered by the lack of natural interfaces for controlling them. Currently multiple operators are required to manage all aspects of a single aircraft, and pilots interact using joysticks or point-and-click interfaces that require their full attention. The work proposed here seeks to enable one operator to exert supervisory control over multiple MAVs using multiple modes of interaction in order to reduce the burden on the operator while maintaining situational awareness. To accomplish this, we will build and evaluate a prototype multi-modal Smart Interaction Device for MAVs (SID/MAV), which will allow pilots to use natural means of interacting such as speech and gesture to interact with multiple MAVs. We will extend our prior work to this new domain, including new input modalities and a new plug-in to interface with RPA control systems via STANAG 4586 messaging. We will evaluate SID/MAV on a portable tablet in a laboratory setting and in a live-flight setting. SID/MAV will enable supervisory control of one or more RPAs, and greatly reduce the workload of the RPA operator. BENEFIT: A natural multi-modal interaction system for supervisory control of RPAs has the potential to change the face of RPA systems. Current battlefield RPAs effectively require multiple people to operate, including navigation, system health monitoring, and payload and mission management. Changing the way in which an operator can interact with each RPA, using more natural input modalities and including feedback from that RPA in natural modalities, frees the operator to work at a higher level of interaction. This in turn will allow for a single operator to more easily manage multiple RPAs, much like a human team lead can manage multiple subordinates. Our system, a Smart Interaction Device for Micro Air Vehicles (SID/MAV) will be capable of managing diverse RPAs though a single natural control interface. SID/MAV will be deployable to multiple computing platforms, such as tablets or laptops, enabling wide access to RPA control, either in a control center or in the field. Beyond RPA applications in reconnaissance, surveillance, and target acquisition (RSTA) missions, supervisory control of autonomous entities has a great deal of potential in a range of fields: managing all kinds of ground and airborne unmanned systems; managing entities in large-scale simulation environments, where hundreds or even thousands of entities play a part; and managing intelligent virtual characters in immersive trainers.