Natural Language Dialogue for Supervisory Control of Autonomous Ground Vehicles

Unmanned Ground Vehicles (UGVs) currently deployed on the battlefield are driven with Operator Control Units (e.g., joystick or point-and-click interface), which requires the full attention of an operator. This means the operator cannot concentrate on maintaining situational awareness, security, or other tasks. To reduce the burden on the operator, SoarTech with our partner Southwest Research Institute propose to develop a natural dialogue interface for interacting with UGVs, where an operator can interact with a UGV using speech and gesture. The UGV will be capable of taking high-level commands, engaging in dialogue with the user to ensure that the command is clear and providing feedback to the user during task execution. To accomplish this, we will design a multi-modal Smart Interaction Device (SID) for UGV control, based on proven Intelligent User Interface system that we have used for supervisory control of tactical simulation entities. We will extend this prior work to a new input modality (e.g., pointing gestures) and a new plug-in to interface with SwRI"s UGV. This approach will enable supervisory control of one or more UGVs, and greatly reduce the workload of the UGV operator.