AI for Manned-Unmanned Team with NLP

To carry out Manned-UnManned Teaming (MUM-T), pilots need the ability to issue spoken commands to unmanned vehicles from the cockpit. Current speech models are insufficient to clean and transcribe spoken words in a cockpit under all of the conditions pilots face. We propose the development of a new machine learning model paradigm that addresses these challenges and proves the viability of MUM-T to the Air Force. Development will require three stages: Audio Cleaning, Language Modeling, and Task-Relaying. A series of audio processing steps will be used to clean up the audio signal coming from the pilot’s microphone. Each step will have a distinct model that will target specific aspects of the high-noise/high-stress environment. Creating focused models will allow the overall model stack to be lightweight, fast, and more adaptable to different audio setups (different microphones, radio systems, aircraft design and configuration, etc). Model-based speech enhancement uses computer modeling to remove excess noise and regenerate the original speech. Our multistep process will clean out general noise gradients then fill in missing parts of speech. To develop the model, we will build a training dataset of complex speech inputs sourced from a range of conditions and speakers. This will allow it to simulate both noisy environments and less clear accents and speakers under stress. From these datasets we will create randomized combinations of known voices mixed with actual/synthesized cockpit noise to create a wider range of training cases than can be recorded manually. Once the audio is cleaned, a novel language modeling system is needed to properly manage noisy and incomplete commands. Second, the new language model will pull out word sequences. Third, a domain-specific ontology will be used by a semantic model to improve word selections and generate task-specific statements. This will allow the model to be able to self-correct any issues in the recording and interpolate missing words and commands. These knowledge functions will initially be built into the ontology, but over time the model will be able to expand its knowledge base to incorporate new learnings about how commands are structured. Once the spoken commands are isolated, a novel machine model will be used to create task-based orders for the unmanned vehicles. Task-based automation focuses on generalizing tasks and learning those as opposed to specific individual commands. The system can then learn to optimize the general task. For example: • Targeting Commands: SATAN 2 TARGETED WEST GROUP BULLSEYE 270/15 • Tactical Ground Information: LOWDOWN. TWO ACTIVE BULLSEYE 090/10, HOSTILE CONVOY BULLSEYE 270/10 TRACK EAST These three steps will allow pilots to effectively communicate with any number of robotic co-pilots under any conditions they will face.