Person Aware Liaison (PAL)

Collaborative humanndash;robot teaming behaviors use the complementary strengths of humans and robots and are at their most successful when the human trusts in their robot companion. However, despite contemporary approaches to make robots more transparent in their behaviors and safer in their operation, humans still do not trust robots the same way they do a human partner.The inability of current systems to observe all communications from their partner, draw usable inference from the contextual clues of the interaction, and respond in a courteous manner may prohibit humans from placing full trust in their robot collaborators.To that end, we propose Person Aware Liaison (PAL) as an intermediary interface between the human and the robot as they complete a collaborative task. PAL generates multimodal observations (speech, gestures, and where the humanrsquo;s gaze lingers) during the execution of the task and fuses the information into a singular natural language transcription of events. This transcript is queried by a Generative Pre-trained Transformer (GPT) architecture, a state-of-the-art text-based inference model to understand the humanrsquo;s intent. Finally, PAL acts upon the extrapolated intent by developing safe, dexterous motion plans for an articulated robot arm and executing them in a courteous manner that avoids interrupting the human user while they are focused on completing task subgoals.PAL is task agnostic and can use modular task knowledge to quickly adapt to new applications. We will demonstrate the capabilities of PAL both in simulation and in a physical setting using a collaborative welding task.