A Specialized Automatic Speech Recognition and Conversational Platform to Enable Socially Assistive Robots for Persons with Mild-to-Moderate Alzheimer's Disease and Related Dementia

Abstract 1 in 3 seniors in the United States dies with dementia, of which Alzheimer’s disease (AD) is the most common form. AD patients suffer from decreased ability to meaningfully communicate and interact, which causes significant stress and burden for both professional caregivers and family members. Socially assistive robots (SARs) have been designed to promote therapeutic interaction and communication. Unfortunately, artificial intelligence (AI) has long been challenged by the speech of elderly persons, who exhibit age-related voice tremors, hesitations, imprecise production of consonants, increased variability of fundamental frequency, and other barriers that can be exacerbated by the neurological changes associated with AD, further complicated by common environmental noises such as the ceiling fan, television, etc. Because of the resulting poor real-world speech and language understanding by available SAR technologies, scarce human caregivers are often required to guide AD patients through SAR interactions, limiting SARs to small deployments, mostly as part of research studies. Unlike existing approaches relying purely on AI, care.coach™ is developing a SAR-like avatar that converses with elderly and AD patients through truly natural speech. Each avatar is controlled by a 24x7 team of trained human staff who can cost-effectively monitor and engage 12 or more patients sequentially (2 simultaneously) through the audio/visual feeds from the patient’s avatar device. The staff communicate with each patient by sending text commands which are converted into the avatar’s voice through a speech synthesis engine. The staff contribute to the system their human abilities for speech and natural language processing (NLP) and for generating free-form conversational responses to help patients build personal relationships with the avatar. The staff are guided by a software-driven expert system embedded into their work interface, which is programmed with evidence-based prompting and protocols to support healthy behaviors and self-care. This SBIR Fast-Track project will leverage the unique data generated by our human- in-the-loop platform to develop new ASR capabilities, enabling fully automatic conversational protocols to engage and support AD patients without human intervention. We aim in Phase I to leverage our unique prior work dataset to train an automatic speech recognition (ASR) engine to enable the understanding of certain types of elderly and AD patient speech more successfully than any currently available engine. We aim in Phase II to incorporate this new engine along with an NLP module into our existing human-in-the-loop avatar system, recruiting a population of AD patients to further train and validate with during a 2-year human subjects study so that we can demonstrate full automation of a significant portion of our avatar conversations with mild- to-moderate level AD patients. Thus, we will improve the commercial scalability of our avatars, while validating our new ASR/NLP engine as the most accurate platform for enabling the next generation of AD-focused SARs.Narrative Artificial intelligence (AI) has long been challenged by the speech of elderly persons, and especially persons with dementia, due to age-related voice tremors, hesitations, imprecise production of consonants, increased variability of fundamental frequency, and other barriers. Unlike existing approaches to socially assistive robots (SARs) relying purely on limited AI for conversation, care.coach™ has been commercializing a SAR-like avatar that converses with elderly and AD patients through truly natural speech, powered by a 24x7 team of trained human staff. The unique data sets that our solution enables us to gather at commercial scale will be leveraged in this SBIR project to develop an automatic speech recognition (ASR) and natural language processing (NLP) engine that is best-in-class for AD applications, improving the commercial scalability of our avatars by reducing our dependence on human staff, while serving as a new AI platform for enabling the next generation of AD- focused, conversational SARs.