Mobile Signing Math Dictionary with Mouth Morphemes

Deaf and Hard-of-Hearing (Deaf/hh) persons rely heavily on facial movements for communication, whether in American Sign Language (ASL), Signed English (SE), or Spoken English. In ASL, signers add “mouth morphemes”, similar to speech movements, to signs to distinguish concepts that use the same manual sign. Hard-ofhearing persons who use spoken communication, read mouth shapes to distinguish similar sounds. Existing animations of facial expressions and speech fall short of addressing the full range of “visible speech” and mouth morphemes. Our Phase I work shows that users of our animated sign language products want to see animated signing with enhanced mouth movement; and there is evidence that this can enhance comprehension. For this Phase II SBIR project, Vcom3D will develop comprehensive models of mouth movements used in ASL and speech, as well as software tools to animate these movements. To evaluate their effectiveness, Vcom3D and TERC will evaluate the animations in mobile versions of our Signing Math Pictionary (SMP), which provides ASL and SE definitions of math terms for K-3 learners. While the SMP will be the first application of the technology, Phase II research and tools will be broadly applicable to future products using animation to promote inclusion of Deaf/HH students.