Hokkien Low Density Language Capability

Toyon proposes to establish the feasibility of a novel approach to performing S2ST between a target language, such as English, and the low density spoken only language Hokkien. While we will focus of Hokkien during the research and development efforts of Phase I, our system can be tailored to any low density spoken language. Our approach leverages our recently developed Android Team Awareness Kit (ATAK) speech processing pipeline for robust noise suppression and speech intelligibility enhancement. Following noise suppression, speech will be transcribed by OpenAI Whisper by OpenAI. While Hokkien is not currently among the 97 languages supported by Whisper, the component sounds and short phrases of Hokkien overlap with Mandarin, Japanese, and some English. Thus, we expect Whisper will transcribe Hokkien into snippets of these and possibly other languages. These transcriptions will form a textual encoding that can be further encoded to a standardized vector representation, via T2U conversion. We will then adopt the VALL-E X framework or similar, in order to decode to phonetic objects of the target language, e.g. English. Last, the decoded objects will be synthesized to speech. The reverse will be similar, but will leverage Whisper’s MT to first translate the spoken language to Mandarin.