Framework for Machine Translation and Processing of Low Resource Languages

Company personnel recently authored the book "Low Resource Social Media Text Mining", and for this project we will draw upon our teamÆs demonstrated expertise in low-resource language NLP, crawling, large-scale AI, and product development. We propose to deliver a framework to extend state-of-the-art capabilities to low resource languages. The solution will extend machine translation, stance detection, event detection, and summarization to heretofore unmodeled or poorly modeled languages of interest.Ā The solution is able to leverage both text and audio for training and supports crawling identified sources. For audio, it utilizes transliteration to an existing script and conversion of such transliteration to clean text. Given the assembled corpus, whether directly sourced or obtained via audio, the solution incorporates this sample in a query to a large language model (LLM) supporting extended context lengths. This approach can rapidly extend the full gamut of state-of-the-art capabilities to a new language. The framework also can make the corpus available to developers of new models, such that the language can be supported natively in future LLMs and thereby automatically garner the advances of the rapidly progressing LLM community. The solution emphasizes ease of integration into arbitrary data science and analytic workflows.