MATES: Multilingual Automatic Translation Engine for Signs

When the United States military commits to operations in foreign territories, it is important for soldiers to have the ability to understand signs accurately and in a timely manner. We propose the development of a sign understanding and translation systemcalled MATES (Multilingual Automatic Translation Engine for Signs) that can be operated by a novice user to obtain and interpret signs in foreign languages. The system will be comprised of a hand-held Personal Digital Assistant (PDA) and camera along witha dynamically reconfigurable embedded component architecture and software developed for text detection, recognition and translation, and a configuration and visualization interface to aid personnel deployed in foreign countries.Our text detection and recognition approach builds upon our previous R&D to provide a lightweight, trainable, and robust solution which can be easily adapted to different foreign languages. The perspective distortion is detected and rectified to furtherimprove the performance. Our novelty lies on the use of context for both recognition and translation/transliteration. In order to deal with uncertainty in OCR and in the translation of sign content, we will rely heavily on contextual information andoptional feedback provided by the user of the system, combined with robust techniques for indexing and incorporating domain information. The general philosophy will be to provide a system where additional context and resources will be supported by dynamicreconfiguration. The component architecture supports a view such that different resources and software components can be loaded on demand. In addition to military applications the development of an autonomous sign translation system has tremendous commercial potential. For example:¿ Assisting visitors to foreign countries that are unable to read the local language for help with reading transportation schedules, or menus or while shopping.¿ The sign detection module and recognition modules can be integrated into systems for the visually impaired.¿ The sign recognition module can be used in numerous applications including automated mapping, robot navigation, autonomous vehicle navigation and driver support systems