Rosetta Phone

Most DoD missions occur in areas of the world where the native language is not English. The warfighters find themselves in situations where a quick, real-time translation of signs, placards, and documents could lead them out of dangerous situations or provide important clues and information relative to the mission at hand. In places such as Afghanistan or North Korea, this turns out to be a particularly difficult problem, since the languages commonly used in these countries are written with characters that cannot be typed into a translation device or even searched for in a dictionary (unless the user has a significant knowledge of the given language.) A valuable tool would be a handheld device (e.g., a mobile telephone or PDA) that would be capable of translating the text found in natural scenes into spoken English and text. The device should also possess the ability to transmit the captured image and text information to other warfighters in the area or to the warfighter’s command and control organization or intelligence agencies that may assist in understanding the image as it applies to the larger overall mission. The Rosetta Phone takes the form of a handheld mobile device (HMD) such as a mobile telephone or a PDA. The HMD is assumed to be equipped with a color camera and, at least part of the time, wireless network capabilities. In order to be able to provide a translation of text found in natural images, the HMD needs to be programmed to perform the following sequence of tasks: • Natural Scene Image Acquisition • Text Segmentation (TS) • Optical Character Recognition (OCR) • Text Translation (TT) • Audio Output Once these tasks have been performed, several output items are available for display/listening, including the original image, the segmented text image, the optical character text, the English text translation, and the audio file (MP3) of the translation. The user will be able to choose to transmit any of these to a server, or to other users’ HMDs supporting the mission. Each of the five tasks needs to be accomplished by the HMD in real-time and autonomously to ensure operation without the help of a remote server.