STTR Phase I: Spatial Artificial Intelligence System for the Visually Impaired (NavigAid)

The broader impact/commercial potential of this Small Business Technology Transfer Phase I project are its two intended contributions: it will advance computer vision for navigation and it will create unprecedented opportunity and independence for individuals with visual impairment. 253 million people with blindness or visual impairment around the world experience difficulty to independently navigate in new spaces. The lack of independence directly leads to disadvantages in getting education and joining the workforce. This project, NavigAid, is aimed at creating a breakthrough spatial intelligence aid offered as a mobile application which will solve spatial tasks from locating objects to identifying paths of navigation, and to generating rich descriptions of the surroundings. Solving this problem will unlock an estimated direct economic benefit of $26B annually for people with visual impairment around the world, by decreasing the need for human assistance. The main technological innovation of this project, Ally Networks, is a novel neural network architecture that can learn more robust representations than existing models are able to. The robust representations prevent the networks from making mistakes and make them more operable and useful in situations that need high reliability. Ally Networks thus represent a potential breakthrough to the field of computer-vision navigation. This Small Business Technology Transfer Phase I project will introduce a novel spatial intelligence system, NavigAid, to assist individuals with visual impairment in crucial navigation tasks. NavigAid will generate contextually relevant, task-oriented spatial information from smartphone cameras. With NavigAid, users will be able to navigate independently in unfamiliar, complex environments, thus achieving unprecedented mobility. NavigAid will advance assistive technologies by providing unprecedented services, such as locating objects and generating functionally relevant natural language descriptions of complex environments. The core innovation in this project is Ally Networks, a novel neural network architecture that learns robust spatial semantics rather than 2-dimensional representations. This unique multimodal learning strategy is a high-risk endeavour, with broad impact if successful. Large-scale multimodal learning is difficult, and our technique?s success will revolutionize the state of the art: neural network vision will transform from systems that fail in inscrutable ways, to systems that never fail under circumstances in which human vision would not also fail. Key objectives of this project are to 1) develop and validate Ally Networks on benchmarks, 2) develop spatial problem solvers that address the most pressing needs of users with visual impairment, and 3) develop a test suite to evaluate the spatial problem solvers. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.