Semantics-Aware Web Screen-Reading Assistant

This Phase II project builds upon previous research to develop a Semantics-Aware Web Screen-Reading Assistant that empowers people with vision impairments to interact with the Web more naturally and efficiently via a spoken-dialog interface that is aware of high-level semantic entities such as an article section, a date picker widget, etc. The Assistant organizes the web content into a collection of meaningful semantic entities by using the semantic cues in HTML5 and ARIA specifications, as well as automatically identifying such entities with intelligent content analysis algorithms. The ability to interact with semantic entities enables users to say commands such as “Read the article,” “Log in,” “Next result,” as well as interact with common widgets such as the Date-Picker by saying “Next month,” “Which day is March 13th,” etc. The Assistant provides a spoken-dialog interface to interact with users, interprets and disambiguates user utterances, maintains a dialog with the user, and automates browsing actions on behalf of the user. With the Assistant, users utilize fewer shortcuts, spend less time listening through irrelevant content, perform browsing tasks more efficiently, and, very importantly, do so without having to let go of the screen readers they are used to.