Fizz Reader: Interactive Accessible Data Visualizations Through an NLG Interface

Specific Aims There are many modern approaches to accessibility in data visualizations, from simple but inadequate alternative data tables to excellent work with tactile graphics, haptics, and sonification. Each has its benefits and limitations, but common underlying weaknesses to the advanced methods are lack of precision and need for reader training. By contrast, most of the 7.7 million Americans with visual disabilities are at least moderately skilled in using a screen reader. However, this typically limits users to serial data point access, the equivalent of a data table, which does not provide an equivalent experience to the advantages of data visualizations. It decreases independence and professional opportunities, increases stress, decreases quality of life, and puts vulnerable people at risk when crucial public health information is disseminated in graphical-only form, such as in the current COVID-19 pandemic. The long range goal of this Phase I project is to create technologies that permit authors and developers of web sites to effortlessly publish charts, diagrams, and infographics that are fully usable by all people, in particular people who are blind, low-vision, or have cognitive disabilities. In this Phase I SBIR project, we will assess the feasibility of creating interactive contextual automatic descriptions that enable the reader to construct an accurate working mental model of the data with minimal effort and time, to perform tasks and make decisions. Fizz Studio has created a software package, Fizz Charts, that generates accessible keyboard-browsable charts for use on any website. We seek to enhance this with Fizz Reader, a novel interactive interface that uses natural language generation (NLG) to enable the user to query the chart for quick answers about each data point, its relationship to other data points and to the chart statistics, and to high-level or detailed trends and patterns in the data. The effect is of one person explaining the chart to another over the phone, and providing relevant and rapid answers to help the listener understand as much of the data as they wish for a core set of 7 common chart types: bar; line; pie; histogram; scatterplot; heatmap; and flowchart. Aim 1: Develop effective interactive NLG model and engine module To concisely communicate relevant details to the user, we will design a comprehensive set of tasks for all supported chart types, and a set of NLG templates for each chart component (e.g. data point, axis, title). We will use these NLG templates to develop a software module which composes colloquial utterances from an internal statistical data model we build from the data extracted from the chart. Each set of options will represent the affordances optimal for the chart type (e.g. comparisons for bar charts, changes over time for line charts). This module will have a client-server API architecture, to make it adaptable to multiple user interface modalities, including the screen reader intermediary in Aim 2, as well as a standalone digital assistant or a component in a smart speaker. To ensure effectiveness and clarity, we will perform multi-phase testing for usability and task accomplishment with 10-20 blind and low-vision human subjects, and integrate their feedback. We seek 80% successful task accomplishment across all chart types and tasks. Aim 2: Implement a browser-based screen reader intermediary User Interface Module (UIM) To integrate our NLG module directly into the browser, for on-demand presentation of the generated NLG chart information to users of screen readers, we will implement an option-based interface using JavaScript and ARIA technologies. Each set of options will be context-dependent on the currently focused item (e.g. data point, axis, title), and will reflect the optimal tasks for that item in that specific chart type, to enable users to use keyboard input to select from a virtual dropdown of options, and the results will be returned as text to be presented by the screen reader or braille display. We will use a Lean development methodology to test for 95% interoperability across common browsers, operating systems, mobile devices, and screen readers. Conclusion and goals for Phase II The purpose of this Phase I grant is to significantly advance the state of the art for the accessibility of data charts for people with visual disabilities to aid them in their personal and professional lives. In Phase II, we will build on this foundation to support more complex diagram types including schematics, refine the UI for more operations, and explore other deployments such as smart speakers, browser extensions, or app plugins.Project Narrative Fizz Reader, a novel interactive interface that uses natural language generation (NLG) to enable blind and low-vision users to make data visualizations accessible. The effect is of one person explaining the chart to another over the phone, and providing relevant and rapid answers to help the listener understand as much of the data as they wish.