Multisensory Augmented Reality as a bridge to audio-only accommodations for inclusive STEM interactive digital media

Only 3% of individuals who are blind or have low vision (BLV) have been employed in a STEM field. The lack of accessible accommodations to visually rich dynamic STEM content creates roadblocks to success for BLV students in STEM education throughout their academic journey. The goal of this SBIR is a usable system of accommodations for Interactive Digital Media (IDM), so students with profound visual challenges can learn in an essentially equivalent environment as their sighted peers, thus increasing STEM content knowledge, independent problem-solving, and building teamwork within an inclusive classroom. The product to be developed in Phase I will be a game-based set of puzzles for optics, delivered through a web browser, and aligned with middle/high school NGSS standards for physical science. Included will be two alternate interfaces for accessibility, one that uses tactile manipulatives that are recognized by a webcam, so the system can deliver audio instructions for the IDM, and the other using audio-only feedback with keyboard controls. The proposed method is a first-of-its-kind accommodation for STEM IDM incorporating insights from the cognitive theory of sensory substitution. The central hypothesis is that a two-step approach for developing foundational spatial skills with the tactile plus audio approach will lead to more success when using an audio-only interface. This dual accommodation method for STEM IDM goes beyond Web Content Accessibility Guidelines to set a new standard for accessibility, usability, and pedagogical effectiveness. This project’s development is grounded by the Universal Design for Learning framework of eliminating systemic barriers by creating methods that match the widest possible range of users’ needs. It is anticipated that the general population of students will find these accommodations helpful, creating a more inclusive learning environment and increasing self- efficacy and success in STEM, in school, college, and careers. Aims in Phase I include 1) Collecting insight from Teachers of Visually Impaired, both for product design and as to the potential for adoption of the product, 2) Iterative development of the Optics Game with two alternate interfaces for blind students, and 3) A multiple baseline single case design (MB SCD) study of blind high school students to test the hypothesis that the tactile plus audio interface can support student use of the more inclusive and equitable audio-only interface, by helping to develop key spatial and visualization skills. With a successful Phase I project, in Phase II the dual accommodation system will be implemented in a suite of game-based IDM for physical science, and a longer- term MB SCD study will be undertaken to measure effectiveness in terms of concept knowledge and STEM self-efficacy. The value proposition of providing truly accessible and inclusive STEM IDM will lead to commercialization through partnership agreements with established educational content providers.