Glove-based Tactile Streaming of Braille Characters and Digital Images for the Visually Impaired

This project will achieve a computerized system that supplies digital content, both lexical and graphical, suitable for children, students and working adults of the Visually Impaired (VId) community, to increase their autonomy and potentiate their Quality of Life and societal contribution. While there is notable academic research, such as a high-density pin display tablet, there is no commercial, broadly functional, full-page Braille reader. A tablet reader is potentially limiting in comparison with bimanual haptic gloves, limiting any content complementary to Braille text. In contrast, haptic gloves support both Braille lexical content and palpable digital images.The proposed system will encapsulate a full-length Braille reader, not limited to a page at a time but supplying a limitless stream of characters readable by four fingers in each hand as dictated by the content, while also supporting haptic actuation of embedded pictorial content. This glove-based approach will unlock, for an underserved VId clientele, a vast collection of multimedia content currently the exclusive domain of the seeing population, as Braille lexical content and palpable bas-relief of graphical content.Phase I will center on a haptic system for Braille display of full-length text, sensed by the fingers of both hands and based on a redesigned commercial bimanual glove-based haptic system. A software algorithm will stream and stagger the Braille content to each finger to replicate the bimanual reading process of a VId reader. The system will also feature a tactile bas-relief version of digital images embedded in the text.Phase II will develop Haptic VR, demonstrating the utility of overlaying Braille characters on a haptic curved surface and paving the way for vitally important haptic sensing of labeled 2D/3D graphics by VId users. A long-term goal is to apply this VR foundation to geolocated Haptic AR, enabling haptic overlay of Braille labels on everyday surfaces, from appliances to groceries, by suitable integration of haptic rendering with 3D range-sensing and DNN-based object recognition. Specific Aims – The Phase I Aims are as follows.Aim 1. High-resolution haptic gloves for Braille streaming- Development of bimanual high-resolution haptic gloves, featuring dense actuation at eight fingertips, by Haptx. The thumbs will also get haptic feedback.Aim 2. Haptic rendering of lexical and graphical content2A. Lexical reader - A software transcription will support spatiotemporal streaming of lexical data that moves the haptic footprint of Braille characters serially from the first to the eighth finger.2B. Graphical reader – The conversion of a digital image to a haptic bas-relief will exploit fast DNN-based image semantic segmentation, where each label will be mapped to an actuator pressure.2C. Finger-based calibration- Both modalities will be supported by a virtual page defined on a tabletop, whose coordinates are found by a simple calibration based on three points identified by the index fingers.