Multimicrophone Adaptive Array Augmented with Visual Cueing

DESCRIPTION (provided by applicant): Advanced Medical Electronics Corporation (AME) proposes to develop a smartphone-based audiovisual array that enables hearing aid users to converse with multiple speakers in reverberant environments with significant speech babble noise where their hearing aids do not function well. The array will consist of a smartphone, a smartphone accessory, and a smartphone software application. The smartphone multi-microphone audiovisual array accessory will be in a form factor thatallows attachment to the back of the smartphone. The accessory will also contain a lower power radio by which it can transmit audio signals to compatible hearing aids. The smartphone software application will use the smartphone's built in camera to acquireimages and perform real-time face detection using the built-in face detection support of the smartphone. The location of talking targets will be used to implement an audiovisual beamforming algorithm to improve the signal to noise ratio and consequentlyimprove the user's speech intelligibility. Since the proposed array system leverages a handheld consumer electronic device, it will be portable and low cost. AME will design a production prototype in phase II and perform validation. PUBLIC HEALTHRELEVANCE: The primary complaint of hearing-aid users is their difficulty understanding speech in noise. Only 51% of hearing aid users report they are satisfied with their hearing aids in noise. A discrete device that augments hearing aids in difficult reverberant environments would allow hearing aid users to converse with multiple people in almost any environment including restaurants, automobiles, etc.