SBIR Phase I: Virtual Music helper, next-generation educational social platform for conservatory music practice and performance empowered by AI and AR

This SBIR Phase I project focuses on development of a prototype for a Virtual Music Helper (VMH) for children. Research confirms that music and music learning enhances intelligence, learning and IQ specially in kids. Even children with attention deficit/ hyperactivity disorder benefit from listening to music beforehand in mathematics tests. Children who learn or practice music have shown better results in STEM subjects. However, studies also suggest children opt out of music class based on false belief that they lack musical ability and many kids often find the music practice non-engaging, and boring eventually losing interest and quitting. Parents find it challenging to entice the kids to sit down and practice or they cannot help their kids because either they don?t have time or are not music savvy themselves to help them. The virtual music helper can read the kids? music homework, guide them through the exercise and provide immediate correction. This helper is not to replace teachers, but help the parents at home and encourage kids to practice more. In broader terms this project may spur significant research in finding effective methods for tutoring kids in subjects other than music such as math or language, or for kids with special needs. This SBIR Phase I project proposes a Virtual Music Helper (VMH) that is empowered by Artificial Intelligence and Augmented Reality for kids. VMH is a 3D mobile-virtual-human (avatar) that can be personalized for each kid and conduct live dialog with kids both from visual and behavioral perspectives. VHM offers smart content for teachers about each student's progress and weak points. The technical challenges this project will address is the detection and correction of mistakes kids make including polyphonic pitch detection from acoustic instrument, conversion of note sheets to machine readable music format with high accuracy and speed, empowering the avatar with decision making algorithms to provide proper correction and feedback verbally and visually. This platform is also intended to support multiple popular instruments and multiple languages. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.