STTR Phase I: A Self-Learning Approach for In-Vehicle Driver and Passenger Monitoring Through a Sensor Fusion Approach

The broader impact of this Small Business Technology Transfer (STTR) Phase I project will result from the introduction of a state-of-the-art driver monitoring system using artificial intelligence to detect distracted driving or poor driving practices. It can also be used for driver coaching and education, as well as to improve driver attention. The system will help minimize accidents and create safer roads and work environments. End users include automotive original equipment manufacturers (OEMs), commercial fleet operators, taxi and ride-sharing companies, heavy machinery and crane operators, rail and aviation operators, and operators of specialized transportation systems, such as school bus services and charter vehicles. This Small Business Technology Transfer (STTR) Phase I project will exploit data from different camera and inertial sensors inside a vehicle to monitor and assess the attention of the driver. The driver’s gaze and upper body pose will be evaluated separately using artificial intelligence (AI) methods and the results combined to generate an overall estimate of the level of driver distraction. The proposed framework is expected to generate reliable results even in cases of high face occlusion. The technical objectives of the project include to: 1) Explore supervised and unsupervised methods to track the driver's body movement using depth and RGB sensors, addressing the challenges and drawbacks of current vision-based algorithms in real-world driving conditions; 2) Design a novel deep learning framework to integrate the driver's body pose with his/her attention level to infer driver's activities (e.g., such as using portable devices, eating, drinking, and other activities); 3) Develop new models of driver visual attention to obtain confidence levels in the estimated driver's gaze, estimated shoulder pose and joints positions; 4) Develop multi-modal end-to-end deep learning frameworks that integrate multiple sensors to provide important features for monitoring and assisting the driver; 5) Implement the system on low-power commodity hardware that is cost-effective and scalable. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.