Augmented Reality Training Tools with Realistic Animations and Voice Navigation on Tablet and AR Headset for Combat Casualty Care

Development of Augmented Reality (AR) training tools for combat casualty care with realistic animations and voice navigation capability on tablets and AR headset is proposed to enhance the tools that have already been developed during the Phase II effort on fasciotomy, cricothyroidotomy, and lateral canthotomy. Such enhancements are made possible by the recent advancement in both hardware and software technologies, and have the potential to improve the tools’ usability and user experience significantly. Most of the physical task trainers and virtual software tools for medical procedure training present openings of incisions with no bleeding which is unrealistic and affects training effectiveness. The proposed enhancements utilize the lasted augmented reality technology to overlay realistic animations with bleeding effects on top of physical task trainers as well as on virtual models in software tools to increase fidelity in simulation and effectiveness in training.  One of the challenges in using AR devices such as iPad or HoloLens for surgical procedure training is that a user has to use his/her hands for clicking through the software and holding medical instrument to perform the procedure on task trainers. Switching between a software tool and a medical instrument breaks the actual sequence of actions, which affects the training effectiveness. The proposed enhancement leverages the latest artificial intelligence (AI), machine learning (ML) and natural language processing (NLP) technology to help the user to navigate through a procedure with voice commands while keeping the hands free for performing the procedure on a task trainer.  In addition to voice navigation, AR headset such as Microsoft’s HoloLens 2 for commercial applications and IVAS (Integrated Visual Augmentation System) for military applications frees the user’s hands completely in conjunction with voice commands. The proposed enhancement to migrate all three procedures to HoloLens 2 which will accommodate both military applications when IVAS devices are available and commercial applications now. The proposed enhancements will be developed for fasciotomy, cricothyroidotomy and lateral canthotomy tools on iOS, Android, and HoloLens hardware platforms, which would accommodate both government and commercial user groups. The technology developed for the proposed effort can be easily used for developing training tools for other procedures commonly performed in private sectors such as hospitals, medical schools, and medical training facilities.