Multimodal Dynamic Spatial Tracking for Enhanced Automatic Manufacturing – Extended Reality (STEAM-XR)

XR combined with 3D AI technology has the potential to provide a new platform for our maintainers to continuously develop and augment their skills. Current solutions suffer from rigid pre-recorded training content that is not appropriate for the high mix low volume nature of aircraft maintenance and repair. Specifically, active training and task assistance with automated generation of training content and digital twins can enable maintainers to fully realize the potential of XR-based for aircraft sustainment. Due to the challenging operational environments and high-mix maintenance tasks, it is not feasible to install external 3D scanning and tracking hardware for every possible use case. Doing the tracking through the XR device is key to solving the challenges for the envisioned high-mix use cases. However, there are significant challenges that need to be addressed to do active training and task assistance.The key challenges in providing active task assistance and automated training content generation for high-mix low-volume tasks in operational environments are primarily the following:•    Dynamic fast-changing environment •    Complex 3D geometries of parts that vary from serial number to serial number due to differing stresses, repair cycles, and manufacturing variations•    Different tasks requiring guidance based on the specific nature and state of the task that cannot be pre-programmed. GridRaster’s Spatial AI and Dynamic XR solution overcomes above challenges to enable active training and task assistance. The scope of the Phase II effort is to research and develop a prototype solution for dynamic object and scene tracking and provide active assistance to a user to complete the task without errors and with high efficiency. The scope will include R &D, testing and evaluation of a prototype of Multi-modal Dynamic Spatial Tracking for Enhanced Automatic Manufacturing – Extended Reality (STEAM-XR) to demonstrate the following:•    Register multiple objects in the scene, estimating their pose, and overlaying and rendering the 3D model of the objects.•    Automatic tracking of multiple objects in the scene, continuously estimating their pose, and overlaying and rendering the 3D model of the objects.•    Generate “state knowledge graph” of a scene using multimodal visual inputs – images, 3D captures from LIDAR, and video data captured from the XR-device and/or external cameras.•    Track the current state and the user activity and provide active guidance to complete a task.