3D-SAM: Segment Anything Model in 3D using Dynamic Extended Reality (XR)

WR-ALC has a national defense need to provide efficient and safe repairs by automating processes and preventing human errors to optimize maintenance activities. The use of mobile industrial robots in sustainment and depot environments has grown significantly over the years and provides great improvements in safety, quality, agility, and throughput metrics. A key challenge for mobile robots is to have an accurate 3D spatial map of dynamically changing environments in order to reach the target workpieces without accidentally colliding into other 3D objects or humans in the environment. Segment Anything Model (SAM), and their iterations and improvements, demonstrate promise in generating accurate, prompt enabled, and user tunable 2D image segmentation without additional AI training. Extending this 2D SAM to 3D with using depth camera captures could enable 3D “volumetric segmentation” by stitching together the 2D masks for both color and depth images. The SAM models, however, are large and require significant computation (CPU, GPU, and memory) in order to perform adequately, and is not suitable to run on XR devices. GridRaster proposes to extent its client-server based 3D object tracking and rendering solution and incorporate SAM in the workflow to enable real time zero-shot 3D-SAM with “volumetric mask” visualized on an XR device. The mask generation will be automatic and prompt based (eye gaze, user pointer, bounding box, etc.) and will be adjustable and tunable based on the users’ needs. The scope of the Phase II effort is to research and develop a prototype solution for fast and accurate 3D segmentation mask generation using zero-shot 3D-SAM that would minimize the need for AI training for all assets of interest on which various operational tasks are performed, such as aircraft parts, tools, vehicles, etc. Additionally other 3D objects in the environment, such as toolboxes, fans, chairs, tables, stands, etc., also need to be localized accurately to ensure a mobile robot is able to path plan properly and do not collide with such objects.This real time segmentation and mask generation has been demonstrated to be feasible from XR captures and the following R&D effort would be required to successfully transition this into a production environment:R&D is required to capture the user prompt (eye gaze intersection with the object and/or user selected bounding box) and map that to each captured frame – both color and depth in real time.R&D is required to apply SAM model to each color and depth frame to generate multiple color and depth masks – all from different user pose (location and orientation) – and stitch them together to generate 3D masks in real time.R&D is required to evaluate different SAM models to ensure fast and efficient data processing on the server.Testing and Evaluation of the solution in operational environment at WR-ALC.